promote: main → release #132
128 changed files with 9574 additions and 1805 deletions
219
.claude/BRANCHING.md
Normal file
219
.claude/BRANCHING.md
Normal file
|
|
@ -0,0 +1,219 @@
|
||||||
|
# Orchestrator instructions — trunk-based flow
|
||||||
|
|
||||||
|
`feat/*` → `dev` → `main` → `release`. One PR per hop, merges serialised by the
|
||||||
|
orchestrator. Forgejo has no native merge queue, so you are the merge queue.
|
||||||
|
|
||||||
|
Use the Centralis `forge_*` tools for every remote operation — PRs, protection,
|
||||||
|
merges, tags. Use local git only for rebases you have to resolve by hand.
|
||||||
|
Never call the Forgejo API with curl and a pasted token; the server holds it.
|
||||||
|
|
||||||
|
The same file exists in `centralis` with the same rules. Where the two repos
|
||||||
|
differ it is noted inline.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The branches
|
||||||
|
|
||||||
|
Each branch has its own environment here, and `infra/deploy/env-topology.sh` is
|
||||||
|
the single source of truth for where each one lives:
|
||||||
|
|
||||||
|
| Branch | Deploys to | Receives |
|
||||||
|
|---|---|---|
|
||||||
|
| `feat/*`, `fix/*` | nothing | your work |
|
||||||
|
| `dev` | dev — vps1, mesh-only, own Postgres | feature PRs |
|
||||||
|
| `main` | beta — beta.thermograph.org, vps2 | promotions from `dev` |
|
||||||
|
| `release` | prod — thermograph.org, vps2 | promotions from `main`, and hotfixes |
|
||||||
|
|
||||||
|
`deploy.yml` handles all three: the branch selects the environment, a matrix
|
||||||
|
covers backend and frontend, and each leg checks whether the push actually
|
||||||
|
touched its domain before rolling. So **FE and BE ship out of lockstep** — a
|
||||||
|
backend change lands without a frontend deploy. That is the point of the split,
|
||||||
|
and it is why `/api/version` and `PAYLOAD_VER` discipline are load-bearing:
|
||||||
|
whatever you promote, the other half may already be running something older.
|
||||||
|
|
||||||
|
`dev` is the default branch, so PRs target it without anyone remembering to
|
||||||
|
retarget. All three are protected: **everything is a PR**, for humans and agents
|
||||||
|
alike.
|
||||||
|
|
||||||
|
## Who may merge what
|
||||||
|
|
||||||
|
This is not the same question as how a change travels, and it is deliberately
|
||||||
|
asymmetric. `MERGE_AUTHORITY` in centralis'
|
||||||
|
`src/lib/promotion.ts` is the whole policy; `forge_pr_merge` enforces it.
|
||||||
|
|
||||||
|
- **Into `dev` — free.** Merge feature PRs yourself, and do not leave them open.
|
||||||
|
A PR opened against `dev` and abandoned is an unfinished task wearing the
|
||||||
|
costume of a delivered one.
|
||||||
|
- **`dev` → `main` — yours, batched.** Beta is the test environment, so
|
||||||
|
withholding this merge withholds the testing. The judgement being asked of you
|
||||||
|
is *when a batch is coherent enough to be worth testing*, not whether you are
|
||||||
|
allowed to test it.
|
||||||
|
- **`main` → `release` — the owner's.** That merge deploys to prod. Call
|
||||||
|
`promote(from="main", to="release")`, which prepares the PR and returns what
|
||||||
|
would ship, the CI evidence and a recommendation. Then put it to the owner —
|
||||||
|
including "I would wait", if that is what you think. No flag overrides this,
|
||||||
|
and routing around it is not an option.
|
||||||
|
|
||||||
|
`confirm_protected_base` is unrelated to the above: it is the hotfix path for a
|
||||||
|
*non-promotion* PR aimed straight at `main` or `release`, and it is audited
|
||||||
|
loudly.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1 — repo setup
|
||||||
|
|
||||||
|
Idempotent. Re-run it whenever you are unsure; every step reports "already
|
||||||
|
correct" rather than erroring.
|
||||||
|
|
||||||
|
1. **The three branches exist.**
|
||||||
|
`forge_branches(create={branch:"main", from:"dev"})` and the same for
|
||||||
|
`release`. If a branch exists but points elsewhere, the tool says so — that
|
||||||
|
is a divergence to reconcile deliberately, not to overwrite.
|
||||||
|
2. **`dev` is the default branch.**
|
||||||
|
`forge_repo_settings(apply_flow_defaults=true)` — sets the default branch and
|
||||||
|
permits every merge style the chain uses, including `fast-forward-only`, plus
|
||||||
|
rebase-updates (without which the merge queue's core primitive 403s).
|
||||||
|
3. **Protection.** `forge_branch_protection(branch="dev", apply=true)`, then
|
||||||
|
`main`, then `release`.
|
||||||
|
|
||||||
|
`dev` and `main` get the required check; **`release` does not**, and that is
|
||||||
|
deliberate. `pr-build.yml` fires only for PRs into `dev` and `main`, so a PR
|
||||||
|
into `release` reports no status at all — requiring a context nothing
|
||||||
|
produces would make every production promotion permanently unmergeable, with
|
||||||
|
a 405 that names no cause. This estate has already lost an afternoon to
|
||||||
|
exactly that, from a single mistyped check name. If you want it closed
|
||||||
|
properly, do it in this order: add `release` to `pr-build.yml`'s
|
||||||
|
`on.pull_request.branches`, confirm on a real PR that the context appears,
|
||||||
|
*then* pass `require_checks_on_release: true`. The other order blocks prod.
|
||||||
|
4. **Stale branches.** `forge_branches()` lists every branch with its age, how
|
||||||
|
much unmerged work it carries, and a verdict:
|
||||||
|
- `merged` — nothing on it the trunk lacks. Delete freely.
|
||||||
|
- `stale` — unmerged work, untouched 14+ days. **File a `stale-branch: <name>`
|
||||||
|
issue summarising its diff first**, then delete. Never discard work
|
||||||
|
silently.
|
||||||
|
- `ageing` — 2+ days with work still on it. Report it; do not delete.
|
||||||
|
5. **Module map.** `.claude/ownership.md` — read it before partitioning work.
|
||||||
|
|
||||||
|
## Phase 2 — the ongoing loop
|
||||||
|
|
||||||
|
### Dispatching subagents
|
||||||
|
|
||||||
|
Partition by module using `.claude/ownership.md`. Never give two concurrent
|
||||||
|
agents tasks touching the same files; if that is unavoidable, sequence them.
|
||||||
|
|
||||||
|
Inject this into every subagent task:
|
||||||
|
|
||||||
|
```
|
||||||
|
Branch off latest `dev` as feat/<slug>. Touch only these files/modules: <scope>.
|
||||||
|
One logical change, small diff. Rebase onto dev before opening your PR.
|
||||||
|
PR targets `dev`. The description must list: what changed, how it was tested,
|
||||||
|
files touched. Do not merge your own PR — the orchestrator handles all merges.
|
||||||
|
```
|
||||||
|
|
||||||
|
Keep a branch alive **less than a day**. A task bigger than that is several
|
||||||
|
stacked tasks.
|
||||||
|
|
||||||
|
### The merge queue
|
||||||
|
|
||||||
|
You merge **one PR into `dev` at a time**. For each:
|
||||||
|
|
||||||
|
1. `forge_prs(base="dev")` — the queue, with each PR's head sha and triage flag.
|
||||||
|
2. `forge_pr_update(pr=N)` — replay it onto the current tip. If the head sha
|
||||||
|
does not move, it was already current and no CI will re-fire; do not sit
|
||||||
|
waiting for a run that was never scheduled.
|
||||||
|
3. `forge_pr_await(pr=N, until="checks_complete")` — checks on the **rebased**
|
||||||
|
head. The run you saw before the update was for a merge base that no longer
|
||||||
|
exists.
|
||||||
|
4. `forge_pr_merge(pr=N, method="rebase", delete_branch=true)`.
|
||||||
|
5. Only then move to the next PR.
|
||||||
|
|
||||||
|
`block_on_outdated_branch` is set on `dev`, so Forgejo enforces step 2 rather
|
||||||
|
than trusting you to remember it: after one merge, the next PR is refused until
|
||||||
|
it has been replayed. Treat that 405 as the queue working, not as a fault.
|
||||||
|
|
||||||
|
If a rebase conflicts: resolve it yourself if it is trivial and mechanical
|
||||||
|
(under ~20 lines), otherwise hand it back to the owning agent with the conflict
|
||||||
|
context. Do not resolve a large conflict on someone else's behalf.
|
||||||
|
|
||||||
|
### Promotion: `dev` → `main`
|
||||||
|
|
||||||
|
Promote when `dev` is green and a coherent batch is complete — not on a timer,
|
||||||
|
not mid-feature. `promote(from="dev", to="main")` opens the PR with the
|
||||||
|
divergence, the conflict prediction and the changelog written into its body.
|
||||||
|
Merge it with `method="merge"`.
|
||||||
|
|
||||||
|
Unready work belongs behind a feature flag, or not on `dev` yet. **Never promote
|
||||||
|
around it with cherry-picks.**
|
||||||
|
|
||||||
|
`promote` refuses a hop whose two tips have identical trees — the commit counts
|
||||||
|
look like real work while the content is already on the target under different
|
||||||
|
SHAs, and merging it would fire the target's deploys for nothing.
|
||||||
|
|
||||||
|
### Promotion: `main` → `release`
|
||||||
|
|
||||||
|
Prepare with `promote`, then ask. On the owner's yes, merge with `method="merge"`
|
||||||
|
and tag it: `forge_releases(create={tag:"v<semver>", target:"release", ...})`.
|
||||||
|
|
||||||
|
**On fast-forward.** The written policy is that `release` moves only by
|
||||||
|
fast-forward. It cannot today: every promotion so far has been a merge commit
|
||||||
|
*into* the target, which the source never receives, so the branches are mutually
|
||||||
|
divergent by construction and Forgejo will refuse a `fast-forward-only` merge —
|
||||||
|
correctly. The style is implemented and accepted by `forge_pr_merge`; adopting
|
||||||
|
it needs a one-time reconciliation of the protected branches, which is the
|
||||||
|
owner's decision. Until that happens, promotions are merge commits. Do not
|
||||||
|
attempt to "fix" this with a rebase or a force-push.
|
||||||
|
|
||||||
|
### Hotfix
|
||||||
|
|
||||||
|
1. `forge_branches(create={branch:"hotfix/<slug>", from:"release"})`.
|
||||||
|
2. Fix, PR into `release`, checks green, merge with
|
||||||
|
`confirm_protected_base: true`, tag `v<semver-patch>`.
|
||||||
|
3. **Immediately down-merge** `release` → `main` and `main` → `dev`, as merges,
|
||||||
|
not cherry-picks, in the same session. A hotfix that exists only on `release`
|
||||||
|
is a regression scheduled for the next promotion.
|
||||||
|
|
||||||
|
Note that CI does not gate PRs into `release` — `pr-build.yml` fires only for
|
||||||
|
`dev` and `main`. A hotfix's green evidence therefore has to come from somewhere
|
||||||
|
else; say plainly where it came from rather than implying a check passed.
|
||||||
|
|
||||||
|
A hotfix that touches only one domain deploys only that domain. Check the other
|
||||||
|
half's `/api/version` before assuming the fix is live end to end.
|
||||||
|
|
||||||
|
### When you genuinely need to push directly
|
||||||
|
|
||||||
|
`apply_to_admins` is on, which means these rules bind the owner too. That is
|
||||||
|
deliberate — protection an admin walks straight through is documentation, not a
|
||||||
|
control, and every account here is an admin — but it does remove the escape
|
||||||
|
hatch, so the way back in is worth writing down before it is needed at 3am.
|
||||||
|
|
||||||
|
There is no force flag. The supported route is to lift the rule, do the thing,
|
||||||
|
and put it back:
|
||||||
|
|
||||||
|
```
|
||||||
|
forge_branch_protection(branch="dev", delete=true) # rule gone; direct push allowed
|
||||||
|
git push origin dev # the emergency change
|
||||||
|
forge_branch_protection(branch="dev", apply=true) # restored to policy
|
||||||
|
```
|
||||||
|
|
||||||
|
Both the removal and the restore are audited, which is the point: the hatch is
|
||||||
|
open on the record rather than by a quiet exception. Restore it in the same
|
||||||
|
session — a protection rule "temporarily" removed is how a repo ends up
|
||||||
|
unprotected for a month.
|
||||||
|
|
||||||
|
The failure mode that actually justifies this: if the required status context
|
||||||
|
is ever renamed and no longer matches, **nothing can merge into `dev` or `main`
|
||||||
|
at all**, including by you, because `apply_to_admins` no longer exempts anyone.
|
||||||
|
`forge_pr_await` and `forge_pr_merge` both name that case explicitly when they
|
||||||
|
see it. The fix is to correct `pr-build.yml`'s workflow/job names and
|
||||||
|
`CENTRALIS_REQUIRED_CHECK` together — but if you need to land the fix itself,
|
||||||
|
lift the rule as above.
|
||||||
|
|
||||||
|
## Guardrails
|
||||||
|
|
||||||
|
- Never force-push `dev`, `main` or `release`.
|
||||||
|
- Never merge a red PR, hotfix included. Fix CI, or fix the tests in the same PR.
|
||||||
|
- Never merge two green PRs back to back without re-checking the second against
|
||||||
|
the new tip.
|
||||||
|
- If two down-merges or promotions conflict beyond the trivial, stop and report
|
||||||
|
rather than resolving autonomously.
|
||||||
|
- Weekly: `forge_branches()` and report anything `ageing` or `stale`.
|
||||||
|
|
@ -13,7 +13,7 @@ They are wired up in `.claude/settings.json`.
|
||||||
|
|
||||||
## Why prod-guard exists
|
## Why prod-guard exists
|
||||||
|
|
||||||
`agent` has passwordless sudo on prod and beta, and the operator's global settings
|
`agent` has passwordless sudo on vps1 and vps2, and the operator's global settings
|
||||||
allow `Bash(ssh prod:*)` outright. Nothing about `ssh prod 'docker service rm …'`
|
allow `Bash(ssh prod:*)` outright. Nothing about `ssh prod 'docker service rm …'`
|
||||||
goes through a pull request, so CI cannot see it — before this hook, any session in
|
goes through a pull request, so CI cannot see it — before this hook, any session in
|
||||||
any directory could delete production with no prompt.
|
any directory could delete production with no prompt.
|
||||||
|
|
@ -24,9 +24,17 @@ is wrong by construction — the first destructive verb nobody thought of sails
|
||||||
straight through. Being wrong in the "ask" direction costs a keystroke; being wrong
|
straight through. Being wrong in the "ask" direction costs a keystroke; being wrong
|
||||||
the other way costs production.
|
the other way costs production.
|
||||||
|
|
||||||
Beta is guarded as strictly as prod: it serves beta.thermograph.org *and* hosts
|
Beta is guarded as strictly as prod — more strictly than the names alone suggest,
|
||||||
Forgejo, so a destructive command there takes out git, CI and the registry at once.
|
now: vps2 (`169.58.46.181`) runs **both** prod and beta as separate Swarm stacks on
|
||||||
The LAN dev box is deliberately unguarded.
|
the same manager, sharing one TimescaleDB instance, so a "beta" command there has
|
||||||
|
prod's blast radius (see `infra/deploy/env-topology.sh`). vps1 (`75.119.132.91`) is
|
||||||
|
the other guarded host: Forgejo (git + CI + registry) and Grafana/Loki live there,
|
||||||
|
plus the `dev` environment, so a destructive command there takes out git, CI and the
|
||||||
|
registry at once. Mesh IPs did not move in the vps1/vps2 split (vps1 is still
|
||||||
|
`10.10.0.2`, vps2 is still `10.10.0.1`); what moved is which environment runs where
|
||||||
|
— `HOST_TOKEN_RE` in `prod-guard.sh` recognises `vps1`/`vps2` alongside the
|
||||||
|
environment names (`prod`/`beta`) and the raw IPs, so a command naming the host
|
||||||
|
either way still gets caught. The LAN dev box is deliberately unguarded.
|
||||||
|
|
||||||
## Changing the classifier
|
## Changing the classifier
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -19,9 +19,15 @@
|
||||||
# That is also what happens if this script errors or jq is missing, so a bug here
|
# That is also what happens if this script errors or jq is missing, so a bug here
|
||||||
# fails toward the prompt rather than toward silent execution.
|
# fails toward the prompt rather than toward silent execution.
|
||||||
#
|
#
|
||||||
# Beta is guarded as strictly as prod: it serves beta.thermograph.org AND hosts
|
# Beta is guarded as strictly as prod -- more strictly than the names alone
|
||||||
# Forgejo, so a destructive command there can take out the git server, CI and the
|
# suggest, now: vps2 (169.58.46.181) runs prod AND beta as two Swarm stacks on
|
||||||
# container registry at once.
|
# the SAME manager, sharing the same TimescaleDB instance, so a "beta" command
|
||||||
|
# there has PROD's blast radius (shared Docker host, shared Postgres server --
|
||||||
|
# see infra/deploy/env-topology.sh). vps1 (75.119.132.91) is the other guarded
|
||||||
|
# host: Forgejo (git + CI + registry), Grafana/Loki, and the dev environment --
|
||||||
|
# a destructive command there can take out the git server, CI and the container
|
||||||
|
# registry at once. Mesh IPs did NOT move in the vps1/vps2 split (vps1 is still
|
||||||
|
# 10.10.0.2, vps2 is still 10.10.0.1); what moved is which environment runs where.
|
||||||
set -uo pipefail
|
set -uo pipefail
|
||||||
|
|
||||||
payload=$(cat)
|
payload=$(cat)
|
||||||
|
|
@ -32,7 +38,10 @@ tool=$(printf '%s' "$payload" | jq -r '.tool_name // ""')
|
||||||
# An ssh target naming a live host, matched as a whole token after any `user@`.
|
# An ssh target naming a live host, matched as a whole token after any `user@`.
|
||||||
# Backslashes are doubled because awk -v processes escape sequences in the value
|
# Backslashes are doubled because awk -v processes escape sequences in the value
|
||||||
# before the regex ever sees it; a single \. arrives as a bare dot (and warns).
|
# before the regex ever sees it; a single \. arrives as a bare dot (and warns).
|
||||||
readonly HOST_TOKEN_RE='^(prod|beta|169\\.58\\.46\\.181|75\\.119\\.132\\.91|10\\.10\\.0\\.[12])$'
|
# vps2 (169.58.46.181) carries BOTH prod and beta; vps1 (75.119.132.91) carries
|
||||||
|
# Forgejo/Grafana/dev -- "vps1"/"vps2" are recognised alongside the environment
|
||||||
|
# names and raw IPs so a command naming the host either way still gets caught.
|
||||||
|
readonly HOST_TOKEN_RE='^(prod|beta|vps1|vps2|169\\.58\\.46\\.181|75\\.119\\.132\\.91|10\\.10\\.0\\.[12])$'
|
||||||
|
|
||||||
emit() { # $1=allow|ask|deny $2=reason
|
emit() { # $1=allow|ask|deny $2=reason
|
||||||
jq -n --arg d "$1" --arg r "$2" \
|
jq -n --arg d "$1" --arg r "$2" \
|
||||||
|
|
|
||||||
60
.claude/ownership.md
Normal file
60
.claude/ownership.md
Normal file
|
|
@ -0,0 +1,60 @@
|
||||||
|
# Module map — what can be worked on concurrently
|
||||||
|
|
||||||
|
Used for partitioning work across concurrent subagents. The rule is one owner
|
||||||
|
per file at a time: **never** give two concurrent agents tasks that touch the
|
||||||
|
same module. If two tasks genuinely need the same file, sequence them instead of
|
||||||
|
racing them — a rebase conflict costs more than the parallelism saved.
|
||||||
|
|
||||||
|
Read this before dispatching. It answers one question: *can these two tasks run
|
||||||
|
at the same time?*
|
||||||
|
|
||||||
|
The monorepo's four domains are the natural seam, and they are a real one: FE
|
||||||
|
and BE build and deploy independently, and `deploy.yml` checks per-leg whether a
|
||||||
|
push actually touched that domain. Two agents in two different domains will not
|
||||||
|
collide in CI either.
|
||||||
|
|
||||||
|
## Safe to assign concurrently
|
||||||
|
|
||||||
|
| Module | Files | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| Backend — API | `backend/api/**`, `backend/app.py` | Route handlers and serialisation. |
|
||||||
|
| Backend — core | `backend/core/**` | Grading, percentiles, climatology. The domain logic. |
|
||||||
|
| Backend — accounts | `backend/accounts/**` | Auth, sessions, OAuth links. |
|
||||||
|
| Backend — daemon | `backend/daemon/**` | Scheduled work. Anything named "prefetch" must not spend the Open-Meteo quota. |
|
||||||
|
| Backend — data & lake | `backend/data/**`, `backend/gen_era5_lake.py`, `backend/gen_cities.py` | |
|
||||||
|
| Backend — migrations | `backend/alembic/**`, `backend/alembic.ini` | One owner, always. Two agents generating revisions produce two heads. |
|
||||||
|
| Frontend — server | `frontend/server/**`, `frontend/app.py`, `frontend/api_client.py` | Go SSR and the Python app. |
|
||||||
|
| Frontend — static | `frontend/static/**` | |
|
||||||
|
| Frontend — templates | `frontend/templates/**`, `frontend/content/**`, `frontend/content.py` | |
|
||||||
|
| Infra — deploy | `infra/deploy/**` (excluding `secrets/`), `infra/docker-compose*.yml` | |
|
||||||
|
| Infra — secrets | `infra/deploy/secrets/**` | SOPS vault. One owner, and never alongside `infra/deploy`. |
|
||||||
|
| Infra — ops | `infra/ops/**`, `infra/lake-iceberg/**` | |
|
||||||
|
| Infra — terraform | `infra/terraform/**` | |
|
||||||
|
| Observability | `observability/**` | Grafana dashboards are provisioned from this JSON; a UI edit is overwritten. |
|
||||||
|
| Docs | `docs/**` | |
|
||||||
|
| Agent tooling | `.claude/hooks/**` | |
|
||||||
|
|
||||||
|
## Serialise — shared spine
|
||||||
|
|
||||||
|
Touched by most changes. A change here is its own task with nothing else running
|
||||||
|
against it.
|
||||||
|
|
||||||
|
| File | Why |
|
||||||
|
|---|---|
|
||||||
|
| `.forgejo/workflows/**` | CI lives **only** here, path-filtered per domain. Every domain's changes route through the same files, so two agents editing workflows conflict even when their domains do not. |
|
||||||
|
| `CLAUDE.md` and every domain `CLAUDE.md` | Read before every change; a stale one is a correctness bug. Small edits, landed fast. |
|
||||||
|
| `CUTOVER-NOTES.md` | The source of truth for what is and is not live. |
|
||||||
|
| `infra/deploy/env-topology.sh` | The single source of truth for where each environment's checkout, branch, stack and ports live. |
|
||||||
|
| `infra/deploy/deploy.sh`, `infra/deploy/stack/deploy-stack.sh` | The deploy contract's entry points. |
|
||||||
|
| `backend/CLAUDE.md` + the `/api/version` contract | FE and BE ship out of lockstep, so `PAYLOAD_VER` discipline is load-bearing. A change to the payload shape is one task spanning both domains — not two concurrent ones. |
|
||||||
|
|
||||||
|
## The recurring collisions
|
||||||
|
|
||||||
|
- **A schema change is not a backend-only task.** It is `backend/alembic` plus
|
||||||
|
whatever reads the column, and if the payload shape moves it is `PAYLOAD_VER`
|
||||||
|
and the frontend too. Scope it as one task across both domains rather than
|
||||||
|
two agents discovering each other mid-flight.
|
||||||
|
- **Adding CI for a domain edits the shared workflows.** Sequence it against any
|
||||||
|
other workflow work.
|
||||||
|
- **`infra/deploy/secrets/` and `infra/deploy/` are one owner between them.**
|
||||||
|
The rendered artifact and the thing that renders it move together.
|
||||||
|
|
@ -10,16 +10,32 @@ name: Deploy
|
||||||
# parameterised, so the duplication bought nothing and cost six files to keep in
|
# parameterised, so the duplication bought nothing and cost six files to keep in
|
||||||
# step.
|
# step.
|
||||||
#
|
#
|
||||||
# The two *-deploy-dev.yml workflows are NOT ported. They were documented as
|
# DEV IS A REAL DEPLOY TARGET AGAIN. It previously was not: dev meant a
|
||||||
# inert: they call the monorepo layout at ~/thermograph-dev on the LAN box,
|
# sudo-free compose stack on the operator's desktop, and the two
|
||||||
# which is still a split-era thermograph-infra checkout, so that path does not
|
# *-deploy-dev.yml workflows that drove it were inert. Dev now lives on vps1 at
|
||||||
# exist there. LAN dev is a local `make dev-up` concern, not a CI environment.
|
# /opt/thermograph-dev — a normal fleet host, reached over SSH exactly like beta
|
||||||
|
# and prod — so it gets a leg here.
|
||||||
|
#
|
||||||
|
# SECRETS ARE KEYED BY HOST, NOT BY ENVIRONMENT. `SSH_*` used to mean "beta" and
|
||||||
|
# `PROD_SSH_*` "prod", which worked only while each environment owned a box. It
|
||||||
|
# stopped being true in two directions at once: vps2 now runs beta AND prod, and
|
||||||
|
# the box `SSH_*` pointed at is now vps1, which runs dev, Forgejo and Grafana.
|
||||||
|
# The old names would have made "the beta secret" and "the Forgejo box secret"
|
||||||
|
# the same value by accident — the ops-cron file already records one incident
|
||||||
|
# caused by exactly that conflation. So: VPS1_SSH_* and VPS2_SSH_*, named for
|
||||||
|
# the machine, and the environment is passed separately as THERMOGRAPH_ENV.
|
||||||
|
#
|
||||||
|
# BETA AND PROD DEPLOYING TO ONE BOX DO NOT RACE. They have separate checkouts
|
||||||
|
# (/opt/thermograph-beta and /opt/thermograph), so the `git reset --hard` in one
|
||||||
|
# cannot pull the tree out from under the other, and deploy.sh's flock is per
|
||||||
|
# checkout. The concurrency groups below stay keyed by ref+service, which keeps
|
||||||
|
# two pushes to the SAME branch serialised — the property that actually matters.
|
||||||
#
|
#
|
||||||
# DELIBERATELY BORING EXPRESSIONS. No dynamic matrix (fromJSON), no
|
# DELIBERATELY BORING EXPRESSIONS. No dynamic matrix (fromJSON), no
|
||||||
# `cond && secrets.A || secrets.B` ternary. Those are GitHub idioms that a
|
# `cond && secrets.A || secrets.B` ternary. Those are GitHub idioms that a
|
||||||
# Forgejo/act runner may evaluate differently, and the failure mode here is
|
# Forgejo/act runner may evaluate differently, and the failure mode here is
|
||||||
# "production does not deploy" or, worse, "deploys with an empty SSH host". The
|
# "production does not deploy" or, worse, "deploys with an empty SSH host". The
|
||||||
# two environments therefore get two explicit, mutually exclusive steps.
|
# three environments therefore get three explicit, mutually exclusive steps.
|
||||||
#
|
#
|
||||||
# What is preserved from the originals, all of it load-bearing:
|
# What is preserved from the originals, all of it load-bearing:
|
||||||
# - fetch-depth: 0, because the image tag is keyed to the LAST COMMIT THAT
|
# - fetch-depth: 0, because the image tag is keyed to the LAST COMMIT THAT
|
||||||
|
|
@ -29,14 +45,18 @@ name: Deploy
|
||||||
# - The 12-hex truncation, matching build-push exactly.
|
# - The 12-hex truncation, matching build-push exactly.
|
||||||
# - Per-service, per-environment concurrency with cancel-in-progress: false --
|
# - Per-service, per-environment concurrency with cancel-in-progress: false --
|
||||||
# a half-finished deploy must never be cancelled by a newer one.
|
# a half-finished deploy must never be cancelled by a newer one.
|
||||||
# - Separate PROD_SSH_* credentials, so a beta credential leak cannot reach
|
# - Separate credentials per host. Note what this does and does not buy now:
|
||||||
# prod.
|
# it keeps vps1 (Forgejo, its CI, and whatever unreviewed branch dev is
|
||||||
|
# running) away from vps2 entirely. It no longer puts a host boundary
|
||||||
|
# between beta and prod, because they share vps2 by design — that boundary
|
||||||
|
# is now at the database (separate roles and databases) and the filesystem
|
||||||
|
# (separate checkouts and rendered env files).
|
||||||
# - appleboy/ssh-action by full URL; it is not mirrored in Forgejo's default
|
# - appleboy/ssh-action by full URL; it is not mirrored in Forgejo's default
|
||||||
# action registry.
|
# action registry.
|
||||||
|
|
||||||
on:
|
on:
|
||||||
push:
|
push:
|
||||||
branches: [main, release]
|
branches: [dev, main, release]
|
||||||
paths:
|
paths:
|
||||||
- 'backend/**'
|
- 'backend/**'
|
||||||
- 'frontend/**'
|
- 'frontend/**'
|
||||||
|
|
@ -71,10 +91,15 @@ jobs:
|
||||||
run: |
|
run: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
# Branch selects the environment. main -> beta, release -> prod.
|
# Branch selects the environment; the environment selects the host and
|
||||||
|
# the script path. dev -> dev (vps1), main -> beta (vps2),
|
||||||
|
# release -> prod (vps2). The paths differ because each environment
|
||||||
|
# has its own checkout — two environments on vps2 must never share
|
||||||
|
# one — and dev has its own entry point for its secrets policy.
|
||||||
case "${{ github.ref_name }}" in
|
case "${{ github.ref_name }}" in
|
||||||
main) environment=beta ;;
|
dev) environment=dev ; script=/opt/thermograph-dev/infra/deploy/deploy-dev.sh ;;
|
||||||
release) environment=prod ;;
|
main) environment=beta ; script=/opt/thermograph-beta/infra/deploy/deploy.sh ;;
|
||||||
|
release) environment=prod ; script=/opt/thermograph/infra/deploy/deploy.sh ;;
|
||||||
*) echo "::error::Deploy triggered on unexpected ref '${{ github.ref_name }}'"; exit 1 ;;
|
*) echo "::error::Deploy triggered on unexpected ref '${{ github.ref_name }}'"; exit 1 ;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
|
|
@ -105,6 +130,7 @@ jobs:
|
||||||
|
|
||||||
{
|
{
|
||||||
echo "environment=$environment"
|
echo "environment=$environment"
|
||||||
|
echo "script=$script"
|
||||||
echo "changed=$changed"
|
echo "changed=$changed"
|
||||||
echo "tag=$tag"
|
echo "tag=$tag"
|
||||||
} >> "$GITHUB_OUTPUT"
|
} >> "$GITHUB_OUTPUT"
|
||||||
|
|
@ -126,38 +152,58 @@ jobs:
|
||||||
|
|
||||||
echo "==> ${{ matrix.service }} -> $environment | changed=$changed | tag=$tag"
|
echo "==> ${{ matrix.service }} -> $environment | changed=$changed | tag=$tag"
|
||||||
|
|
||||||
- name: Deploy to beta
|
- name: Deploy to dev (vps1)
|
||||||
|
if: steps.plan.outputs.changed == 'true' && steps.plan.outputs.environment == 'dev'
|
||||||
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
||||||
|
with:
|
||||||
|
host: ${{ secrets.VPS1_SSH_HOST }}
|
||||||
|
username: ${{ secrets.VPS1_SSH_USER }}
|
||||||
|
key: ${{ secrets.VPS1_SSH_KEY }}
|
||||||
|
port: ${{ secrets.VPS1_SSH_PORT }}
|
||||||
|
envs: SERVICE,BACKEND_IMAGE_TAG,FRONTEND_IMAGE_TAG,THERMOGRAPH_ENV
|
||||||
|
script: ${{ steps.plan.outputs.script }}
|
||||||
|
env:
|
||||||
|
SERVICE: ${{ matrix.service }}
|
||||||
|
THERMOGRAPH_ENV: dev
|
||||||
|
BACKEND_IMAGE_TAG: ${{ matrix.service == 'backend' && steps.plan.outputs.tag || '' }}
|
||||||
|
FRONTEND_IMAGE_TAG: ${{ matrix.service == 'frontend' && steps.plan.outputs.tag || '' }}
|
||||||
|
|
||||||
|
- name: Deploy to beta (vps2)
|
||||||
if: steps.plan.outputs.changed == 'true' && steps.plan.outputs.environment == 'beta'
|
if: steps.plan.outputs.changed == 'true' && steps.plan.outputs.environment == 'beta'
|
||||||
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
||||||
with:
|
with:
|
||||||
host: ${{ secrets.SSH_HOST }}
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
||||||
username: ${{ secrets.SSH_USER }}
|
username: ${{ secrets.VPS2_SSH_USER }}
|
||||||
key: ${{ secrets.SSH_KEY }}
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
||||||
port: ${{ secrets.SSH_PORT }}
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
||||||
envs: SERVICE,BACKEND_IMAGE_TAG,FRONTEND_IMAGE_TAG
|
envs: SERVICE,BACKEND_IMAGE_TAG,FRONTEND_IMAGE_TAG,THERMOGRAPH_ENV
|
||||||
script: /opt/thermograph/infra/deploy/deploy.sh
|
script: ${{ steps.plan.outputs.script }}
|
||||||
env:
|
env:
|
||||||
SERVICE: ${{ matrix.service }}
|
SERVICE: ${{ matrix.service }}
|
||||||
|
# THERMOGRAPH_ENV is what makes this a BETA deploy rather than a prod
|
||||||
|
# one: same host, same credentials, same script — the environment is
|
||||||
|
# the only thing that differs, and deploy.sh refuses to run if it
|
||||||
|
# disagrees with the checkout it was invoked from.
|
||||||
|
THERMOGRAPH_ENV: beta
|
||||||
# Only the matching one is read by deploy.sh for a single-service roll;
|
# Only the matching one is read by deploy.sh for a single-service roll;
|
||||||
# the other stays empty and the persisted .image-tags.env supplies the
|
# the other stays empty and the persisted .image-tags.env supplies the
|
||||||
# sibling's live tag, so this roll never disturbs it.
|
# sibling's live tag, so this roll never disturbs it.
|
||||||
BACKEND_IMAGE_TAG: ${{ matrix.service == 'backend' && steps.plan.outputs.tag || '' }}
|
BACKEND_IMAGE_TAG: ${{ matrix.service == 'backend' && steps.plan.outputs.tag || '' }}
|
||||||
FRONTEND_IMAGE_TAG: ${{ matrix.service == 'frontend' && steps.plan.outputs.tag || '' }}
|
FRONTEND_IMAGE_TAG: ${{ matrix.service == 'frontend' && steps.plan.outputs.tag || '' }}
|
||||||
|
|
||||||
- name: Deploy to prod
|
- name: Deploy to prod (vps2)
|
||||||
if: steps.plan.outputs.changed == 'true' && steps.plan.outputs.environment == 'prod'
|
if: steps.plan.outputs.changed == 'true' && steps.plan.outputs.environment == 'prod'
|
||||||
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
||||||
with:
|
with:
|
||||||
# A completely separate secret set from beta's, deliberately: a beta
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
||||||
# credential leak must not reach prod.
|
username: ${{ secrets.VPS2_SSH_USER }}
|
||||||
host: ${{ secrets.PROD_SSH_HOST }}
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
||||||
username: ${{ secrets.PROD_SSH_USER }}
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
||||||
key: ${{ secrets.PROD_SSH_KEY }}
|
envs: SERVICE,BACKEND_IMAGE_TAG,FRONTEND_IMAGE_TAG,THERMOGRAPH_ENV
|
||||||
port: ${{ secrets.PROD_SSH_PORT }}
|
script: ${{ steps.plan.outputs.script }}
|
||||||
envs: SERVICE,BACKEND_IMAGE_TAG,FRONTEND_IMAGE_TAG
|
|
||||||
script: /opt/thermograph/infra/deploy/deploy.sh
|
|
||||||
env:
|
env:
|
||||||
SERVICE: ${{ matrix.service }}
|
SERVICE: ${{ matrix.service }}
|
||||||
|
THERMOGRAPH_ENV: prod
|
||||||
BACKEND_IMAGE_TAG: ${{ matrix.service == 'backend' && steps.plan.outputs.tag || '' }}
|
BACKEND_IMAGE_TAG: ${{ matrix.service == 'backend' && steps.plan.outputs.tag || '' }}
|
||||||
FRONTEND_IMAGE_TAG: ${{ matrix.service == 'frontend' && steps.plan.outputs.tag || '' }}
|
FRONTEND_IMAGE_TAG: ${{ matrix.service == 'frontend' && steps.plan.outputs.tag || '' }}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -2,65 +2,137 @@ name: Sync infra to hosts
|
||||||
|
|
||||||
# The infra DOMAIN's own pipeline (new with the monorepo -- the split era had
|
# The infra DOMAIN's own pipeline (new with the monorepo -- the split era had
|
||||||
# no infra deploy workflow at all; infra changes rode along lazily with the
|
# no infra deploy workflow at all; infra changes rode along lazily with the
|
||||||
# next app deploy's `git reset`). On a push to `main` touching infra/**, SSH
|
# next app deploy's `git reset`). On a push touching infra/**, SSH to every
|
||||||
# to beta AND prod, fast-forward the host's /opt/thermograph monorepo checkout
|
# host, fast-forward each ENVIRONMENT's monorepo checkout and re-render that
|
||||||
# and re-render /etc/thermograph.env from the SOPS vault -- so a compose edit
|
# environment's env file from the SOPS vault -- so a compose edit or a secret
|
||||||
# or a secret rotation lands push-button instead of waiting for the next app
|
# rotation lands push-button instead of waiting for the next app deploy.
|
||||||
# deploy.
|
|
||||||
#
|
#
|
||||||
# Deliberately does NOT roll any service: image tags are the app domains'
|
# Deliberately does NOT roll any service: image tags are the app domains'
|
||||||
# axis, not infra's. A compose change that must recreate containers takes
|
# axis, not infra's. A compose or stack change that must recreate containers
|
||||||
# effect on the next app deploy, or a by-hand
|
# takes effect on the next app deploy, or a by-hand
|
||||||
# `SERVICE=all ... infra/deploy/deploy.sh` run (see that script's header).
|
# `SERVICE=all ... infra/deploy/deploy.sh` run (see that script's header).
|
||||||
#
|
#
|
||||||
# Both hosts track infra via `main` (the split-era model, unchanged): prod's
|
# THREE CHECKOUTS, TWO HOSTS. This is the shape the vps1/vps2 split forces:
|
||||||
# *app images* are staged by the `release` branch, but its checkout follows
|
#
|
||||||
# main -- deploy.sh's own BRANCH default encodes the same thing.
|
# vps1 /opt/thermograph-dev branch dev -> /etc/thermograph.env
|
||||||
|
# vps2 /opt/thermograph-beta branch main -> /etc/thermograph-beta.env
|
||||||
|
# vps2 /opt/thermograph branch main -> /etc/thermograph.env
|
||||||
|
#
|
||||||
|
# The two checkouts on vps2 are separate directories on purpose: a `git reset
|
||||||
|
# --hard` for beta must not be able to move prod's tree, and each environment
|
||||||
|
# renders its own env file from its own vault file. Because the paths differ,
|
||||||
|
# the beta and prod jobs below can safely run at the same time on one box.
|
||||||
|
#
|
||||||
|
# Beta and prod both track `main` (infra is not environment-staged -- only app
|
||||||
|
# IMAGES are, via tags). Dev tracks `dev`, which is why this workflow now
|
||||||
|
# triggers on both branches: a dev-branch infra change has to reach the dev
|
||||||
|
# checkout, and only that one.
|
||||||
|
#
|
||||||
|
# Each `render_thermograph_secrets` call passes the environment name and the
|
||||||
|
# output path EXPLICITLY. It used to rely on the host's
|
||||||
|
# /etc/thermograph/secrets-env marker, which cannot answer the question on vps2
|
||||||
|
# -- one marker, two environments. Passing them here means a beta sync can
|
||||||
|
# never render prod's vault, and vice versa.
|
||||||
|
|
||||||
on:
|
on:
|
||||||
push:
|
push:
|
||||||
branches: [main]
|
branches: [dev, main]
|
||||||
paths: ['infra/**']
|
paths: ['infra/**']
|
||||||
workflow_dispatch: {}
|
workflow_dispatch: {}
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
|
sync-dev:
|
||||||
|
# Only the dev branch feeds the dev checkout.
|
||||||
|
if: github.ref_name == 'dev'
|
||||||
|
runs-on: docker
|
||||||
|
concurrency:
|
||||||
|
group: infra-sync-dev
|
||||||
|
cancel-in-progress: false
|
||||||
|
steps:
|
||||||
|
- name: Sync dev checkout on vps1 + re-render secrets
|
||||||
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
||||||
|
with:
|
||||||
|
host: ${{ secrets.VPS1_SSH_HOST }}
|
||||||
|
username: ${{ secrets.VPS1_SSH_USER }}
|
||||||
|
key: ${{ secrets.VPS1_SSH_KEY }}
|
||||||
|
port: ${{ secrets.VPS1_SSH_PORT }}
|
||||||
|
script: |
|
||||||
|
set -euo pipefail
|
||||||
|
cd /opt/thermograph-dev
|
||||||
|
git fetch --prune origin dev
|
||||||
|
git reset --hard origin/dev
|
||||||
|
echo "synced to $(git log --oneline -1)"
|
||||||
|
|
||||||
|
# RENDER ONLY IF THIS HOST IS ACTUALLY DEV.
|
||||||
|
#
|
||||||
|
# dev's env file is /etc/thermograph.env — the same path beta uses,
|
||||||
|
# and during the vps1/vps2 cutover beta is STILL ON VPS1. Rendering
|
||||||
|
# unconditionally would overwrite the live env file of the
|
||||||
|
# environment currently serving beta.thermograph.org with dev's
|
||||||
|
# credentials: beta's running containers would survive (env_file is
|
||||||
|
# read at container start) and then come up wrong on its next
|
||||||
|
# deploy, pointed at dev's database password.
|
||||||
|
#
|
||||||
|
# The host marker is the box's own statement of which environment it
|
||||||
|
# is, and it is the right authority for "is it safe to write this
|
||||||
|
# file here". It flips to `dev` in provision-dev.sh, which is run
|
||||||
|
# only after beta has vacated. Until then this skips loudly rather
|
||||||
|
# than failing: the checkout sync above is still useful and correct.
|
||||||
|
marker=$(cat /etc/thermograph/secrets-env 2>/dev/null || true)
|
||||||
|
if [ "$marker" != dev ]; then
|
||||||
|
echo "!! /etc/thermograph/secrets-env says '${marker:-<unset>}', not 'dev'."
|
||||||
|
echo "!! Skipping the secrets render — this host is not dev yet, and"
|
||||||
|
echo "!! /etc/thermograph.env here belongs to '${marker:-another environment}'."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
||||||
|
. infra/deploy/render-secrets.sh
|
||||||
|
# SKIP_COMMON is dev's standing rule, not a preference: common.yaml
|
||||||
|
# is the fleet's shared production credential set, and vps1 also
|
||||||
|
# runs Forgejo and its CI. See render-secrets.sh.
|
||||||
|
THERMOGRAPH_SECRETS_SKIP_COMMON=1 \
|
||||||
|
render_thermograph_secrets /opt/thermograph-dev/infra dev /etc/thermograph.env
|
||||||
|
fi
|
||||||
|
|
||||||
sync-beta:
|
sync-beta:
|
||||||
|
if: github.ref_name == 'main'
|
||||||
runs-on: docker
|
runs-on: docker
|
||||||
concurrency:
|
concurrency:
|
||||||
group: infra-sync-beta
|
group: infra-sync-beta
|
||||||
cancel-in-progress: false
|
cancel-in-progress: false
|
||||||
steps:
|
steps:
|
||||||
- name: Sync beta checkout + re-render secrets
|
- name: Sync beta checkout on vps2 + re-render secrets
|
||||||
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
||||||
with:
|
with:
|
||||||
host: ${{ secrets.SSH_HOST }}
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
||||||
username: ${{ secrets.SSH_USER }}
|
username: ${{ secrets.VPS2_SSH_USER }}
|
||||||
key: ${{ secrets.SSH_KEY }}
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
||||||
port: ${{ secrets.SSH_PORT }}
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
||||||
script: |
|
script: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
cd /opt/thermograph
|
cd /opt/thermograph-beta
|
||||||
git fetch --prune origin main
|
git fetch --prune origin main
|
||||||
git reset --hard origin/main
|
git reset --hard origin/main
|
||||||
if [ -f infra/deploy/render-secrets.sh ]; then
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
||||||
. infra/deploy/render-secrets.sh
|
. infra/deploy/render-secrets.sh
|
||||||
render_thermograph_secrets /opt/thermograph/infra
|
render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
|
||||||
fi
|
fi
|
||||||
echo "synced to $(git log --oneline -1)"
|
echo "synced to $(git log --oneline -1)"
|
||||||
|
|
||||||
sync-prod:
|
sync-prod:
|
||||||
|
if: github.ref_name == 'main'
|
||||||
runs-on: docker
|
runs-on: docker
|
||||||
concurrency:
|
concurrency:
|
||||||
group: infra-sync-prod
|
group: infra-sync-prod
|
||||||
cancel-in-progress: false
|
cancel-in-progress: false
|
||||||
steps:
|
steps:
|
||||||
- name: Sync prod checkout + re-render secrets
|
- name: Sync prod checkout on vps2 + re-render secrets
|
||||||
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
||||||
with:
|
with:
|
||||||
host: ${{ secrets.PROD_SSH_HOST }}
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
||||||
username: ${{ secrets.PROD_SSH_USER }}
|
username: ${{ secrets.VPS2_SSH_USER }}
|
||||||
key: ${{ secrets.PROD_SSH_KEY }}
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
||||||
port: ${{ secrets.PROD_SSH_PORT }}
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
||||||
script: |
|
script: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
cd /opt/thermograph
|
cd /opt/thermograph
|
||||||
|
|
@ -68,6 +140,6 @@ jobs:
|
||||||
git reset --hard origin/main
|
git reset --hard origin/main
|
||||||
if [ -f infra/deploy/render-secrets.sh ]; then
|
if [ -f infra/deploy/render-secrets.sh ]; then
|
||||||
. infra/deploy/render-secrets.sh
|
. infra/deploy/render-secrets.sh
|
||||||
render_thermograph_secrets /opt/thermograph/infra
|
render_thermograph_secrets /opt/thermograph/infra prod /etc/thermograph.env
|
||||||
fi
|
fi
|
||||||
echo "synced to $(git log --oneline -1)"
|
echo "synced to $(git log --oneline -1)"
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
name: Validate observability stack
|
name: Validate observability stack
|
||||||
|
|
||||||
# The observability DOMAIN deploys by hand (`docker compose up -d` on beta +
|
# The observability DOMAIN deploys by hand (`docker compose up -d` on vps1 —
|
||||||
|
# the monitoring host, not the beta environment, which runs on vps2 now — plus
|
||||||
# the Alloy agent on each node) with no build step, so nothing caught a
|
# the Alloy agent on each node) with no build step, so nothing caught a
|
||||||
# malformed compose file, a broken dashboard JSON, or an unparseable config
|
# malformed compose file, a broken dashboard JSON, or an unparseable config
|
||||||
# until it failed live on the host. This is that missing guard: it parses
|
# until it failed live on the host. This is that missing guard: it parses
|
||||||
|
|
|
||||||
|
|
@ -6,19 +6,32 @@ name: Ops cron (backup + IndexNow)
|
||||||
# classification table in
|
# classification table in
|
||||||
# thermograph-docs/architecture/repo-topology-and-infrastructure.md §7.
|
# thermograph-docs/architecture/repo-topology-and-infrastructure.md §7.
|
||||||
#
|
#
|
||||||
# Both jobs SSH into the prod host and run inside the already-running compose
|
# The jobs SSH into a host and run inside the already-running stack (docker
|
||||||
# stack (docker compose exec), the same way deploy.sh already runs its own
|
# exec), the same way deploy.sh runs its own post-deploy IndexNow ping -- no new
|
||||||
# post-deploy IndexNow ping -- no new network exposure, no separate dependency
|
# network exposure, no separate dependency install. Runs on the `docker` label
|
||||||
# install. Runs on the `docker` label (the always-on Swarm-hosted runner
|
# (the always-on Swarm-hosted runner deploy/forgejo/ stood up).
|
||||||
# deploy/forgejo/ stood up).
|
|
||||||
#
|
#
|
||||||
# Targets PROD via the PROD_SSH_* secrets (prod = 169.58.46.181, `agent` user,
|
# WHICH BOX EACH JOB TALKS TO, and why the secret names say so:
|
||||||
# in the docker group so no sudo needed; /etc/thermograph.env is agent-readable).
|
#
|
||||||
# These are the same secrets deploy-prod.yml uses for the release->prod deploy --
|
# backup, indexnow -> VPS2_SSH_* (vps2 = 169.58.46.181: prod AND beta, the
|
||||||
# NOT the SSH_* secrets, which point at BETA (deploy.yml's `main`->beta path). An
|
# shared TimescaleDB, Postfix, the backups)
|
||||||
# earlier revision reused SSH_* here, so the "prod" backup was silently dumping
|
# forgejo-backup -> VPS1_SSH_* (vps1 = 75.119.132.91: Forgejo, Grafana,
|
||||||
# beta; prod itself had no backup at all. The prod database is the one that must
|
# Loki, and the dev environment)
|
||||||
# be backed up, so both jobs use PROD_SSH_*.
|
#
|
||||||
|
# The secrets are keyed by HOST rather than by environment because an
|
||||||
|
# environment no longer implies a machine: vps2 runs two of them. The previous
|
||||||
|
# names encoded the opposite assumption and had already caused one incident --
|
||||||
|
# an early revision used SSH_* here, which meant "beta", so the job labelled
|
||||||
|
# "prod backup" was silently dumping beta while prod had no backup at all. The
|
||||||
|
# same conflation is why Forgejo's backup used to be described as running "on
|
||||||
|
# beta": beta and Forgejo happened to share a box, so one secret served both
|
||||||
|
# meanings. It does not any more.
|
||||||
|
#
|
||||||
|
# BOTH APPLICATION DATABASES ARE BACKED UP. prod and beta are separate databases
|
||||||
|
# (`thermograph` and `thermograph_beta`) on one shared instance. Dumping only
|
||||||
|
# the prod database would recreate the original failure in a new shape: a job
|
||||||
|
# that looks like "the backup" while one environment's data is silently
|
||||||
|
# uncovered. The loop below dumps each, to its own off-box prefix.
|
||||||
|
|
||||||
# Monorepo port: the compose file now lives under infra/ of the host's
|
# Monorepo port: the compose file now lives under infra/ of the host's
|
||||||
# /opt/thermograph monorepo checkout, so every docker-compose invocation cd's
|
# /opt/thermograph monorepo checkout, so every docker-compose invocation cd's
|
||||||
|
|
@ -44,22 +57,22 @@ jobs:
|
||||||
group: ops-backup
|
group: ops-backup
|
||||||
cancel-in-progress: false
|
cancel-in-progress: false
|
||||||
steps:
|
steps:
|
||||||
- name: Dump the prod database over SSH
|
- name: Dump both application databases over SSH
|
||||||
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
||||||
# S3 creds for the encrypted off-box copy to Contabo Object Storage, passed
|
# S3 creds for the encrypted off-box copy to Contabo Object Storage, passed
|
||||||
# into the remote script via `envs:` (the host env file isn't a reliable
|
# into the remote script via `envs:` (a host env file isn't a reliable
|
||||||
# source across boxes -- beta's /etc/thermograph.env isn't readable by the
|
# source across boxes -- on vps1 it isn't readable by the deploy user --
|
||||||
# deploy user -- so the CI secret is the uniform home, like PROD_SSH_*).
|
# so the CI secret is the uniform home, like the VPS*_SSH_* pairs).
|
||||||
env:
|
env:
|
||||||
S3_ENDPOINT: ${{ secrets.S3_ENDPOINT }}
|
S3_ENDPOINT: ${{ secrets.S3_ENDPOINT }}
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
||||||
S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}
|
S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}
|
||||||
S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}
|
S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}
|
||||||
with:
|
with:
|
||||||
host: ${{ secrets.PROD_SSH_HOST }}
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
||||||
username: ${{ secrets.PROD_SSH_USER }}
|
username: ${{ secrets.VPS2_SSH_USER }}
|
||||||
key: ${{ secrets.PROD_SSH_KEY }}
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
||||||
port: ${{ secrets.PROD_SSH_PORT }}
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
||||||
envs: S3_ENDPOINT,S3_BUCKET,S3_ACCESS_KEY,S3_SECRET_KEY
|
envs: S3_ENDPOINT,S3_BUCKET,S3_ACCESS_KEY,S3_SECRET_KEY
|
||||||
script: |
|
script: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
@ -73,23 +86,45 @@ jobs:
|
||||||
backup_dir="$HOME/thermograph-backups"
|
backup_dir="$HOME/thermograph-backups"
|
||||||
mkdir -p "$backup_dir"
|
mkdir -p "$backup_dir"
|
||||||
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||||
out="$backup_dir/thermograph-$stamp.dump"
|
# The db may run under plain compose OR as a Swarm stack task;
|
||||||
# The db may run under plain compose OR as a Swarm stack task
|
# resolve the container either way so the backup survives a
|
||||||
# (prod post-cutover); resolve the container either way so the
|
# deploy-mode switch. One instance now serves both environments.
|
||||||
# backup survives the deploy-mode switch.
|
|
||||||
dbc=$(docker ps -q --filter "label=com.docker.swarm.service.name=thermograph_db" | head -1)
|
dbc=$(docker ps -q --filter "label=com.docker.swarm.service.name=thermograph_db" | head -1)
|
||||||
[ -z "$dbc" ] && dbc=$(cd /opt/thermograph/infra && docker compose ps -q db 2>/dev/null | head -1)
|
[ -z "$dbc" ] && dbc=$(cd /opt/thermograph/infra && docker compose ps -q db 2>/dev/null | head -1)
|
||||||
[ -n "$dbc" ] || { echo "!! no db container found (compose or stack)"; exit 1; }
|
[ -n "$dbc" ] || { echo "!! no db container found (compose or stack)"; exit 1; }
|
||||||
# Write to a .partial and rename on success so a mid-dump failure can
|
# env:database pairs. Both are dumped every night: they are separate
|
||||||
# never leave a truncated file that looks like a good backup.
|
# databases on one instance, and backing up only one would leave the
|
||||||
docker exec "$dbc" pg_dump -U thermograph -d thermograph \
|
# other silently uncovered.
|
||||||
--format=custom > "$out.partial"
|
for pair in prod:thermograph beta:thermograph_beta; do
|
||||||
mv "$out.partial" "$out"
|
envname="${pair%%:*}"; dbname="${pair##*:}"
|
||||||
echo "wrote $out ($(du -h "$out" | cut -f1))"
|
# A missing database is a HARD failure, never a quiet skip -- a
|
||||||
|
# backup job that shrugs off an absent database is exactly how an
|
||||||
|
# environment ends up with no backups and nobody noticing.
|
||||||
|
if ! docker exec "$dbc" psql -U thermograph -d postgres -tAc \
|
||||||
|
"select 1 from pg_database where datname='$dbname'" | grep -q 1; then
|
||||||
|
echo "!! database '$dbname' ($envname) does not exist on this instance."
|
||||||
|
if [ "$envname" = prod ]; then
|
||||||
|
echo "!! That is prod's own database on prod's own instance — this is not a"
|
||||||
|
echo "!! provisioning gap, something is badly wrong. Do not 'fix' it by creating"
|
||||||
|
echo "!! an empty database; find out where the real one went."
|
||||||
|
else
|
||||||
|
echo "!! provision it first: sudo bash /opt/thermograph-beta/infra/deploy/db/provision-env-db.sh $envname"
|
||||||
|
fi
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
out="$backup_dir/$dbname-$stamp.dump"
|
||||||
|
# Write to a .partial and rename on success so a mid-dump failure can
|
||||||
|
# never leave a truncated file that looks like a good backup.
|
||||||
|
docker exec "$dbc" pg_dump -U thermograph -d "$dbname" \
|
||||||
|
--format=custom > "$out.partial"
|
||||||
|
mv "$out.partial" "$out"
|
||||||
|
echo "wrote $out ($(du -h "$out" | cut -f1))"
|
||||||
|
dumps="${dumps:-} $envname:$out"
|
||||||
|
done
|
||||||
# The dumps are the disaster-recovery copy, not a versioned
|
# The dumps are the disaster-recovery copy, not a versioned
|
||||||
# archive -- keep the last 14 days and let the rest age out.
|
# archive -- keep the last 14 days and let the rest age out.
|
||||||
find "$backup_dir" -name 'thermograph-*.dump' -mtime +14 -delete
|
find "$backup_dir" -name '*.dump' -mtime +14 -delete
|
||||||
find "$backup_dir" -name 'thermograph-*.dump.partial' -mtime +1 -delete
|
find "$backup_dir" -name '*.dump.partial' -mtime +1 -delete
|
||||||
# --- off-box copy to Contabo Object Storage (age-encrypted) ---
|
# --- off-box copy to Contabo Object Storage (age-encrypted) ---
|
||||||
# Streams the just-written dump through age (to the vault's age
|
# Streams the just-written dump through age (to the vault's age
|
||||||
# recipient -- the same key each host renders secrets with, so
|
# recipient -- the same key each host renders secrets with, so
|
||||||
|
|
@ -105,23 +140,33 @@ jobs:
|
||||||
RCLONE_CONFIG_ARCHIVE_SECRET_ACCESS_KEY="$S3_SECRET_KEY" \
|
RCLONE_CONFIG_ARCHIVE_SECRET_ACCESS_KEY="$S3_SECRET_KEY" \
|
||||||
RCLONE_CONFIG_ARCHIVE_REGION=default RCLONE_CONFIG_ARCHIVE_FORCE_PATH_STYLE=true
|
RCLONE_CONFIG_ARCHIVE_REGION=default RCLONE_CONFIG_ARCHIVE_FORCE_PATH_STYLE=true
|
||||||
recip=age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
|
recip=age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
|
||||||
s3base="$S3_BUCKET/backups/db/prod"
|
# One prefix per environment, so a restore never has to guess which
|
||||||
age -r "$recip" < "$out" | rclone rcat "archive:$s3base/$(basename "$out").age"
|
# database a dump came from: backups/db/prod/ and backups/db/beta/.
|
||||||
echo "off-box: uploaded $(basename "$out").age to $s3base"
|
for entry in $dumps; do
|
||||||
rclone delete --min-age 30d "archive:$s3base/" 2>/dev/null || true
|
envname="${entry%%:*}"; f="${entry##*:}"
|
||||||
|
s3base="$S3_BUCKET/backups/db/$envname"
|
||||||
|
age -r "$recip" < "$f" | rclone rcat "archive:$s3base/$(basename "$f").age"
|
||||||
|
echo "off-box: uploaded $(basename "$f").age to $s3base"
|
||||||
|
rclone delete --min-age 30d "archive:$s3base/" 2>/dev/null || true
|
||||||
|
done
|
||||||
else
|
else
|
||||||
echo "!! S3_* secrets unset -- skipped off-box push (add them as repo secrets)"
|
echo "!! S3_* secrets unset -- skipped off-box push (add them as repo secrets)"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
forgejo-backup:
|
forgejo-backup:
|
||||||
name: Forgejo backup (beta) -> S3
|
name: Forgejo backup (vps1) -> S3
|
||||||
runs-on: docker
|
runs-on: docker
|
||||||
# Forgejo (the git host + all CI history) lives on beta and had NO backup at
|
# Forgejo (the git host + all CI history) lives on vps1 and had NO backup at
|
||||||
# all. Dumps its Postgres db (consistent) + tars its data volume (repos, LFS,
|
# all. Dumps its Postgres db (consistent) + tars its data volume (repos, LFS,
|
||||||
# config, avatars), age-encrypts both, pushes off-box, keeps 30 days. Runs as
|
# config, avatars), age-encrypts both, pushes off-box, keeps 30 days. Runs as
|
||||||
# the beta SSH user (SSH_*), which is in the docker group -- so docker exec/run
|
# vps1's SSH user (VPS1_SSH_*), which is in the docker group -- so docker
|
||||||
# need no host sudo, and the data tar runs inside a throwaway container (the
|
# exec/run need no host sudo, and the data tar runs inside a throwaway
|
||||||
# volume dir is root-owned on the host). rclone + age must be present on beta.
|
# container (the volume dir is root-owned on the host). rclone + age must be
|
||||||
|
# present on vps1.
|
||||||
|
#
|
||||||
|
# This job follows FORGEJO, not beta. It used to use the same secret as the
|
||||||
|
# beta deploy purely because Forgejo and beta shared a box; beta has since
|
||||||
|
# moved to vps2 and Forgejo has not moved at all.
|
||||||
concurrency:
|
concurrency:
|
||||||
group: ops-forgejo-backup
|
group: ops-forgejo-backup
|
||||||
cancel-in-progress: false
|
cancel-in-progress: false
|
||||||
|
|
@ -134,16 +179,16 @@ jobs:
|
||||||
S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}
|
S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}
|
||||||
S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}
|
S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}
|
||||||
with:
|
with:
|
||||||
host: ${{ secrets.SSH_HOST }}
|
host: ${{ secrets.VPS1_SSH_HOST }}
|
||||||
username: ${{ secrets.SSH_USER }}
|
username: ${{ secrets.VPS1_SSH_USER }}
|
||||||
key: ${{ secrets.SSH_KEY }}
|
key: ${{ secrets.VPS1_SSH_KEY }}
|
||||||
port: ${{ secrets.SSH_PORT }}
|
port: ${{ secrets.VPS1_SSH_PORT }}
|
||||||
envs: S3_ENDPOINT,S3_BUCKET,S3_ACCESS_KEY,S3_SECRET_KEY
|
envs: S3_ENDPOINT,S3_BUCKET,S3_ACCESS_KEY,S3_SECRET_KEY
|
||||||
script: |
|
script: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
[ -n "${S3_ACCESS_KEY:-}" ] || { echo "!! S3_* secrets unset"; exit 1; }
|
[ -n "${S3_ACCESS_KEY:-}" ] || { echo "!! S3_* secrets unset"; exit 1; }
|
||||||
command -v rclone >/dev/null 2>&1 || { echo "!! rclone missing on beta"; exit 1; }
|
command -v rclone >/dev/null 2>&1 || { echo "!! rclone missing on vps1"; exit 1; }
|
||||||
command -v age >/dev/null 2>&1 || { echo "!! age missing on beta"; exit 1; }
|
command -v age >/dev/null 2>&1 || { echo "!! age missing on vps1"; exit 1; }
|
||||||
export RCLONE_CONFIG_ARCHIVE_TYPE=s3 RCLONE_CONFIG_ARCHIVE_PROVIDER=Other \
|
export RCLONE_CONFIG_ARCHIVE_TYPE=s3 RCLONE_CONFIG_ARCHIVE_PROVIDER=Other \
|
||||||
RCLONE_CONFIG_ARCHIVE_ENDPOINT="$S3_ENDPOINT" \
|
RCLONE_CONFIG_ARCHIVE_ENDPOINT="$S3_ENDPOINT" \
|
||||||
RCLONE_CONFIG_ARCHIVE_ACCESS_KEY_ID="$S3_ACCESS_KEY" \
|
RCLONE_CONFIG_ARCHIVE_ACCESS_KEY_ID="$S3_ACCESS_KEY" \
|
||||||
|
|
@ -177,10 +222,10 @@ jobs:
|
||||||
- name: Ping IndexNow if the URL set changed
|
- name: Ping IndexNow if the URL set changed
|
||||||
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
uses: https://github.com/appleboy/ssh-action@v1.2.0
|
||||||
with:
|
with:
|
||||||
host: ${{ secrets.PROD_SSH_HOST }}
|
host: ${{ secrets.VPS2_SSH_HOST }}
|
||||||
username: ${{ secrets.PROD_SSH_USER }}
|
username: ${{ secrets.VPS2_SSH_USER }}
|
||||||
key: ${{ secrets.PROD_SSH_KEY }}
|
key: ${{ secrets.VPS2_SSH_KEY }}
|
||||||
port: ${{ secrets.PROD_SSH_PORT }}
|
port: ${{ secrets.VPS2_SSH_PORT }}
|
||||||
script: |
|
script: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
cd /opt/thermograph/infra
|
cd /opt/thermograph/infra
|
||||||
|
|
|
||||||
65
CLAUDE.md
65
CLAUDE.md
|
|
@ -15,24 +15,35 @@ every change, so a stale one is a correctness bug, not a documentation bug.
|
||||||
| Branch | Deploys to | Workflow |
|
| Branch | Deploys to | Workflow |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| feature branch | nothing | PR into `dev` |
|
| feature branch | nothing | PR into `dev` |
|
||||||
| `dev` | nothing | integration branch only — see below |
|
| `dev` | dev (vps1, mesh-only, own Postgres) | `deploy.yml` |
|
||||||
| `main` | beta (beta.thermograph.org) | `deploy.yml` |
|
| `main` | beta (beta.thermograph.org, vps2) | `deploy.yml` |
|
||||||
| `release` | prod (thermograph.org) | `deploy.yml` |
|
| `release` | prod (thermograph.org, vps2) | `deploy.yml` |
|
||||||
|
|
||||||
`dev`, `main` and `release` are protected: **everything is a PR**, for humans and
|
`dev`, `main` and `release` are protected: **everything is a PR**, for humans and
|
||||||
agents alike. Promotion is one PR per hop, `dev` → `main` → `release`.
|
agents alike. Promotion is one PR per hop, `dev` → `main` → `release`.
|
||||||
|
|
||||||
**`dev` deploys nowhere.** It is purely the integration branch that feature PRs
|
`.claude/BRANCHING.md` is the orchestrator's runbook for that flow — the
|
||||||
land on before promotion to `main`. The two LAN-dev deploy workflows were deleted
|
serialised merge queue, promotions, hotfix down-merges — and
|
||||||
rather than kept: they called a monorepo path that does not exist on the LAN box
|
`.claude/ownership.md` is the module map saying which tasks may run
|
||||||
(`~/thermograph-dev` is still a split-era `thermograph-infra` checkout), so they
|
concurrently. Read both before dispatching parallel work.
|
||||||
had been inert since cutover. Run LAN dev locally with `infra/`'s `make dev-up`.
|
|
||||||
|
|
||||||
**One workflow deploys everything.** `deploy.yml` handles both services and both
|
**`dev` is a first-class hosted environment, not just an integration branch.**
|
||||||
environments: the branch selects the environment (`main` → beta, `release` →
|
It runs on vps1 — the same box as Forgejo and the monitoring stack — with its
|
||||||
prod), a matrix covers backend and frontend, and each leg checks whether this
|
own Postgres container, reachable only on the WireGuard mesh
|
||||||
push actually touched its domain before rolling. `build-push.yml` is the same
|
(`10.10.0.2:8137`): no public DNS record, no Caddy site, no TLS. It is deployed
|
||||||
shape for images. They replaced six and two near-identical files respectively.
|
by CI like beta and prod. The desktop hosts no Thermograph environment at all;
|
||||||
|
`make dev-up` there is a laptop convenience for running the stack locally, not
|
||||||
|
a deployment target.
|
||||||
|
|
||||||
|
**One workflow deploys everything.** `deploy.yml` handles both services and all
|
||||||
|
three environments: the branch selects the environment (`dev` → vps1, `main` →
|
||||||
|
beta on vps2, `release` → prod on vps2), a matrix covers backend and frontend,
|
||||||
|
and each leg checks whether this push actually touched its domain before
|
||||||
|
rolling. `build-push.yml` is the same shape for images. `THERMOGRAPH_ENV` is the
|
||||||
|
input that tells a leg which environment it's deploying — load-bearing on vps2,
|
||||||
|
which runs beta and prod side by side and has no other way to tell them apart.
|
||||||
|
See `infra/deploy/env-topology.sh` for the single source of truth on where each
|
||||||
|
environment's checkout, branch, stack and ports live.
|
||||||
|
|
||||||
## The deploy contract
|
## The deploy contract
|
||||||
|
|
||||||
|
|
@ -44,15 +55,25 @@ SERVICE=backend|frontend|all BACKEND_IMAGE_TAG=sha-<12hex> FRONTEND_IMAGE_TAG=
|
||||||
```
|
```
|
||||||
|
|
||||||
`deploy.sh` resets the host checkout, renders secrets from the SOPS vault, then
|
`deploy.sh` resets the host checkout, renders secrets from the SOPS vault, then
|
||||||
either rolls compose services or — if `/etc/thermograph/deploy-mode` contains
|
either rolls compose services or — if `infra/deploy/env-topology.sh` says the
|
||||||
`stack` — execs `infra/deploy/stack/deploy-stack.sh`.
|
environment's deploy mode is `stack` — execs `infra/deploy/stack/deploy-stack.sh`.
|
||||||
|
The old host-wide `/etc/thermograph/deploy-mode` marker survives only as a
|
||||||
|
fallback for a by-hand run with no environment resolvable any other way; it
|
||||||
|
cannot describe vps2 alone, since vps2 runs beta and prod side by side.
|
||||||
|
|
||||||
- **prod runs Swarm.** Its stack is `infra/deploy/stack/thermograph-stack.yml`
|
- **prod and beta both run Swarm, as two separate stacks co-resident on vps2.**
|
||||||
(db, web, worker, lake, daemon, frontend, autoscaler, autoscaler-lake).
|
Prod's is `infra/deploy/stack/thermograph-stack.yml` (db, web, worker, lake,
|
||||||
`deploy-stack.sh` also offers `STACK_TEST=1`: a full parallel rehearsal on
|
daemon, frontend, autoscaler, autoscaler-lake). Beta's is the separate
|
||||||
throwaway volumes and ports that cannot touch live data.
|
`infra/deploy/stack/thermograph-beta-stack.yml` — its services are prefixed
|
||||||
- **beta and LAN dev run compose**, from `infra/docker-compose.yml`
|
(`beta-web`, `beta-worker`, …) because Swarm registers a service's short name
|
||||||
(db, backend, lake, daemon, frontend).
|
as a DNS alias on every network it joins, and beta shares prod's `data`
|
||||||
|
network to reach the database. Beta has no `db` service of its own: one
|
||||||
|
TimescaleDB instance serves both, on separate databases and separate
|
||||||
|
NOSUPERUSER roles. `deploy-stack.sh` also offers `STACK_TEST=1`: a full
|
||||||
|
parallel rehearsal on throwaway volumes and ports that cannot touch live data.
|
||||||
|
- **dev runs compose**, from `infra/docker-compose.yml` (db, backend, lake,
|
||||||
|
daemon, frontend), on its own host (vps1) with its own Postgres container —
|
||||||
|
the one environment not sharing a database with anything else.
|
||||||
- Each service's live tag is persisted host-side, so a single-service roll never
|
- Each service's live tag is persisted host-side, so a single-service roll never
|
||||||
disturbs the sibling's running tag.
|
disturbs the sibling's running tag.
|
||||||
|
|
||||||
|
|
@ -76,7 +97,7 @@ reunification.
|
||||||
- Anything named "prefetch" must never spend the Open-Meteo quota.
|
- Anything named "prefetch" must never spend the Open-Meteo quota.
|
||||||
Nominatim ≤ 1 req/s.
|
Nominatim ≤ 1 req/s.
|
||||||
- The compose project name is pinned (`name: thermograph` in
|
- The compose project name is pinned (`name: thermograph` in
|
||||||
`infra/docker-compose.yml`); LAN dev overrides with
|
`infra/docker-compose.yml`); dev (and a local `make dev-up`) overrides with
|
||||||
`COMPOSE_PROJECT_NAME=thermograph-dev`. Don't remove either half — the pinned
|
`COMPOSE_PROJECT_NAME=thermograph-dev`. Don't remove either half — the pinned
|
||||||
name is what makes the Swarm stack's external volume names line up.
|
name is what makes the Swarm stack's external volume names line up.
|
||||||
- `CUTOVER-NOTES.md` is the source of truth for what is and isn't live yet.
|
- `CUTOVER-NOTES.md` is the source of truth for what is and isn't live yet.
|
||||||
|
|
|
||||||
|
|
@ -153,3 +153,38 @@ Checked every split-repo branch by dry-run merge; migrated everything real:
|
||||||
historical.
|
historical.
|
||||||
- Infra `Makefile` local-image targets still assume sibling checkouts — update
|
- Infra `Makefile` local-image targets still assume sibling checkouts — update
|
||||||
when first needed locally.
|
when first needed locally.
|
||||||
|
|
||||||
|
## vps1/vps2 host-topology re-architecture (landed 2026-07-25, NOT YET EXECUTED)
|
||||||
|
|
||||||
|
**This section documents a second re-architecture, layered on top of the
|
||||||
|
monorepo cutover above. It is landed IN THE REPO — code, stack files, deploy
|
||||||
|
scripts, docs — but has NOT been run against the live estate.** The boxes
|
||||||
|
described elsewhere in this file (prod at `169.58.46.181`, beta at
|
||||||
|
`75.119.132.91`, a desktop LAN dev box) are still what is actually running
|
||||||
|
today. Do not treat anything below as live until the cutover runbook has been
|
||||||
|
executed and this note updated.
|
||||||
|
|
||||||
|
What changes, in one line: hosts are renamed by **role** instead of by
|
||||||
|
environment, and beta moves onto the same box as prod.
|
||||||
|
|
||||||
|
- **vps1** (mesh `10.10.0.2`, public `75.119.132.91`) — unchanged box, new job.
|
||||||
|
Keeps Forgejo, Grafana/Loki/Alloy and the `emigriffith.dev` portfolio; gains
|
||||||
|
the **dev** environment (its own Postgres, plain compose, mesh-only —
|
||||||
|
`10.10.0.2:8137`, no DNS record, no Caddy site, no TLS). This is the box that
|
||||||
|
used to be called "beta".
|
||||||
|
- **vps2** (mesh `10.10.0.1`, public `169.58.46.181`) — unchanged box, new job.
|
||||||
|
Keeps prod; gains **beta** as a second, co-resident Docker Swarm stack
|
||||||
|
(prefixed services, own checkout, own env file, own loopback LB ports). One
|
||||||
|
TimescaleDB instance now serves both prod and beta, on separate databases and
|
||||||
|
separate roles. This is the box that used to be called "prod".
|
||||||
|
- **desktop** (mesh `10.10.0.3`) — loses the LAN dev stack and the Forgejo
|
||||||
|
Actions runner entirely; becomes AI-model hosting plus flex Swarm capacity.
|
||||||
|
`make dev-up` still works there as a laptop convenience, but it is no longer
|
||||||
|
an environment.
|
||||||
|
|
||||||
|
The mesh IPs did not move — only which environment lives on which box did. See
|
||||||
|
`infra/deploy/env-topology.sh` for the single source of truth on per-environment
|
||||||
|
checkout paths, branches, stack names, ports and DB roles, and
|
||||||
|
**[`infra/deploy/RUNBOOK-vps1-vps2-cutover.md`](infra/deploy/RUNBOOK-vps1-vps2-cutover.md)**
|
||||||
|
for the execution runbook. Until that runbook has been run, `CLAUDE.md`,
|
||||||
|
`README.md` and `docs/onboarding/` describe the target state, not today's.
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,7 @@ for: **per-domain images, per-domain deploys, and an async FE/BE contract**.
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `backend/` | FastAPI graded-climate API, accounts, notifications (Discord bot, push, mail), data pipeline | `build-push` → image `emi/thermograph/backend`; `deploy` |
|
| `backend/` | FastAPI graded-climate API, accounts, notifications (Discord bot, push, mail), data pipeline | `build-push` → image `emi/thermograph/backend`; `deploy` |
|
||||||
| `frontend/` | Public client: static JS/CSS + SSR pages | same `build-push` / `deploy` workflows, matrixed by domain; image `emi/thermograph/frontend` |
|
| `frontend/` | Public client: static JS/CSS + SSR pages | same `build-push` / `deploy` workflows, matrixed by domain; image `emi/thermograph/frontend` |
|
||||||
| `infra/` | Compose (beta, LAN dev) + the Swarm stack (prod), deploy scripts, terraform, SOPS secrets vault, ops cron | `infra-sync` (host checkout + secrets render), `secrets-guard`, `ops-cron` |
|
| `infra/` | Compose (dev, on vps1) + two Swarm stacks co-resident on vps2 (beta, prod), deploy scripts, terraform, SOPS secrets vault, ops cron | `infra-sync` (host checkout + secrets render), `secrets-guard`, `ops-cron` |
|
||||||
| `observability/` | Loki + Grafana + Alloy stack | `observability-validate` |
|
| `observability/` | Loki + Grafana + Alloy stack | `observability-validate` |
|
||||||
|
|
||||||
`thermograph-docs` deliberately **stays its own repo** (ADRs + runbooks, no
|
`thermograph-docs` deliberately **stays its own repo** (ADRs + runbooks, no
|
||||||
|
|
@ -36,6 +36,9 @@ The one intentionally *coupled* piece is `pr-build.yml`: a single always-running
|
||||||
path-filtered required check would deadlock auto-merge).
|
path-filtered required check would deadlock auto-merge).
|
||||||
|
|
||||||
Branch model (unchanged from the split era): PRs → `dev`, `main` → beta,
|
Branch model (unchanged from the split era): PRs → `dev`, `main` → beta,
|
||||||
`release` → prod; infra tracked via `main` on all hosts.
|
`release` → prod. Infra isn't environment-staged the same way app images are:
|
||||||
|
beta's and prod's checkouts (both on vps2) track `main`; dev's checkout (on
|
||||||
|
vps1) tracks `dev` itself, since it's the one environment that isn't a
|
||||||
|
rehearsal for something downstream.
|
||||||
|
|
||||||
**Before pointing anything live at this repo, read `CUTOVER-NOTES.md`.**
|
**Before pointing anything live at this repo, read `CUTOVER-NOTES.md`.**
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
"""Authenticated API for subscriptions and notifications.
|
"""Authenticated API for subscriptions, notifications, and bookmarks.
|
||||||
|
|
||||||
All routes require a logged-in user (the fastapi-users cookie session) and are
|
All routes require a logged-in user (the fastapi-users cookie session) and are
|
||||||
scoped to that user — a row that isn't theirs reads as 404, never 403, so the API
|
scoped to that user — a row that isn't theirs reads as 404, never 403, so the API
|
||||||
|
|
@ -7,7 +7,7 @@ doesn't leak which ids exist. Mounted under {BASE}/api/v2 alongside the rest of
|
||||||
import os
|
import os
|
||||||
import time
|
import time
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response
|
||||||
from fastapi.concurrency import run_in_threadpool
|
from fastapi.concurrency import run_in_threadpool
|
||||||
from sqlalchemy import func, select
|
from sqlalchemy import func, select
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
@ -17,8 +17,15 @@ from data import climate
|
||||||
from data import grid
|
from data import grid
|
||||||
from notifications import push
|
from notifications import push
|
||||||
from accounts.db import get_async_session, get_read_session
|
from accounts.db import get_async_session, get_read_session
|
||||||
from accounts.models import Notification, PushSubscription, Subscription
|
from accounts.models import Bookmark, Notification, PushSubscription, Subscription
|
||||||
from accounts.schemas import (
|
from accounts.schemas import (
|
||||||
|
BOOKMARK_MAX_PER_USER,
|
||||||
|
BookmarkImportIn,
|
||||||
|
BookmarkImportOut,
|
||||||
|
BookmarkIn,
|
||||||
|
BookmarkList,
|
||||||
|
BookmarkOut,
|
||||||
|
BookmarkPatch,
|
||||||
NotificationList,
|
NotificationList,
|
||||||
NotificationOut,
|
NotificationOut,
|
||||||
PushSubscriptionIn,
|
PushSubscriptionIn,
|
||||||
|
|
@ -323,3 +330,171 @@ async def push_test(
|
||||||
# `failed` (delivery rejected — e.g. VAPID key mismatch) is surfaced so the client
|
# `failed` (delivery rejected — e.g. VAPID key mismatch) is surfaced so the client
|
||||||
# can tell "request accepted" apart from "actually delivered".
|
# can tell "request accepted" apart from "actually delivered".
|
||||||
return {"devices": len(rows), "sent": sent, "pruned": pruned, "failed": failed}
|
return {"devices": len(rows), "sent": sent, "pruned": pruned, "failed": failed}
|
||||||
|
|
||||||
|
|
||||||
|
# --- bookmarks -----------------------------------------------------------------
|
||||||
|
async def _owned_bookmark(session: AsyncSession, bookmark_id: int, user) -> Bookmark:
|
||||||
|
bookmark = (
|
||||||
|
await session.execute(
|
||||||
|
select(Bookmark).where(
|
||||||
|
Bookmark.id == bookmark_id, Bookmark.user_id == user.id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
).scalar_one_or_none()
|
||||||
|
if bookmark is None:
|
||||||
|
raise HTTPException(status_code=404, detail="Bookmark not found.")
|
||||||
|
return bookmark
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/bookmarks", response_model=BookmarkList)
|
||||||
|
async def list_bookmarks(
|
||||||
|
user=Depends(current_active_user),
|
||||||
|
session: AsyncSession = Depends(get_read_session), # pure read → read-only engine
|
||||||
|
):
|
||||||
|
rows = (
|
||||||
|
await session.execute(
|
||||||
|
select(Bookmark)
|
||||||
|
.where(Bookmark.user_id == user.id)
|
||||||
|
.order_by(Bookmark.created_at)
|
||||||
|
)
|
||||||
|
).scalars().all()
|
||||||
|
return BookmarkList(bookmarks=[BookmarkOut.model_validate(r) for r in rows])
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/bookmarks", response_model=BookmarkOut, status_code=201)
|
||||||
|
async def create_bookmark(
|
||||||
|
body: BookmarkIn,
|
||||||
|
response: Response,
|
||||||
|
user=Depends(current_active_user),
|
||||||
|
session: AsyncSession = Depends(get_async_session),
|
||||||
|
):
|
||||||
|
"""Create a bookmark, or — if one already exists for this (user, cell) —
|
||||||
|
rename it in place and return 200. Idempotent upsert rather than a 409, so
|
||||||
|
re-bookmarking a place you already saved is never an error."""
|
||||||
|
cell = grid.snap(body.lat, body.lon)
|
||||||
|
cell_id = cell["id"]
|
||||||
|
|
||||||
|
existing = (
|
||||||
|
await session.execute(
|
||||||
|
select(Bookmark).where(
|
||||||
|
Bookmark.user_id == user.id, Bookmark.cell_id == cell_id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
).scalar_one_or_none()
|
||||||
|
if existing is not None:
|
||||||
|
existing.label = body.label
|
||||||
|
await session.commit()
|
||||||
|
await session.refresh(existing)
|
||||||
|
audit.log_activity("bookmark.update", {"user_id": str(user.id),
|
||||||
|
"bookmark_id": existing.id})
|
||||||
|
response.status_code = 200
|
||||||
|
return existing
|
||||||
|
|
||||||
|
count = (
|
||||||
|
await session.execute(
|
||||||
|
select(func.count()).select_from(Bookmark).where(Bookmark.user_id == user.id)
|
||||||
|
)
|
||||||
|
).scalar_one()
|
||||||
|
if count >= BOOKMARK_MAX_PER_USER:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=409,
|
||||||
|
detail=f"You can have at most {BOOKMARK_MAX_PER_USER} bookmarks.",
|
||||||
|
)
|
||||||
|
|
||||||
|
bookmark = Bookmark(
|
||||||
|
user_id=user.id,
|
||||||
|
cell_id=cell_id,
|
||||||
|
label=body.label,
|
||||||
|
lat=body.lat,
|
||||||
|
lon=body.lon,
|
||||||
|
)
|
||||||
|
session.add(bookmark)
|
||||||
|
await session.commit()
|
||||||
|
await session.refresh(bookmark)
|
||||||
|
audit.log_activity("bookmark.create", {"user_id": str(user.id), "bookmark_id": bookmark.id,
|
||||||
|
"cell_id": cell_id})
|
||||||
|
return bookmark
|
||||||
|
|
||||||
|
|
||||||
|
@router.patch("/bookmarks/{bookmark_id}", response_model=BookmarkOut)
|
||||||
|
async def update_bookmark(
|
||||||
|
bookmark_id: int,
|
||||||
|
body: BookmarkPatch,
|
||||||
|
user=Depends(current_active_user),
|
||||||
|
session: AsyncSession = Depends(get_async_session),
|
||||||
|
):
|
||||||
|
bookmark = await _owned_bookmark(session, bookmark_id, user)
|
||||||
|
bookmark.label = body.label
|
||||||
|
await session.commit()
|
||||||
|
await session.refresh(bookmark)
|
||||||
|
audit.log_activity("bookmark.rename", {"user_id": str(user.id), "bookmark_id": bookmark.id})
|
||||||
|
return bookmark
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/bookmarks/{bookmark_id}", status_code=204)
|
||||||
|
async def delete_bookmark(
|
||||||
|
bookmark_id: int,
|
||||||
|
user=Depends(current_active_user),
|
||||||
|
session: AsyncSession = Depends(get_async_session),
|
||||||
|
):
|
||||||
|
bookmark = await _owned_bookmark(session, bookmark_id, user)
|
||||||
|
await session.delete(bookmark)
|
||||||
|
await session.commit()
|
||||||
|
audit.log_activity("bookmark.delete", {"user_id": str(user.id), "bookmark_id": bookmark_id})
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/bookmarks/import", response_model=BookmarkImportOut)
|
||||||
|
async def import_bookmarks(
|
||||||
|
body: BookmarkImportIn,
|
||||||
|
user=Depends(current_active_user),
|
||||||
|
session: AsyncSession = Depends(get_async_session),
|
||||||
|
):
|
||||||
|
"""Bulk-import a visitor's localStorage bookmarks at login. Existing rows keep
|
||||||
|
their label (never overwritten); duplicates within the payload itself are also
|
||||||
|
skipped. Stops importing once the user's total hits BOOKMARK_MAX_PER_USER —
|
||||||
|
any remaining items count as skipped rather than erroring."""
|
||||||
|
existing_rows = (
|
||||||
|
await session.execute(select(Bookmark).where(Bookmark.user_id == user.id))
|
||||||
|
).scalars().all()
|
||||||
|
existing_cells = {r.cell_id for r in existing_rows}
|
||||||
|
count = len(existing_rows)
|
||||||
|
|
||||||
|
imported = 0
|
||||||
|
skipped = 0
|
||||||
|
seen_cells: set[str] = set()
|
||||||
|
for item in body.items:
|
||||||
|
cell_id = grid.snap(item.lat, item.lon)["id"]
|
||||||
|
if cell_id in existing_cells or cell_id in seen_cells:
|
||||||
|
skipped += 1
|
||||||
|
continue
|
||||||
|
if count >= BOOKMARK_MAX_PER_USER:
|
||||||
|
skipped += 1
|
||||||
|
continue
|
||||||
|
session.add(Bookmark(
|
||||||
|
user_id=user.id,
|
||||||
|
cell_id=cell_id,
|
||||||
|
label=item.label,
|
||||||
|
lat=item.lat,
|
||||||
|
lon=item.lon,
|
||||||
|
))
|
||||||
|
seen_cells.add(cell_id)
|
||||||
|
count += 1
|
||||||
|
imported += 1
|
||||||
|
|
||||||
|
if imported:
|
||||||
|
await session.commit()
|
||||||
|
audit.log_activity("bookmark.import", {"user_id": str(user.id),
|
||||||
|
"imported": imported, "skipped": skipped})
|
||||||
|
|
||||||
|
rows = (
|
||||||
|
await session.execute(
|
||||||
|
select(Bookmark)
|
||||||
|
.where(Bookmark.user_id == user.id)
|
||||||
|
.order_by(Bookmark.created_at)
|
||||||
|
)
|
||||||
|
).scalars().all()
|
||||||
|
return BookmarkImportOut(
|
||||||
|
imported=imported,
|
||||||
|
skipped=skipped,
|
||||||
|
bookmarks=[BookmarkOut.model_validate(r) for r in rows],
|
||||||
|
)
|
||||||
|
|
|
||||||
|
|
@ -62,11 +62,21 @@ def _apply_sqlite_pragmas(dbapi_conn, _rec):
|
||||||
#
|
#
|
||||||
# pool_size=1/max_overflow=1 (the original setting on both) meant a 3rd
|
# pool_size=1/max_overflow=1 (the original setting on both) meant a 3rd
|
||||||
# concurrent request per worker had to wait out pool_timeout (~30s default) for a
|
# concurrent request per worker had to wait out pool_timeout (~30s default) for a
|
||||||
# slot — easy to hit with 4 uvicorn workers x real traffic. 5+5 gives each async
|
# slot — easy to hit with 4 uvicorn workers x real traffic. pool_size=5 keeps that
|
||||||
# engine headroom for a burst of concurrent account requests without coming close
|
# headroom for the steady state.
|
||||||
# to Postgres's own connection ceiling (two async engines here plus the sync
|
#
|
||||||
# engine below, times N workers, all share that budget).
|
# max_overflow is deliberately *tighter* than pool_size (2, not 5), because the
|
||||||
_ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=5, pool_pre_ping=True,
|
# overflow is the part nobody budgets for and it is what saturated Postgres on
|
||||||
|
# 2026-07-26. The connection ceiling is shared far more widely than this process
|
||||||
|
# knows: two async engines here plus the sync engine below, times N uvicorn
|
||||||
|
# workers, times the replica count (web autoscales), and prod and beta are two
|
||||||
|
# databases on ONE Postgres instance. At 5+5 the configured worst case across all
|
||||||
|
# of that exceeded the server's max_connections; at 5+2 prod web's ceiling is
|
||||||
|
# 4x14 rather than 4x20. Steady-state behaviour is unchanged — only the burst
|
||||||
|
# ceiling narrows, and a burst that would have opened a 6th..10th connection now
|
||||||
|
# waits on pool_timeout instead of being refused outright by the server (which is
|
||||||
|
# an error, not a wait). See infra/deploy/db/init/20-tuning.sh for the server side.
|
||||||
|
_ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=2, pool_pre_ping=True,
|
||||||
pool_recycle=1800, future=True)
|
pool_recycle=1800, future=True)
|
||||||
# Smaller than the async engines: only the notifier leader (one process
|
# Smaller than the async engines: only the notifier leader (one process
|
||||||
# cluster-wide, see core/singleton.py) uses the sync engine, driving its periodic
|
# cluster-wide, see core/singleton.py) uses the sync engine, driving its periodic
|
||||||
|
|
@ -74,7 +84,8 @@ _ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=5, pool_pre_ping=True,
|
||||||
# statement_timeout (via psycopg's `options`) bound how long a wedged Postgres
|
# statement_timeout (via psycopg's `options`) bound how long a wedged Postgres
|
||||||
# can hang the notifier — it runs for the lifetime of that process's leader
|
# can hang the notifier — it runs for the lifetime of that process's leader
|
||||||
# flock, so an indefinite hang here would silently stop every subscription
|
# flock, so an indefinite hang here would silently stop every subscription
|
||||||
# notification until the process is restarted.
|
# notification until the process is restarted. Left at 2+3: one process holds it,
|
||||||
|
# so it is not multiplied by the worker or replica count.
|
||||||
_SYNC_POOL_KWARGS = dict(pool_size=2, max_overflow=3, pool_pre_ping=True,
|
_SYNC_POOL_KWARGS = dict(pool_size=2, max_overflow=3, pool_pre_ping=True,
|
||||||
pool_recycle=1800, future=True,
|
pool_recycle=1800, future=True,
|
||||||
connect_args={"connect_timeout": 5,
|
connect_args={"connect_timeout": 5,
|
||||||
|
|
|
||||||
|
|
@ -24,7 +24,7 @@ from sqlalchemy import (
|
||||||
UniqueConstraint,
|
UniqueConstraint,
|
||||||
false,
|
false,
|
||||||
)
|
)
|
||||||
from sqlalchemy.orm import Mapped, mapped_column
|
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||||
|
|
||||||
from accounts.db import Base
|
from accounts.db import Base
|
||||||
|
|
||||||
|
|
@ -34,13 +34,67 @@ class User(SQLAlchemyBaseUserTableUUID, Base):
|
||||||
# is_superuser, is_verified. Optional extras:
|
# is_superuser, is_verified. Optional extras:
|
||||||
display_name: Mapped[str | None] = mapped_column(String(120), nullable=True)
|
display_name: Mapped[str | None] = mapped_column(String(120), nullable=True)
|
||||||
# Linked Discord account id (OAuth2 identify) — the key DM alerts reach the user
|
# Linked Discord account id (OAuth2 identify) — the key DM alerts reach the user
|
||||||
# by. NB: create_all only makes this on a fresh DB; an existing prod accounts.db
|
# by, and the identity "Sign in with Discord" resolves an account from. Unique,
|
||||||
# needs the manual migration in deploy/migrations/ (no Alembic in this project).
|
# so one Discord account can never own two Thermograph accounts.
|
||||||
discord_id: Mapped[str | None] = mapped_column(String(32), unique=True, nullable=True)
|
discord_id: Mapped[str | None] = mapped_column(String(32), unique=True, nullable=True)
|
||||||
# Whether to also deliver alerts as a Discord DM. Set True on linking (an active
|
# Whether to also deliver alerts as a Discord DM. Set True on linking (an active
|
||||||
# opt-in); the user can mute it while staying linked. Same migration caveat.
|
# opt-in); the user can mute it while staying linked. Same migration caveat.
|
||||||
discord_dm: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False,
|
discord_dm: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False,
|
||||||
server_default=false())
|
server_default=false())
|
||||||
|
# True when the account was created by signing in with an OAuth provider and so
|
||||||
|
# has no password its owner has ever seen (hashed_password is NOT NULL, so the
|
||||||
|
# row carries a generated one nobody knows). Load-bearing: unlinking the *last*
|
||||||
|
# linked provider from such an account would remove its only way in, so
|
||||||
|
# accounts/oauth.py refuses that. Was `discord_only` before Google was added.
|
||||||
|
oauth_only: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False,
|
||||||
|
server_default=false())
|
||||||
|
|
||||||
|
# selectin, not the lazy default: these are read while serialising /users/me on
|
||||||
|
# the async engine, where a lazy load raises MissingGreenlet rather than
|
||||||
|
# quietly issuing a query. One extra SELECT per user load is the price.
|
||||||
|
oauth_accounts: Mapped[list["OAuthAccount"]] = relationship(
|
||||||
|
"OAuthAccount", lazy="selectin", cascade="all, delete-orphan",
|
||||||
|
)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def oauth_providers(self) -> list[str]:
|
||||||
|
"""Linked provider names — read straight onto UserRead by from_attributes."""
|
||||||
|
return sorted(a.provider for a in self.oauth_accounts)
|
||||||
|
|
||||||
|
|
||||||
|
class OAuthAccount(Base):
|
||||||
|
"""One external identity — a provider plus that provider's own user id — bound
|
||||||
|
to a Thermograph account. This is what "sign in with X" resolves against.
|
||||||
|
|
||||||
|
Keyed on ``subject``, never email: the provider's id for an account is stable,
|
||||||
|
while an email address can be changed or reassigned. Email is stored only for
|
||||||
|
support and debugging, and is deliberately not used for lookup.
|
||||||
|
|
||||||
|
Note ``User.discord_id`` is *not* redundant with a discord row here. That column
|
||||||
|
is a delivery address — notify.py DMs it — and survives on its own terms; this
|
||||||
|
table is purely about authentication.
|
||||||
|
"""
|
||||||
|
__tablename__ = "oauth_account"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||||
|
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||||
|
GUID, ForeignKey("user.id", ondelete="CASCADE"), nullable=False
|
||||||
|
)
|
||||||
|
provider: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||||
|
# "sub" for Google, "id" for Discord.
|
||||||
|
subject: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
|
email: Mapped[str | None] = mapped_column(String(320), nullable=True)
|
||||||
|
created_at: Mapped[float] = mapped_column(Float, nullable=False, default=time.time)
|
||||||
|
|
||||||
|
__table_args__ = (
|
||||||
|
# A provider account signs into exactly one Thermograph account — this is
|
||||||
|
# the constraint that stops one Google login resolving two ways.
|
||||||
|
UniqueConstraint("provider", "subject", name="uq_oauth_provider_subject"),
|
||||||
|
# And an account holds at most one identity per provider, so "connect
|
||||||
|
# Google" is idempotent rather than accumulating rows.
|
||||||
|
UniqueConstraint("user_id", "provider", name="uq_oauth_user_provider"),
|
||||||
|
Index("idx_oauth_user", "user_id"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class AccessToken(SQLAlchemyBaseAccessTokenTableUUID, Base):
|
class AccessToken(SQLAlchemyBaseAccessTokenTableUUID, Base):
|
||||||
|
|
@ -189,3 +243,31 @@ class PendingDigest(Base):
|
||||||
# duplicating, which is what makes the form idempotent.
|
# duplicating, which is what makes the form idempotent.
|
||||||
UniqueConstraint("email", name="uq_pending_digest_email"),
|
UniqueConstraint("email", name="uq_pending_digest_email"),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class Bookmark(Base):
|
||||||
|
"""A saved location, owned by a user — the "bookmarked locations" feature.
|
||||||
|
|
||||||
|
Keyed like ``Subscription`` on ``grid.snap(lat, lon)["id"]``: one bookmark per
|
||||||
|
(user, cell), so re-bookmarking the same cell is an upsert of the label rather
|
||||||
|
than a duplicate row (see ``create_bookmark`` / ``import_bookmarks`` in
|
||||||
|
api_accounts.py, which enforce this at the application layer too).
|
||||||
|
"""
|
||||||
|
__tablename__ = "bookmark"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||||
|
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||||
|
GUID, ForeignKey("user.id", ondelete="CASCADE"), nullable=False
|
||||||
|
)
|
||||||
|
# grid.snap(lat, lon)["id"] — the stable per-location key used across the app.
|
||||||
|
cell_id: Mapped[str] = mapped_column(String(40), nullable=False)
|
||||||
|
label: Mapped[str] = mapped_column(String(80), nullable=False)
|
||||||
|
lat: Mapped[float] = mapped_column(Float, nullable=False)
|
||||||
|
lon: Mapped[float] = mapped_column(Float, nullable=False)
|
||||||
|
created_at: Mapped[float] = mapped_column(Float, nullable=False, default=time.time)
|
||||||
|
|
||||||
|
__table_args__ = (
|
||||||
|
# One bookmark per location per user — re-bookmarking upserts the label.
|
||||||
|
UniqueConstraint("user_id", "cell_id", name="uq_bookmark_user_cell"),
|
||||||
|
Index("idx_bookmark_user", "user_id"),
|
||||||
|
)
|
||||||
|
|
|
||||||
528
backend/accounts/oauth.py
Normal file
528
backend/accounts/oauth.py
Normal file
|
|
@ -0,0 +1,528 @@
|
||||||
|
"""Sign in with an external provider, and connect one to an existing account.
|
||||||
|
|
||||||
|
One engine, two providers (Discord, Google). Adding a third means adding a
|
||||||
|
``Provider`` entry and nothing else — the flow, the signed state, the account
|
||||||
|
resolution and the session issuance are all provider-agnostic on purpose. That is
|
||||||
|
not tidiness for its own sake: account resolution is the security-critical part,
|
||||||
|
and a second hand-rolled copy is where a subtle divergence becomes a takeover.
|
||||||
|
|
||||||
|
Two flows over one authorization-code grant, told apart by the signed `state`:
|
||||||
|
|
||||||
|
link — /oauth/{provider}/link/start a signed-in user attaches an identity
|
||||||
|
login — /oauth/{provider}/login/start an anonymous visitor authenticates
|
||||||
|
|
||||||
|
`state` is signed with the app's auth secret (stdlib hmac, no new dependency) and
|
||||||
|
carries the provider, the purpose, and — for a link — the Thermograph user id. All
|
||||||
|
three are inside the HMAC, so a state cannot be replayed, bound to a different
|
||||||
|
account, replayed as the other flow, or replayed against a different provider.
|
||||||
|
|
||||||
|
How a login resolves to an account, in order:
|
||||||
|
|
||||||
|
1. An ``oauth_account`` row for (provider, subject) — the durable key, no email
|
||||||
|
needed and immune to the user changing their address at the provider.
|
||||||
|
2. Otherwise the provider's email, but **only if it reports it verified**. That
|
||||||
|
flag is the sole evidence the person owns the address; matching an existing
|
||||||
|
Thermograph account on an unverified one would hand that account to whoever
|
||||||
|
typed the address into Discord or Google. Unverified is refused outright.
|
||||||
|
3. No account for that email: create one, flagged ``oauth_only`` because it has
|
||||||
|
no password its owner has ever seen.
|
||||||
|
|
||||||
|
**Legacy routes.** Discord shipped first, under /discord/*, and its callback URL is
|
||||||
|
registered in Discord's developer portal — so /discord/link/callback keeps working
|
||||||
|
forever, and the /discord/* aliases stay because the frontend deploys independently
|
||||||
|
of this service and an older one must keep working against a newer backend.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import hashlib
|
||||||
|
import hmac
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
import urllib.parse
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Callable
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||||
|
from fastapi.responses import RedirectResponse
|
||||||
|
from sqlalchemy import delete, func, select, update
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
|
from accounts.db import get_async_session
|
||||||
|
from accounts.models import OAuthAccount, User
|
||||||
|
from accounts.users import (
|
||||||
|
SECRET,
|
||||||
|
attach_session,
|
||||||
|
current_active_user,
|
||||||
|
current_user_optional,
|
||||||
|
get_access_token_db,
|
||||||
|
get_user_manager,
|
||||||
|
)
|
||||||
|
from core import audit
|
||||||
|
|
||||||
|
router = APIRouter(tags=["oauth"])
|
||||||
|
|
||||||
|
BASE = os.environ.get("THERMOGRAPH_BASE", "/thermograph").rstrip("/")
|
||||||
|
|
||||||
|
# How long a start->callback round trip may take before the signed state expires.
|
||||||
|
_STATE_TTL = 600
|
||||||
|
_TIMEOUT = httpx.Timeout(10.0)
|
||||||
|
|
||||||
|
|
||||||
|
# --- providers ----------------------------------------------------------------
|
||||||
|
def _parse_discord(p: dict) -> tuple[str, str, bool]:
|
||||||
|
return (str(p.get("id") or ""),
|
||||||
|
str(p.get("email") or "").strip().lower(),
|
||||||
|
p.get("verified") is True)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_google(p: dict) -> tuple[str, str, bool]:
|
||||||
|
# OpenID Connect userinfo. `email_verified` arrives as a real bool from the
|
||||||
|
# JSON endpoint, but Google has historically also serialised it as the string
|
||||||
|
# "true" in some responses, so accept both rather than silently reading a
|
||||||
|
# verified address as unverified and refusing every Google login.
|
||||||
|
verified = p.get("email_verified")
|
||||||
|
return (str(p.get("sub") or ""),
|
||||||
|
str(p.get("email") or "").strip().lower(),
|
||||||
|
verified is True or str(verified).lower() == "true")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Provider:
|
||||||
|
name: str
|
||||||
|
label: str
|
||||||
|
authorize_url: str
|
||||||
|
token_url: str
|
||||||
|
userinfo_url: str
|
||||||
|
# Linking needs only an identity; signing in also needs the address to resolve
|
||||||
|
# or create an account by, which is a separate scope the user is prompted for.
|
||||||
|
scope_link: str
|
||||||
|
scope_login: str
|
||||||
|
# Where this provider's callback lives. Discord's is grandfathered to the path
|
||||||
|
# already registered in its portal; anything new gets the canonical one.
|
||||||
|
callback_path: str
|
||||||
|
parse: Callable[[dict], tuple[str, str, bool]]
|
||||||
|
# Extra authorize-URL params. Google needs prompt=consent to re-issue consent,
|
||||||
|
# and Discord takes the same key, so this stays per-provider rather than shared.
|
||||||
|
extra_authorize: dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
PROVIDERS: dict[str, Provider] = {
|
||||||
|
"discord": Provider(
|
||||||
|
name="discord",
|
||||||
|
label="Discord",
|
||||||
|
authorize_url="https://discord.com/api/oauth2/authorize",
|
||||||
|
token_url="https://discord.com/api/oauth2/token",
|
||||||
|
userinfo_url="https://discord.com/api/users/@me",
|
||||||
|
scope_link="identify",
|
||||||
|
scope_login="identify email",
|
||||||
|
callback_path="/api/v2/discord/link/callback",
|
||||||
|
parse=_parse_discord,
|
||||||
|
extra_authorize={"prompt": "consent"},
|
||||||
|
),
|
||||||
|
"google": Provider(
|
||||||
|
name="google",
|
||||||
|
label="Google",
|
||||||
|
authorize_url="https://accounts.google.com/o/oauth2/v2/auth",
|
||||||
|
token_url="https://oauth2.googleapis.com/token",
|
||||||
|
userinfo_url="https://openidconnect.googleapis.com/v1/userinfo",
|
||||||
|
# Google has no identity-only scope worth the name: openid alone yields a
|
||||||
|
# subject but no address, and we want the address recorded on the link too.
|
||||||
|
scope_link="openid email",
|
||||||
|
scope_login="openid email",
|
||||||
|
callback_path="/api/v2/oauth/google/callback",
|
||||||
|
parse=_parse_google,
|
||||||
|
# access_type=online: there is no offline work to do on the user's behalf,
|
||||||
|
# so asking for a refresh token would be collecting a credential we would
|
||||||
|
# never use. include_granted_scopes keeps incremental consent sane.
|
||||||
|
extra_authorize={"prompt": "consent", "access_type": "online",
|
||||||
|
"include_granted_scopes": "true"},
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
# Credentials, read once at import. A dict rather than per-provider constants so a
|
||||||
|
# test can enable or disable one provider without touching the others.
|
||||||
|
CREDENTIALS: dict[str, tuple[str, str]] = {
|
||||||
|
"discord": (os.environ.get("THERMOGRAPH_DISCORD_APP_ID", "").strip(),
|
||||||
|
os.environ.get("THERMOGRAPH_DISCORD_CLIENT_SECRET", "").strip()),
|
||||||
|
"google": (os.environ.get("THERMOGRAPH_GOOGLE_CLIENT_ID", "").strip(),
|
||||||
|
os.environ.get("THERMOGRAPH_GOOGLE_CLIENT_SECRET", "").strip()),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def enabled(provider: str) -> bool:
|
||||||
|
"""Whether this provider is configured. An unconfigured provider surfaces no UI
|
||||||
|
and every one of its routes bounces, so a half-set environment is inert rather
|
||||||
|
than broken."""
|
||||||
|
cid, secret = CREDENTIALS.get(provider, ("", ""))
|
||||||
|
return bool(cid and secret)
|
||||||
|
|
||||||
|
|
||||||
|
def _provider_or_404(name: str) -> Provider:
|
||||||
|
p = PROVIDERS.get(name)
|
||||||
|
if p is None:
|
||||||
|
raise HTTPException(status_code=404, detail="Unknown provider.")
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
# --- signed state ------------------------------------------------------------
|
||||||
|
def _b64(raw: bytes) -> str:
|
||||||
|
return base64.urlsafe_b64encode(raw).decode().rstrip("=")
|
||||||
|
|
||||||
|
|
||||||
|
def _unb64(s: str) -> bytes:
|
||||||
|
return base64.urlsafe_b64decode(s + "=" * (-len(s) % 4))
|
||||||
|
|
||||||
|
|
||||||
|
def _sign_state(uid: str | None, purpose: str, provider: str) -> str:
|
||||||
|
payload = _b64(json.dumps(
|
||||||
|
{"u": uid or "", "t": int(time.time()), "p": purpose, "pr": provider}
|
||||||
|
).encode())
|
||||||
|
mac = hmac.new(SECRET.encode(), payload.encode(), hashlib.sha256).hexdigest()[:32]
|
||||||
|
return f"{payload}.{mac}"
|
||||||
|
|
||||||
|
|
||||||
|
def _verify_state(state: str, purpose: str, provider: str) -> str | None:
|
||||||
|
"""The Thermograph user id this state was minted for, or None if it's missing,
|
||||||
|
tampered, expired, or was minted for a different purpose or provider.
|
||||||
|
|
||||||
|
A login state has no user id yet, so it verifies as ``""`` — falsy, but distinct
|
||||||
|
from the None that means reject. Callers must test against None, not
|
||||||
|
truthiness, or a valid login state reads as a failure.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
payload, mac = state.split(".", 1)
|
||||||
|
except (ValueError, AttributeError):
|
||||||
|
return None
|
||||||
|
expect = hmac.new(SECRET.encode(), payload.encode(), hashlib.sha256).hexdigest()[:32]
|
||||||
|
if not hmac.compare_digest(mac, expect):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
data = json.loads(_unb64(payload))
|
||||||
|
except (ValueError, json.JSONDecodeError):
|
||||||
|
return None
|
||||||
|
if time.time() - float(data.get("t", 0)) > _STATE_TTL:
|
||||||
|
return None
|
||||||
|
if data.get("p") != purpose:
|
||||||
|
return None
|
||||||
|
# States minted before the provider field existed are Discord's; they age out
|
||||||
|
# in _STATE_TTL, so this fallback only spans a deploy.
|
||||||
|
if data.get("pr", "discord") != provider:
|
||||||
|
return None
|
||||||
|
return data.get("u") or ""
|
||||||
|
|
||||||
|
|
||||||
|
# --- redirects ---------------------------------------------------------------
|
||||||
|
def _redirect_uri(request: Request, provider: Provider) -> str:
|
||||||
|
"""The callback URL, matched to how the app is actually reached (scheme/host +
|
||||||
|
base path). Must equal the redirect registered in the provider's console."""
|
||||||
|
proto = request.headers.get("x-forwarded-proto") or request.url.scheme
|
||||||
|
host = request.headers.get("host") or request.url.netloc
|
||||||
|
return f"{proto}://{host}{BASE}{provider.callback_path}"
|
||||||
|
|
||||||
|
|
||||||
|
def _authorize_redirect(request: Request, provider: Provider, scope: str,
|
||||||
|
state: str) -> RedirectResponse:
|
||||||
|
cid, _ = CREDENTIALS[provider.name]
|
||||||
|
params = {
|
||||||
|
"client_id": cid,
|
||||||
|
"redirect_uri": _redirect_uri(request, provider),
|
||||||
|
"response_type": "code",
|
||||||
|
"scope": scope,
|
||||||
|
"state": state,
|
||||||
|
**provider.extra_authorize,
|
||||||
|
}
|
||||||
|
return RedirectResponse(
|
||||||
|
url=f"{provider.authorize_url}?{urllib.parse.urlencode(params)}", status_code=303)
|
||||||
|
|
||||||
|
|
||||||
|
def _account_redirect(status: str, provider: str) -> RedirectResponse:
|
||||||
|
"""Back to the alerts page, which surfaces the outcome as a toast; 303 so the
|
||||||
|
browser issues a GET after the callback.
|
||||||
|
|
||||||
|
Emits `discord=` as well for Discord, because the frontend deploys
|
||||||
|
independently of this service and the one in production reads that parameter.
|
||||||
|
Drop it once no deployed frontend predates `oauth=`."""
|
||||||
|
params = {"oauth": status, "provider": provider}
|
||||||
|
if provider == "discord":
|
||||||
|
params["discord"] = status
|
||||||
|
return RedirectResponse(url=f"{BASE}/alerts?{urllib.parse.urlencode(params)}",
|
||||||
|
status_code=303)
|
||||||
|
|
||||||
|
|
||||||
|
# --- provider API -------------------------------------------------------------
|
||||||
|
async def _fetch_profile(request: Request, provider: Provider, code: str) -> dict | None:
|
||||||
|
"""Trade the authorization code for the provider's user object, or None if any
|
||||||
|
step fails. Callers treat None as a graceful error, never a crash."""
|
||||||
|
cid, secret = CREDENTIALS[provider.name]
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
|
||||||
|
tok = await client.post(provider.token_url, data={
|
||||||
|
"client_id": cid,
|
||||||
|
"client_secret": secret,
|
||||||
|
"grant_type": "authorization_code",
|
||||||
|
"code": code,
|
||||||
|
"redirect_uri": _redirect_uri(request, provider),
|
||||||
|
}, headers={"Content-Type": "application/x-www-form-urlencoded"})
|
||||||
|
if tok.status_code >= 300:
|
||||||
|
return None
|
||||||
|
access = tok.json().get("access_token")
|
||||||
|
if not access:
|
||||||
|
return None
|
||||||
|
me = await client.get(provider.userinfo_url,
|
||||||
|
headers={"Authorization": f"Bearer {access}"})
|
||||||
|
if me.status_code >= 300:
|
||||||
|
return None
|
||||||
|
profile = me.json()
|
||||||
|
except Exception: # noqa: BLE001 - any network/parse failure is a graceful error
|
||||||
|
return None
|
||||||
|
return profile if isinstance(profile, dict) else None
|
||||||
|
|
||||||
|
|
||||||
|
# --- persistence --------------------------------------------------------------
|
||||||
|
async def _account_for(session: AsyncSession, provider: str,
|
||||||
|
subject: str) -> OAuthAccount | None:
|
||||||
|
return (await session.execute(
|
||||||
|
select(OAuthAccount).where(OAuthAccount.provider == provider,
|
||||||
|
OAuthAccount.subject == subject)
|
||||||
|
)).scalar_one_or_none()
|
||||||
|
|
||||||
|
|
||||||
|
async def _link_row(session: AsyncSession, user_id, provider: str, subject: str,
|
||||||
|
email: str | None) -> None:
|
||||||
|
"""Bind (provider, subject) to a user, plus the Discord-only side effect.
|
||||||
|
|
||||||
|
discord_id is a *delivery* address, not an identity — notify.py DMs it — so the
|
||||||
|
Discord link has to write it as well as the oauth_account row. Turning DMs on
|
||||||
|
here mirrors the original behaviour: linking is an active opt-in, and the user
|
||||||
|
can mute it (POST /discord/dm) while staying linked.
|
||||||
|
"""
|
||||||
|
session.add(OAuthAccount(user_id=user_id, provider=provider, subject=subject,
|
||||||
|
email=email or None))
|
||||||
|
if provider == "discord":
|
||||||
|
await session.execute(
|
||||||
|
update(User).where(User.id == user_id)
|
||||||
|
.values(discord_id=subject, discord_dm=True))
|
||||||
|
await session.commit()
|
||||||
|
|
||||||
|
|
||||||
|
async def _create_user(user_manager, email: str) -> User:
|
||||||
|
"""Create an account for an identity that matched none of ours.
|
||||||
|
|
||||||
|
Deliberately goes around ``UserManager.create()``: there is no password to
|
||||||
|
validate, and its ``on_after_register`` mails a confirmation link for an address
|
||||||
|
the provider has already verified. The generated password exists only because
|
||||||
|
``hashed_password`` is NOT NULL — nobody ever sees it, which is what
|
||||||
|
``oauth_only`` records.
|
||||||
|
"""
|
||||||
|
password = user_manager.password_helper.generate()
|
||||||
|
return await user_manager.user_db.create({
|
||||||
|
"email": email,
|
||||||
|
"hashed_password": user_manager.password_helper.hash(password),
|
||||||
|
"is_active": True,
|
||||||
|
"is_superuser": False,
|
||||||
|
"is_verified": True,
|
||||||
|
"oauth_only": True,
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
# --- routes -------------------------------------------------------------------
|
||||||
|
@router.get("/oauth/config")
|
||||||
|
async def oauth_config():
|
||||||
|
"""Which providers this server can actually complete a flow with. The sign-in
|
||||||
|
modal and account menu render from this, so an unconfigured provider shows no
|
||||||
|
dead-end UI and configuring one later makes it appear on its own."""
|
||||||
|
return {"providers": [
|
||||||
|
{"name": p.name, "label": p.label, "enabled": enabled(p.name)}
|
||||||
|
for p in PROVIDERS.values()
|
||||||
|
]}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/oauth/{provider}/link/start")
|
||||||
|
async def link_start(provider: str, request: Request, user=Depends(current_active_user)):
|
||||||
|
p = _provider_or_404(provider)
|
||||||
|
if not enabled(provider):
|
||||||
|
return _account_redirect("unavailable", provider)
|
||||||
|
return _authorize_redirect(request, p, p.scope_link,
|
||||||
|
_sign_state(str(user.id), "link", provider))
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/oauth/{provider}/login/start")
|
||||||
|
async def login_start(provider: str, request: Request,
|
||||||
|
user=Depends(current_user_optional)):
|
||||||
|
"""Sign in with a provider. Unauthenticated by design — this is how you get a
|
||||||
|
session, not something you do with one."""
|
||||||
|
p = _provider_or_404(provider)
|
||||||
|
if not enabled(provider):
|
||||||
|
return _account_redirect("unavailable", provider)
|
||||||
|
# Someone already signed in who lands here wants to connect, not to be switched
|
||||||
|
# into whichever account the external identity resolves to.
|
||||||
|
if user is not None:
|
||||||
|
return _authorize_redirect(request, p, p.scope_link,
|
||||||
|
_sign_state(str(user.id), "link", provider))
|
||||||
|
return _authorize_redirect(request, p, p.scope_login,
|
||||||
|
_sign_state(None, "login", provider))
|
||||||
|
|
||||||
|
|
||||||
|
async def _callback(provider: str, request: Request, user, session, user_manager,
|
||||||
|
access_token_db) -> RedirectResponse:
|
||||||
|
p = _provider_or_404(provider)
|
||||||
|
if not enabled(provider):
|
||||||
|
return _account_redirect("unavailable", provider)
|
||||||
|
# User declined on the provider's screen, or the state doesn't check out.
|
||||||
|
code = request.query_params.get("code")
|
||||||
|
state = request.query_params.get("state", "")
|
||||||
|
if not code:
|
||||||
|
return _account_redirect("cancelled", provider)
|
||||||
|
|
||||||
|
# Which flow minted this state? Purpose and provider are both inside the HMAC,
|
||||||
|
# so a state only verifies under the flow and provider it was signed for.
|
||||||
|
uid = _verify_state(state, "link", provider)
|
||||||
|
if uid is not None:
|
||||||
|
return await _finish_link(p, request, code, uid, user, session)
|
||||||
|
if _verify_state(state, "login", provider) is not None:
|
||||||
|
return await _finish_login(p, request, code, session, user_manager,
|
||||||
|
access_token_db)
|
||||||
|
return _account_redirect("error", provider)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/oauth/{provider}/callback")
|
||||||
|
async def oauth_callback(
|
||||||
|
provider: str,
|
||||||
|
request: Request,
|
||||||
|
user=Depends(current_user_optional),
|
||||||
|
session: AsyncSession = Depends(get_async_session),
|
||||||
|
user_manager=Depends(get_user_manager),
|
||||||
|
access_token_db=Depends(get_access_token_db),
|
||||||
|
):
|
||||||
|
return await _callback(provider, request, user, session, user_manager,
|
||||||
|
access_token_db)
|
||||||
|
|
||||||
|
|
||||||
|
async def _finish_link(p: Provider, request: Request, code: str, uid: str, user,
|
||||||
|
session: AsyncSession) -> RedirectResponse:
|
||||||
|
# Linking always acts on the person actually logged in, and only on the one the
|
||||||
|
# state was minted for.
|
||||||
|
if user is None or str(user.id) != uid:
|
||||||
|
return _account_redirect("error", p.name)
|
||||||
|
profile = await _fetch_profile(request, p, code)
|
||||||
|
if profile is None:
|
||||||
|
return _account_redirect("error", p.name)
|
||||||
|
subject, email, _ = p.parse(profile)
|
||||||
|
if not subject:
|
||||||
|
return _account_redirect("error", p.name)
|
||||||
|
# (provider, subject) is UNIQUE, so without this the insert would raise an
|
||||||
|
# IntegrityError and 500 instead of explaining itself.
|
||||||
|
owner = await _account_for(session, p.name, subject)
|
||||||
|
if owner is not None:
|
||||||
|
return _account_redirect("linked" if owner.user_id == user.id else "taken", p.name)
|
||||||
|
# And (user_id, provider) is UNIQUE: this account already has a different
|
||||||
|
# identity from this provider attached.
|
||||||
|
existing = (await session.execute(
|
||||||
|
select(OAuthAccount).where(OAuthAccount.user_id == user.id,
|
||||||
|
OAuthAccount.provider == p.name)
|
||||||
|
)).scalar_one_or_none()
|
||||||
|
if existing is not None:
|
||||||
|
return _account_redirect("already", p.name)
|
||||||
|
await _link_row(session, user.id, p.name, subject, email)
|
||||||
|
return _account_redirect("linked", p.name)
|
||||||
|
|
||||||
|
|
||||||
|
async def _finish_login(p: Provider, request: Request, code: str,
|
||||||
|
session: AsyncSession, user_manager,
|
||||||
|
access_token_db) -> RedirectResponse:
|
||||||
|
"""Resolve the external identity to an account and sign in as it.
|
||||||
|
|
||||||
|
Deliberately ignores any session already present. A login state carries no user
|
||||||
|
id — it can't, there is no user yet — so it is replayable against whoever
|
||||||
|
happens to be signed in. Treating it as a link would therefore be a
|
||||||
|
forced-linking takeover: an attacker who gets a victim to open this callback
|
||||||
|
with a code from the *attacker's* provider account would attach their identity
|
||||||
|
to the victim's account, then sign in as them. Only link/start, whose state is
|
||||||
|
bound to a specific user id, may link. The worst a replayed login state can do
|
||||||
|
is sign someone into the attacker's own account.
|
||||||
|
"""
|
||||||
|
profile = await _fetch_profile(request, p, code)
|
||||||
|
if profile is None:
|
||||||
|
return _account_redirect("error", p.name)
|
||||||
|
subject, email, verified = p.parse(profile)
|
||||||
|
if not subject:
|
||||||
|
return _account_redirect("error", p.name)
|
||||||
|
|
||||||
|
created = False
|
||||||
|
acct = await _account_for(session, p.name, subject)
|
||||||
|
if acct is not None:
|
||||||
|
user = await user_manager.user_db.get(acct.user_id)
|
||||||
|
if user is None:
|
||||||
|
return _account_redirect("error", p.name)
|
||||||
|
else:
|
||||||
|
# Nothing carries this identity, so fall back to the email — the only other
|
||||||
|
# identity the provider gives us, and only when it vouches for it.
|
||||||
|
if not email:
|
||||||
|
return _account_redirect("noemail", p.name)
|
||||||
|
if not verified:
|
||||||
|
return _account_redirect("unverified", p.name)
|
||||||
|
user = await user_manager.user_db.get_by_email(email)
|
||||||
|
if user is None:
|
||||||
|
user = await _create_user(user_manager, email)
|
||||||
|
await _link_row(session, user.id, p.name, subject, email)
|
||||||
|
audit.log_activity("auth.register", {"user_id": str(user.id), "via": p.name})
|
||||||
|
created = True
|
||||||
|
elif not user.is_active:
|
||||||
|
# Checked before the link, not just before the session: a deactivated
|
||||||
|
# account must not quietly acquire an identity on the way out.
|
||||||
|
return _account_redirect("inactive", p.name)
|
||||||
|
else:
|
||||||
|
# That address belongs to an account already tied to a different
|
||||||
|
# identity from this provider; connecting would silently move it.
|
||||||
|
existing = (await session.execute(
|
||||||
|
select(OAuthAccount).where(OAuthAccount.user_id == user.id,
|
||||||
|
OAuthAccount.provider == p.name)
|
||||||
|
)).scalar_one_or_none()
|
||||||
|
if existing is not None:
|
||||||
|
return _account_redirect("mismatch", p.name)
|
||||||
|
await _link_row(session, user.id, p.name, subject, email)
|
||||||
|
if not user.is_active:
|
||||||
|
return _account_redirect("inactive", p.name)
|
||||||
|
response = _account_redirect("created" if created else "signedin", p.name)
|
||||||
|
return await attach_session(response, user, user_manager, access_token_db, request)
|
||||||
|
|
||||||
|
|
||||||
|
async def _unlink(provider: str, user, session: AsyncSession) -> None:
|
||||||
|
p = _provider_or_404(provider)
|
||||||
|
linked = (await session.execute(
|
||||||
|
select(func.count()).select_from(OAuthAccount)
|
||||||
|
.where(OAuthAccount.user_id == user.id)
|
||||||
|
)).scalar_one()
|
||||||
|
mine = await session.execute(
|
||||||
|
select(OAuthAccount).where(OAuthAccount.user_id == user.id,
|
||||||
|
OAuthAccount.provider == p.name))
|
||||||
|
if mine.scalar_one_or_none() is None:
|
||||||
|
return # already unlinked; idempotent
|
||||||
|
if user.oauth_only and linked <= 1:
|
||||||
|
# No password anyone has ever seen, and this is the last way in. No
|
||||||
|
# reset-password route is mounted to recover from it either.
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=409,
|
||||||
|
detail=f"This account was created with {p.label} and has no password, "
|
||||||
|
"so unlinking would leave no way to sign in.",
|
||||||
|
)
|
||||||
|
await session.execute(
|
||||||
|
delete(OAuthAccount).where(OAuthAccount.user_id == user.id,
|
||||||
|
OAuthAccount.provider == p.name))
|
||||||
|
if p.name == "discord":
|
||||||
|
await session.execute(
|
||||||
|
update(User).where(User.id == user.id)
|
||||||
|
.values(discord_id=None, discord_dm=False))
|
||||||
|
await session.commit()
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/oauth/{provider}/unlink", status_code=204)
|
||||||
|
async def unlink(
|
||||||
|
provider: str,
|
||||||
|
user=Depends(current_active_user),
|
||||||
|
session: AsyncSession = Depends(get_async_session),
|
||||||
|
):
|
||||||
|
await _unlink(provider, user, session)
|
||||||
|
|
@ -8,7 +8,7 @@ import uuid
|
||||||
from typing import Literal
|
from typing import Literal
|
||||||
|
|
||||||
from fastapi_users import schemas
|
from fastapi_users import schemas
|
||||||
from pydantic import BaseModel, Field, field_validator
|
from pydantic import BaseModel, Field, computed_field, field_validator
|
||||||
|
|
||||||
from data import grading
|
from data import grading
|
||||||
|
|
||||||
|
|
@ -16,14 +16,34 @@ from data import grading
|
||||||
ALLOWED_METRICS = tuple(grading.CLIMO_METRICS)
|
ALLOWED_METRICS = tuple(grading.CLIMO_METRICS)
|
||||||
DEFAULT_METRICS = ["tmax", "feels", "precip"]
|
DEFAULT_METRICS = ["tmax", "feels", "precip"]
|
||||||
|
|
||||||
|
# Bookmark limits, shared by the single-create and bulk-import paths.
|
||||||
|
BOOKMARK_LABEL_MAX_LEN = 80
|
||||||
|
BOOKMARK_MAX_PER_USER = 200
|
||||||
|
BOOKMARK_IMPORT_MAX_ITEMS = 200
|
||||||
|
|
||||||
|
|
||||||
# --- users (fastapi-users) ---------------------------------------------------
|
# --- users (fastapi-users) ---------------------------------------------------
|
||||||
class UserRead(schemas.BaseUser[uuid.UUID]):
|
class UserRead(schemas.BaseUser[uuid.UUID]):
|
||||||
display_name: str | None = None
|
display_name: str | None = None
|
||||||
# Present so the frontend can show linked/unlinked state; set via the OAuth
|
# Present so the frontend can show linked/unlinked state; set via the OAuth
|
||||||
# flow (discord_link.py), not editable through the profile.
|
# flow (accounts/oauth.py), not editable through the profile.
|
||||||
discord_id: str | None = None
|
discord_id: str | None = None
|
||||||
discord_dm: bool = False
|
discord_dm: bool = False
|
||||||
|
# True when an OAuth provider is the account's only credential, so the UI can
|
||||||
|
# explain why unlinking the last one is refused rather than just failing.
|
||||||
|
oauth_only: bool = False
|
||||||
|
# Provider names currently linked, e.g. ["discord", "google"] — what the
|
||||||
|
# account menu renders its connected-accounts section from.
|
||||||
|
oauth_providers: list[str] = []
|
||||||
|
|
||||||
|
# Superseded by oauth_only. Kept because frontend and backend deploy
|
||||||
|
# independently: the frontend in production reads this field, and dropping it
|
||||||
|
# would break its account menu the moment this backend shipped ahead. Remove
|
||||||
|
# once no deployed frontend predates oauth_only.
|
||||||
|
@computed_field
|
||||||
|
@property
|
||||||
|
def discord_only(self) -> bool:
|
||||||
|
return self.oauth_only
|
||||||
|
|
||||||
|
|
||||||
class UserCreate(schemas.BaseUserCreate):
|
class UserCreate(schemas.BaseUserCreate):
|
||||||
|
|
@ -132,3 +152,69 @@ class PushSubscriptionIn(BaseModel):
|
||||||
|
|
||||||
class PushUnsubscribeIn(BaseModel):
|
class PushUnsubscribeIn(BaseModel):
|
||||||
endpoint: str
|
endpoint: str
|
||||||
|
|
||||||
|
|
||||||
|
# --- bookmarks ----------------------------------------------------------------
|
||||||
|
def _check_label(v: str) -> str:
|
||||||
|
v = v.strip()
|
||||||
|
if not v:
|
||||||
|
raise ValueError("label is required")
|
||||||
|
if len(v) > BOOKMARK_LABEL_MAX_LEN:
|
||||||
|
raise ValueError(f"label must be at most {BOOKMARK_LABEL_MAX_LEN} characters")
|
||||||
|
return v
|
||||||
|
|
||||||
|
|
||||||
|
class BookmarkIn(BaseModel):
|
||||||
|
lat: float = Field(ge=-90, le=90)
|
||||||
|
lon: float = Field(ge=-180, le=180)
|
||||||
|
label: str
|
||||||
|
|
||||||
|
@field_validator("label")
|
||||||
|
@classmethod
|
||||||
|
def _label(cls, v):
|
||||||
|
return _check_label(v)
|
||||||
|
|
||||||
|
|
||||||
|
class BookmarkPatch(BaseModel):
|
||||||
|
label: str
|
||||||
|
|
||||||
|
@field_validator("label")
|
||||||
|
@classmethod
|
||||||
|
def _label(cls, v):
|
||||||
|
return _check_label(v)
|
||||||
|
|
||||||
|
|
||||||
|
class BookmarkOut(BaseModel):
|
||||||
|
id: int
|
||||||
|
cell_id: str
|
||||||
|
label: str
|
||||||
|
lat: float
|
||||||
|
lon: float
|
||||||
|
created_at: float
|
||||||
|
|
||||||
|
model_config = {"from_attributes": True}
|
||||||
|
|
||||||
|
|
||||||
|
class BookmarkList(BaseModel):
|
||||||
|
bookmarks: list[BookmarkOut]
|
||||||
|
|
||||||
|
|
||||||
|
class BookmarkImportItem(BaseModel):
|
||||||
|
lat: float = Field(ge=-90, le=90)
|
||||||
|
lon: float = Field(ge=-180, le=180)
|
||||||
|
label: str
|
||||||
|
|
||||||
|
@field_validator("label")
|
||||||
|
@classmethod
|
||||||
|
def _label(cls, v):
|
||||||
|
return _check_label(v)
|
||||||
|
|
||||||
|
|
||||||
|
class BookmarkImportIn(BaseModel):
|
||||||
|
items: list[BookmarkImportItem] = Field(max_length=BOOKMARK_IMPORT_MAX_ITEMS)
|
||||||
|
|
||||||
|
|
||||||
|
class BookmarkImportOut(BaseModel):
|
||||||
|
imported: int
|
||||||
|
skipped: int
|
||||||
|
bookmarks: list[BookmarkOut]
|
||||||
|
|
|
||||||
|
|
@ -140,3 +140,23 @@ fastapi_users = FastAPIUsers[User, uuid.UUID](get_user_manager, [auth_backend])
|
||||||
# Dependencies handlers use to require / peek at the logged-in user.
|
# Dependencies handlers use to require / peek at the logged-in user.
|
||||||
current_active_user = fastapi_users.current_user(active=True)
|
current_active_user = fastapi_users.current_user(active=True)
|
||||||
current_user_optional = fastapi_users.current_user(active=True, optional=True)
|
current_user_optional = fastapi_users.current_user(active=True, optional=True)
|
||||||
|
|
||||||
|
|
||||||
|
async def attach_session(response, user, user_manager, access_token_db, request=None):
|
||||||
|
"""Log `user` in by copying a fresh session cookie onto `response`.
|
||||||
|
|
||||||
|
The library's routers own this for password login, but an OAuth callback has to
|
||||||
|
end in a *redirect* the browser follows, not the 204 the login router returns.
|
||||||
|
So mint the session the normal way and lift the Set-Cookie header off the
|
||||||
|
library's response rather than re-deriving the cookie's name/TTL/flags here —
|
||||||
|
duplicating those would let the two logins drift into issuing different cookies.
|
||||||
|
"""
|
||||||
|
strategy = get_database_strategy(access_token_db)
|
||||||
|
issued = await auth_backend.login(strategy, user)
|
||||||
|
for key, value in issued.raw_headers:
|
||||||
|
if key.lower() == b"set-cookie":
|
||||||
|
response.raw_headers.append((key, value))
|
||||||
|
# The routers call this hook; a hand-rolled login has to as well, or Discord
|
||||||
|
# sign-ins go missing from the audit log that every password login appears in.
|
||||||
|
await user_manager.on_after_login(user, request, response)
|
||||||
|
return response
|
||||||
|
|
|
||||||
50
backend/alembic/versions/0003_user_discord_only.py
Normal file
50
backend/alembic/versions/0003_user_discord_only.py
Normal file
|
|
@ -0,0 +1,50 @@
|
||||||
|
"""user.discord_only — mark accounts whose only credential is Discord
|
||||||
|
|
||||||
|
"Sign in with Discord" (notifications/discord_link.py) creates accounts for people
|
||||||
|
who never chose a password. ``hashed_password`` is NOT NULL, so those rows carry a
|
||||||
|
generated one nobody has ever seen; without a flag there is no way to tell such an
|
||||||
|
account apart from a password account that merely linked Discord later. That
|
||||||
|
distinction is load-bearing: unlinking Discord from a Discord-only account would
|
||||||
|
delete its only way in, and no reset-password router is mounted to recover it.
|
||||||
|
|
||||||
|
Conditional on purpose. Revision 0001 builds the schema with
|
||||||
|
``Base.metadata.create_all``, which reads the *current* models.py — so a database
|
||||||
|
created after this column was added to the ORM already has it, and an
|
||||||
|
unconditional ``add_column`` would fail there with a duplicate-column error. Only
|
||||||
|
a database stamped before this revision is missing it.
|
||||||
|
|
||||||
|
Revision ID: 0003_user_discord_only
|
||||||
|
Revises: 0002_climate_hypertables
|
||||||
|
Create Date: 2026-07-26
|
||||||
|
"""
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
revision = "0003_user_discord_only"
|
||||||
|
down_revision = "0002_climate_hypertables"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def _has_column() -> bool:
|
||||||
|
cols = sa.inspect(op.get_bind()).get_columns("user")
|
||||||
|
return any(c["name"] == "discord_only" for c in cols)
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
if _has_column():
|
||||||
|
return
|
||||||
|
# server_default (not just a Python-side default) so the backfill of existing
|
||||||
|
# rows happens in the same statement: every account that predates Discord
|
||||||
|
# login has a password, so False is correct for all of them.
|
||||||
|
op.add_column(
|
||||||
|
"user",
|
||||||
|
sa.Column("discord_only", sa.Boolean(), nullable=False,
|
||||||
|
server_default=sa.false()),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
if not _has_column():
|
||||||
|
return
|
||||||
|
op.drop_column("user", "discord_only")
|
||||||
45
backend/alembic/versions/0004_bookmarks.py
Normal file
45
backend/alembic/versions/0004_bookmarks.py
Normal file
|
|
@ -0,0 +1,45 @@
|
||||||
|
"""bookmark: saved locations
|
||||||
|
|
||||||
|
Adds the ``bookmark`` table backing the "bookmarked locations" feature — one row
|
||||||
|
per (user, grid cell), following the same shape as ``subscription``.
|
||||||
|
|
||||||
|
Conditional on purpose, like 0003_user_discord_only: revision 0001 builds the
|
||||||
|
schema from the *current* models.py via ``Base.metadata.create_all``, so a
|
||||||
|
database created after ``Bookmark`` was added to the ORM (fresh envs, most tests)
|
||||||
|
already has the table, and an unconditional ``create_table`` would fail there
|
||||||
|
with a duplicate-table error. Only a database stamped before this revision is
|
||||||
|
missing it.
|
||||||
|
|
||||||
|
Revision ID: 0004_bookmarks
|
||||||
|
Revises: 0003_user_discord_only
|
||||||
|
Create Date: 2026-07-26
|
||||||
|
"""
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
revision = "0004_bookmarks"
|
||||||
|
down_revision = "0003_user_discord_only"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def _has_table() -> bool:
|
||||||
|
return sa.inspect(op.get_bind()).has_table("bookmark")
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
if _has_table():
|
||||||
|
return
|
||||||
|
# Built from the live ORM table (accounts.models.Bookmark) rather than
|
||||||
|
# hand-duplicated column definitions, so this can never drift from models.py.
|
||||||
|
from accounts.models import Bookmark
|
||||||
|
|
||||||
|
Bookmark.__table__.create(bind=op.get_bind())
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
if not _has_table():
|
||||||
|
return
|
||||||
|
from accounts.models import Bookmark
|
||||||
|
|
||||||
|
Bookmark.__table__.drop(bind=op.get_bind())
|
||||||
104
backend/alembic/versions/0005_oauth_accounts.py
Normal file
104
backend/alembic/versions/0005_oauth_accounts.py
Normal file
|
|
@ -0,0 +1,104 @@
|
||||||
|
"""oauth_account table; user.discord_only -> user.oauth_only
|
||||||
|
|
||||||
|
Generalises "sign in with Discord" to any provider (see accounts/oauth.py). Three
|
||||||
|
steps, each conditional so the revision is safe on a database created before or
|
||||||
|
after the ORM gained these definitions — revision 0001 builds the schema with
|
||||||
|
``Base.metadata.create_all`` off *live* model metadata, so a freshly created
|
||||||
|
database already has both the table and the renamed column, and an unconditional
|
||||||
|
DDL would fail there with duplicate-object errors.
|
||||||
|
|
||||||
|
The backfill is the load-bearing step. Discord identities used to live in
|
||||||
|
``user.discord_id``, and account resolution now keys on ``oauth_account``; without
|
||||||
|
copying the existing rows across, every already-linked user would silently stop
|
||||||
|
being recognised at login and would get a *second* account created on their next
|
||||||
|
sign-in. ``user.discord_id`` is deliberately left in place — it is the DM delivery
|
||||||
|
address notify.py reads, not merely an identity.
|
||||||
|
|
||||||
|
Revision ID: 0005_oauth_accounts
|
||||||
|
Revises: 0004_bookmarks
|
||||||
|
Create Date: 2026-07-26
|
||||||
|
"""
|
||||||
|
import time
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
# The same UUID type the ORM uses for user.id — a plain sa.Uuid renders differently
|
||||||
|
# on SQLite and would not match the column create_all produces.
|
||||||
|
from fastapi_users_db_sqlalchemy.generics import GUID
|
||||||
|
|
||||||
|
revision = "0005_oauth_accounts"
|
||||||
|
down_revision = "0004_bookmarks"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def _columns(table: str) -> set[str]:
|
||||||
|
return {c["name"] for c in sa.inspect(op.get_bind()).get_columns(table)}
|
||||||
|
|
||||||
|
|
||||||
|
def _tables() -> set[str]:
|
||||||
|
return set(sa.inspect(op.get_bind()).get_table_names())
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
cols = _columns("user")
|
||||||
|
|
||||||
|
# 1. Rename the flag. It no longer means "Discord created this account", it
|
||||||
|
# means "no password its owner has ever seen, whichever provider made it".
|
||||||
|
if "oauth_only" in cols:
|
||||||
|
# A database built by 0001 already has oauth_only (create_all reads *current*
|
||||||
|
# metadata), and 0003 then re-added an empty discord_only beside it, because
|
||||||
|
# 0003 tests for the name it knew. Drop that vestige, or a fresh database
|
||||||
|
# carries a column an upgraded one does not — schema drift that would only
|
||||||
|
# surface much later. Safe: on this path the column was created empty
|
||||||
|
# moments ago and nothing has ever read it.
|
||||||
|
if "discord_only" in cols:
|
||||||
|
op.drop_column("user", "discord_only")
|
||||||
|
elif "discord_only" in cols:
|
||||||
|
# The real upgrade path: a deployed database carrying live values. Renamed,
|
||||||
|
# never dropped, so nothing is lost.
|
||||||
|
op.alter_column("user", "discord_only", new_column_name="oauth_only")
|
||||||
|
else:
|
||||||
|
# Database predates even discord_only (never deployed with it).
|
||||||
|
op.add_column("user", sa.Column("oauth_only", sa.Boolean(), nullable=False,
|
||||||
|
server_default=sa.false()))
|
||||||
|
|
||||||
|
# 2. The identity table.
|
||||||
|
if "oauth_account" not in _tables():
|
||||||
|
op.create_table(
|
||||||
|
"oauth_account",
|
||||||
|
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
|
||||||
|
sa.Column("user_id", GUID(), sa.ForeignKey("user.id", ondelete="CASCADE"),
|
||||||
|
nullable=False),
|
||||||
|
sa.Column("provider", sa.String(length=32), nullable=False),
|
||||||
|
sa.Column("subject", sa.String(length=64), nullable=False),
|
||||||
|
sa.Column("email", sa.String(length=320), nullable=True),
|
||||||
|
sa.Column("created_at", sa.Float(), nullable=False),
|
||||||
|
sa.UniqueConstraint("provider", "subject", name="uq_oauth_provider_subject"),
|
||||||
|
sa.UniqueConstraint("user_id", "provider", name="uq_oauth_user_provider"),
|
||||||
|
)
|
||||||
|
op.create_index("idx_oauth_user", "oauth_account", ["user_id"])
|
||||||
|
|
||||||
|
# 3. Backfill every existing Discord link. Guarded by NOT EXISTS so re-running
|
||||||
|
# (or running after create_all already produced the table) cannot violate
|
||||||
|
# either unique constraint.
|
||||||
|
op.execute(sa.text("""
|
||||||
|
INSERT INTO oauth_account (user_id, provider, subject, email, created_at)
|
||||||
|
SELECT u.id, 'discord', u.discord_id, u.email, :now
|
||||||
|
FROM "user" u
|
||||||
|
WHERE u.discord_id IS NOT NULL
|
||||||
|
AND NOT EXISTS (
|
||||||
|
SELECT 1 FROM oauth_account o
|
||||||
|
WHERE o.provider = 'discord'
|
||||||
|
AND (o.subject = u.discord_id OR o.user_id = u.id)
|
||||||
|
)
|
||||||
|
""").bindparams(now=time.time()))
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
if "oauth_account" in _tables():
|
||||||
|
op.drop_index("idx_oauth_user", table_name="oauth_account")
|
||||||
|
op.drop_table("oauth_account")
|
||||||
|
cols = _columns("user")
|
||||||
|
if "oauth_only" in cols and "discord_only" not in cols:
|
||||||
|
op.alter_column("user", "oauth_only", new_column_name="discord_only")
|
||||||
|
|
@ -1,172 +1,67 @@
|
||||||
"""Link a Thermograph account to a Discord account via OAuth2 (identify scope).
|
"""Discord's legacy /discord/* auth routes, plus the DM opt-in toggle.
|
||||||
|
|
||||||
The Discord user id we store here is the durable key a later feature (DM alerts)
|
The OAuth flow itself moved to ``accounts/oauth.py`` when Google was added — this
|
||||||
uses to reach the person. The flow is the standard authorization-code grant:
|
module holds no auth logic, only the paths that have to keep answering:
|
||||||
|
|
||||||
/link/start -> redirect the logged-in user to Discord's consent screen
|
* **/discord/link/callback is registered in Discord's developer portal.** Changing
|
||||||
/link/callback -> exchange the code, read the Discord user id, store it
|
it would need an operator to edit the portal before any Discord login worked, so
|
||||||
/unlink -> forget it
|
it stays exactly where it is and forwards to the shared handler.
|
||||||
|
* The other /discord/* routes are what the currently-deployed frontend calls.
|
||||||
|
Frontend and backend deploy independently (see the root CLAUDE.md), so a newer
|
||||||
|
backend must keep serving an older frontend. They can go once no deployed
|
||||||
|
frontend predates /oauth/*.
|
||||||
|
|
||||||
`state` is signed with the app's auth secret (stdlib hmac, no new dependency) and
|
``POST /discord/dm`` is not legacy and is not moving: it toggles alert delivery,
|
||||||
carries the Thermograph user id, so the callback can't be replayed or bound to a
|
which is a notifications concern, not an authentication one.
|
||||||
different account. Everything requires an active session, so linking always acts on
|
|
||||||
the person who is actually logged in.
|
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import base64
|
|
||||||
import hashlib
|
|
||||||
import hmac
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import time
|
|
||||||
import urllib.parse
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
from fastapi import APIRouter, Depends, Request
|
from fastapi import APIRouter, Depends, Request
|
||||||
from fastapi.responses import RedirectResponse
|
|
||||||
from pydantic import BaseModel
|
from pydantic import BaseModel
|
||||||
from sqlalchemy import update
|
from sqlalchemy import update
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
|
from accounts import oauth
|
||||||
from accounts.db import get_async_session
|
from accounts.db import get_async_session
|
||||||
from accounts.models import User
|
from accounts.models import User
|
||||||
from accounts.users import SECRET, current_active_user
|
from accounts.users import (
|
||||||
|
current_active_user,
|
||||||
|
current_user_optional,
|
||||||
|
get_access_token_db,
|
||||||
|
get_user_manager,
|
||||||
|
)
|
||||||
|
|
||||||
router = APIRouter(tags=["discord"])
|
router = APIRouter(tags=["discord"])
|
||||||
|
|
||||||
CLIENT_ID = os.environ.get("THERMOGRAPH_DISCORD_APP_ID", "").strip()
|
|
||||||
CLIENT_SECRET = os.environ.get("THERMOGRAPH_DISCORD_CLIENT_SECRET", "").strip()
|
|
||||||
BASE = os.environ.get("THERMOGRAPH_BASE", "/thermograph").rstrip("/")
|
|
||||||
|
|
||||||
_AUTHORIZE = "https://discord.com/api/oauth2/authorize"
|
|
||||||
_TOKEN = "https://discord.com/api/oauth2/token"
|
|
||||||
_ME = "https://discord.com/api/users/@me"
|
|
||||||
|
|
||||||
# How long a start->callback round trip may take before the signed state expires.
|
|
||||||
_STATE_TTL = 600
|
|
||||||
_TIMEOUT = httpx.Timeout(10.0)
|
|
||||||
|
|
||||||
|
|
||||||
def enabled() -> bool:
|
|
||||||
return bool(CLIENT_ID and CLIENT_SECRET)
|
|
||||||
|
|
||||||
|
|
||||||
# --- signed state ------------------------------------------------------------
|
|
||||||
def _b64(raw: bytes) -> str:
|
|
||||||
return base64.urlsafe_b64encode(raw).decode().rstrip("=")
|
|
||||||
|
|
||||||
|
|
||||||
def _unb64(s: str) -> bytes:
|
|
||||||
return base64.urlsafe_b64decode(s + "=" * (-len(s) % 4))
|
|
||||||
|
|
||||||
|
|
||||||
def _sign_state(uid: str) -> str:
|
|
||||||
payload = _b64(json.dumps({"u": uid, "t": int(time.time())}).encode())
|
|
||||||
mac = hmac.new(SECRET.encode(), payload.encode(), hashlib.sha256).hexdigest()[:32]
|
|
||||||
return f"{payload}.{mac}"
|
|
||||||
|
|
||||||
|
|
||||||
def _verify_state(state: str) -> str | None:
|
|
||||||
"""The Thermograph user id the state was minted for, or None if it's missing,
|
|
||||||
tampered, or older than the TTL."""
|
|
||||||
try:
|
|
||||||
payload, mac = state.split(".", 1)
|
|
||||||
except (ValueError, AttributeError):
|
|
||||||
return None
|
|
||||||
expect = hmac.new(SECRET.encode(), payload.encode(), hashlib.sha256).hexdigest()[:32]
|
|
||||||
if not hmac.compare_digest(mac, expect):
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
data = json.loads(_unb64(payload))
|
|
||||||
except (ValueError, json.JSONDecodeError):
|
|
||||||
return None
|
|
||||||
if time.time() - float(data.get("t", 0)) > _STATE_TTL:
|
|
||||||
return None
|
|
||||||
return data.get("u")
|
|
||||||
|
|
||||||
|
|
||||||
def _redirect_uri(request: Request) -> str:
|
|
||||||
"""The callback URL, matched to how the app is actually reached (scheme/host +
|
|
||||||
base path). Must equal the redirect registered in the Discord portal."""
|
|
||||||
proto = request.headers.get("x-forwarded-proto") or request.url.scheme
|
|
||||||
host = request.headers.get("host") or request.url.netloc
|
|
||||||
return f"{proto}://{host}{BASE}/api/v2/discord/link/callback"
|
|
||||||
|
|
||||||
|
|
||||||
def _account_redirect(status: str) -> RedirectResponse:
|
|
||||||
# Back to the alerts page, which surfaces the link state; 303 so the browser
|
|
||||||
# issues a GET after the callback.
|
|
||||||
return RedirectResponse(url=f"{BASE}/subscriptions?discord={status}", status_code=303)
|
|
||||||
|
|
||||||
|
|
||||||
# --- routes ------------------------------------------------------------------
|
|
||||||
@router.get("/discord/config")
|
@router.get("/discord/config")
|
||||||
async def discord_config():
|
async def discord_config():
|
||||||
"""Whether Discord account-linking is configured on this server. The account
|
"""Superseded by /oauth/config, which reports every provider. Kept because the
|
||||||
menu uses it to show the "Link Discord" entry only when linking will actually
|
deployed frontend asks this one."""
|
||||||
work — so an unconfigured server surfaces no dead-end Discord UI, and enabling
|
return {"enabled": oauth.enabled("discord")}
|
||||||
it later (setting the OAuth env vars) makes the entry appear on its own."""
|
|
||||||
return {"enabled": enabled()}
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/discord/link/start")
|
@router.get("/discord/link/start")
|
||||||
async def link_start(request: Request, user=Depends(current_active_user)):
|
async def link_start(request: Request, user=Depends(current_active_user)):
|
||||||
if not enabled():
|
return await oauth.link_start("discord", request, user)
|
||||||
return _account_redirect("unavailable")
|
|
||||||
params = {
|
|
||||||
"client_id": CLIENT_ID,
|
@router.get("/discord/login/start")
|
||||||
"redirect_uri": _redirect_uri(request),
|
async def login_start(request: Request, user=Depends(current_user_optional)):
|
||||||
"response_type": "code",
|
return await oauth.login_start("discord", request, user)
|
||||||
"scope": "identify",
|
|
||||||
"state": _sign_state(str(user.id)),
|
|
||||||
"prompt": "consent",
|
|
||||||
}
|
|
||||||
return RedirectResponse(url=f"{_AUTHORIZE}?{urllib.parse.urlencode(params)}", status_code=303)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/discord/link/callback")
|
@router.get("/discord/link/callback")
|
||||||
async def link_callback(
|
async def link_callback(
|
||||||
request: Request,
|
request: Request,
|
||||||
user=Depends(current_active_user),
|
user=Depends(current_user_optional),
|
||||||
session: AsyncSession = Depends(get_async_session),
|
session: AsyncSession = Depends(get_async_session),
|
||||||
|
user_manager=Depends(get_user_manager),
|
||||||
|
access_token_db=Depends(get_access_token_db),
|
||||||
):
|
):
|
||||||
if not enabled():
|
"""The URL registered in Discord's portal. Both Discord flows land here."""
|
||||||
return _account_redirect("unavailable")
|
return await oauth._callback("discord", request, user, session, user_manager,
|
||||||
# User declined on Discord's screen, or the state doesn't check out.
|
access_token_db)
|
||||||
code = request.query_params.get("code")
|
|
||||||
state = request.query_params.get("state", "")
|
|
||||||
if not code:
|
|
||||||
return _account_redirect("cancelled")
|
|
||||||
if _verify_state(state) != str(user.id):
|
|
||||||
return _account_redirect("error")
|
|
||||||
try:
|
|
||||||
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
|
|
||||||
tok = await client.post(_TOKEN, data={
|
|
||||||
"client_id": CLIENT_ID,
|
|
||||||
"client_secret": CLIENT_SECRET,
|
|
||||||
"grant_type": "authorization_code",
|
|
||||||
"code": code,
|
|
||||||
"redirect_uri": _redirect_uri(request),
|
|
||||||
}, headers={"Content-Type": "application/x-www-form-urlencoded"})
|
|
||||||
if tok.status_code >= 300:
|
|
||||||
return _account_redirect("error")
|
|
||||||
access = tok.json().get("access_token")
|
|
||||||
me = await client.get(_ME, headers={"Authorization": f"Bearer {access}"})
|
|
||||||
if me.status_code >= 300:
|
|
||||||
return _account_redirect("error")
|
|
||||||
discord_id = str(me.json().get("id") or "")
|
|
||||||
except Exception: # noqa: BLE001 - any network/parse failure is a graceful error
|
|
||||||
return _account_redirect("error")
|
|
||||||
if not discord_id:
|
|
||||||
return _account_redirect("error")
|
|
||||||
# Linking is an active opt-in, so turn DM alerts on; the user can mute them
|
|
||||||
# (POST /discord/dm) while staying linked.
|
|
||||||
await session.execute(
|
|
||||||
update(User).where(User.id == user.id).values(discord_id=discord_id, discord_dm=True)
|
|
||||||
)
|
|
||||||
await session.commit()
|
|
||||||
return _account_redirect("linked")
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/discord/unlink", status_code=204)
|
@router.post("/discord/unlink", status_code=204)
|
||||||
|
|
@ -174,10 +69,7 @@ async def unlink(
|
||||||
user=Depends(current_active_user),
|
user=Depends(current_active_user),
|
||||||
session: AsyncSession = Depends(get_async_session),
|
session: AsyncSession = Depends(get_async_session),
|
||||||
):
|
):
|
||||||
await session.execute(
|
await oauth._unlink("discord", user, session)
|
||||||
update(User).where(User.id == user.id).values(discord_id=None, discord_dm=False)
|
|
||||||
)
|
|
||||||
await session.commit()
|
|
||||||
|
|
||||||
|
|
||||||
class DmToggle(BaseModel):
|
class DmToggle(BaseModel):
|
||||||
|
|
|
||||||
160
backend/tests/accounts/test_bookmarks.py
Normal file
160
backend/tests/accounts/test_bookmarks.py
Normal file
|
|
@ -0,0 +1,160 @@
|
||||||
|
"""Route-level tests for the bookmarks feature: CRUD, upsert-on-duplicate,
|
||||||
|
ownership, auth, bulk import (dedupe/counts), and the per-user cap. Uses a
|
||||||
|
throwaway accounts DB (conftest points THERMOGRAPH_ACCOUNTS_DB at a temp file),
|
||||||
|
same as test_api_accounts.py."""
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from web import app as appmod
|
||||||
|
from accounts import db
|
||||||
|
|
||||||
|
V2 = "/thermograph/api/v2"
|
||||||
|
PW = "supersecret123"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module", autouse=True)
|
||||||
|
def _tables():
|
||||||
|
# TestClient(app) (no context manager) skips the lifespan, so create the
|
||||||
|
# account tables directly against the temp DB.
|
||||||
|
db.Base.metadata.create_all(db.sync_engine)
|
||||||
|
|
||||||
|
|
||||||
|
def _client():
|
||||||
|
return TestClient(appmod.app)
|
||||||
|
|
||||||
|
|
||||||
|
def _login(client, email):
|
||||||
|
r = client.post(f"{V2}/auth/register", json={"email": email, "password": PW})
|
||||||
|
assert r.status_code in (201, 400) # 400 only if the email already exists
|
||||||
|
r = client.post(f"{V2}/auth/login", data={"username": email, "password": PW})
|
||||||
|
assert r.status_code == 204 # sets the session cookie on the client
|
||||||
|
|
||||||
|
|
||||||
|
def test_bookmarks_require_auth():
|
||||||
|
c = _client()
|
||||||
|
assert c.get(f"{V2}/bookmarks").status_code == 401
|
||||||
|
assert c.post(f"{V2}/bookmarks", json={"lat": 1, "lon": 1, "label": "x"}).status_code == 401
|
||||||
|
assert c.post(f"{V2}/bookmarks/import", json={"items": []}).status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_bookmark_crud():
|
||||||
|
c = _client()
|
||||||
|
_login(c, "bm-crud@example.com")
|
||||||
|
|
||||||
|
r = c.post(f"{V2}/bookmarks", json={"lat": 47.6, "lon": -122.3, "label": "Seattle"})
|
||||||
|
assert r.status_code == 201
|
||||||
|
bm = r.json()
|
||||||
|
assert bm["cell_id"] and bm["label"] == "Seattle"
|
||||||
|
assert bm["lat"] == 47.6 and bm["lon"] == -122.3
|
||||||
|
|
||||||
|
listed = c.get(f"{V2}/bookmarks").json()["bookmarks"]
|
||||||
|
assert len(listed) == 1 and listed[0]["id"] == bm["id"]
|
||||||
|
|
||||||
|
patched = c.patch(f"{V2}/bookmarks/{bm['id']}", json={"label": "Home"})
|
||||||
|
assert patched.status_code == 200
|
||||||
|
assert patched.json()["label"] == "Home"
|
||||||
|
|
||||||
|
assert c.delete(f"{V2}/bookmarks/{bm['id']}").status_code == 204
|
||||||
|
assert c.get(f"{V2}/bookmarks").json()["bookmarks"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_bookmark_upsert_on_duplicate():
|
||||||
|
c = _client()
|
||||||
|
_login(c, "bm-upsert@example.com")
|
||||||
|
body = {"lat": 33.4, "lon": -112.0, "label": "Phoenix"}
|
||||||
|
|
||||||
|
first = c.post(f"{V2}/bookmarks", json=body)
|
||||||
|
assert first.status_code == 201
|
||||||
|
bm_id = first.json()["id"]
|
||||||
|
|
||||||
|
# Re-bookmarking the same location (same cell) upserts the label — 200, not 409.
|
||||||
|
second = c.post(f"{V2}/bookmarks", json={**body, "label": "Phoenix Home"})
|
||||||
|
assert second.status_code == 200
|
||||||
|
assert second.json()["id"] == bm_id
|
||||||
|
assert second.json()["label"] == "Phoenix Home"
|
||||||
|
|
||||||
|
assert len(c.get(f"{V2}/bookmarks").json()["bookmarks"]) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_bookmark_validation():
|
||||||
|
c = _client()
|
||||||
|
_login(c, "bm-valid@example.com")
|
||||||
|
assert c.post(f"{V2}/bookmarks", json={"lat": 1, "lon": 1, "label": " "}).status_code == 422
|
||||||
|
assert c.post(f"{V2}/bookmarks", json={"lat": 1, "lon": 1, "label": "x" * 81}).status_code == 422
|
||||||
|
assert c.post(f"{V2}/bookmarks", json={"lat": 91, "lon": 1, "label": "x"}).status_code == 422
|
||||||
|
assert c.post(f"{V2}/bookmarks", json={"lat": 1, "lon": 181, "label": "x"}).status_code == 422
|
||||||
|
# a label that's only over-length due to surrounding whitespace is fine once trimmed
|
||||||
|
ok = c.post(f"{V2}/bookmarks", json={"lat": 2, "lon": 2, "label": " Trimmed "})
|
||||||
|
assert ok.status_code == 201 and ok.json()["label"] == "Trimmed"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ownership_is_404():
|
||||||
|
a, b = _client(), _client()
|
||||||
|
_login(a, "bm-owner-a@example.com")
|
||||||
|
_login(b, "bm-owner-b@example.com")
|
||||||
|
bm_id = a.post(f"{V2}/bookmarks", json={"lat": 40.7, "lon": -74.0, "label": "NYC"}).json()["id"]
|
||||||
|
|
||||||
|
# B cannot see, patch, or delete A's bookmark — 404, not 403 (no existence leak)
|
||||||
|
assert b.patch(f"{V2}/bookmarks/{bm_id}", json={"label": "Nope"}).status_code == 404
|
||||||
|
assert b.delete(f"{V2}/bookmarks/{bm_id}").status_code == 404
|
||||||
|
assert b.get(f"{V2}/bookmarks").json()["bookmarks"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_bookmark_import_dedupe_and_counts():
|
||||||
|
c = _client()
|
||||||
|
_login(c, "bm-import@example.com")
|
||||||
|
# Pre-existing bookmark at the Seattle cell, with a label import must NOT overwrite.
|
||||||
|
existing = c.post(
|
||||||
|
f"{V2}/bookmarks", json={"lat": 47.6, "lon": -122.3, "label": "Original"}
|
||||||
|
).json()
|
||||||
|
assert existing["label"] == "Original"
|
||||||
|
|
||||||
|
r = c.post(f"{V2}/bookmarks/import", json={"items": [
|
||||||
|
{"lat": 47.6, "lon": -122.3, "label": "Should Not Overwrite"}, # dup of existing cell
|
||||||
|
{"lat": 34.0, "lon": -118.2, "label": "LA"}, # new
|
||||||
|
{"lat": 34.0, "lon": -118.2, "label": "LA Dup"}, # dup within payload
|
||||||
|
{"lat": 51.5, "lon": -0.1, "label": "London"}, # new
|
||||||
|
]})
|
||||||
|
assert r.status_code == 200
|
||||||
|
body = r.json()
|
||||||
|
assert body["imported"] == 2
|
||||||
|
assert body["skipped"] == 2
|
||||||
|
assert len(body["bookmarks"]) == 3
|
||||||
|
|
||||||
|
by_label = {b["label"] for b in body["bookmarks"]}
|
||||||
|
assert by_label == {"Original", "LA", "London"} # existing label untouched
|
||||||
|
|
||||||
|
|
||||||
|
def test_bookmark_cap():
|
||||||
|
c = _client()
|
||||||
|
_login(c, "bm-cap@example.com")
|
||||||
|
for i in range(200):
|
||||||
|
r = c.post(f"{V2}/bookmarks", json={"lat": i * 0.1, "lon": 0.0, "label": f"spot-{i}"})
|
||||||
|
assert r.status_code == 201, r.text
|
||||||
|
|
||||||
|
over = c.post(f"{V2}/bookmarks", json={"lat": 89.0, "lon": 179.0, "label": "one too many"})
|
||||||
|
assert over.status_code == 409
|
||||||
|
|
||||||
|
assert len(c.get(f"{V2}/bookmarks").json()["bookmarks"]) == 200
|
||||||
|
|
||||||
|
|
||||||
|
def test_bookmark_import_stops_at_cap():
|
||||||
|
c = _client()
|
||||||
|
_login(c, "bm-import-cap@example.com")
|
||||||
|
for i in range(198):
|
||||||
|
r = c.post(f"{V2}/bookmarks", json={"lat": i * 0.1, "lon": 10.0, "label": f"spot-{i}"})
|
||||||
|
assert r.status_code == 201, r.text
|
||||||
|
|
||||||
|
r = c.post(f"{V2}/bookmarks/import", json={"items": [
|
||||||
|
{"lat": 60.0, "lon": 60.0, "label": "a"},
|
||||||
|
{"lat": 61.0, "lon": 61.0, "label": "b"},
|
||||||
|
{"lat": 62.0, "lon": 62.0, "label": "c"},
|
||||||
|
{"lat": 63.0, "lon": 63.0, "label": "d"},
|
||||||
|
{"lat": 64.0, "lon": 64.0, "label": "e"},
|
||||||
|
]})
|
||||||
|
assert r.status_code == 200
|
||||||
|
body = r.json()
|
||||||
|
# Only 2 slots remained before the 200 cap; the rest count as skipped.
|
||||||
|
assert body["imported"] == 2
|
||||||
|
assert body["skipped"] == 3
|
||||||
|
assert len(body["bookmarks"]) == 200
|
||||||
|
|
@ -9,6 +9,28 @@ from sqlalchemy import select
|
||||||
|
|
||||||
from accounts import db
|
from accounts import db
|
||||||
|
|
||||||
|
# The server-side ceiling these pools spend from, and the deployment shape that
|
||||||
|
# spends it. Kept here so a pool change that stops fitting is a failing test
|
||||||
|
# rather than a 2am "sorry, too many clients already". Mirrors max_connections in
|
||||||
|
# infra/deploy/db/init/20-tuning.sh. ONE Postgres instance serves prod and beta.
|
||||||
|
MAX_CONNECTIONS = 200
|
||||||
|
PROD_WEB_WORKERS = 4 # uvicorn workers per prod web replica
|
||||||
|
BETA_WEB_WORKERS = 2
|
||||||
|
ASYNC_ENGINES_PER_PROCESS = 2 # RW + RO, both built from _ASYNC_POOL_KWARGS
|
||||||
|
|
||||||
|
|
||||||
|
def _async_ceiling_per_process() -> int:
|
||||||
|
"""Connections one backend process can hold from its async pools at once."""
|
||||||
|
k = db._ASYNC_POOL_KWARGS
|
||||||
|
return ASYNC_ENGINES_PER_PROCESS * (k["pool_size"] + k["max_overflow"])
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_ceiling() -> int:
|
||||||
|
"""The notifier's sync pool. One process holds it cluster-wide (leader flock),
|
||||||
|
so it is NOT multiplied by workers or replicas."""
|
||||||
|
k = db._SYNC_POOL_KWARGS
|
||||||
|
return k["pool_size"] + k["max_overflow"]
|
||||||
|
|
||||||
|
|
||||||
def test_defaults_to_sqlite_when_no_database_url():
|
def test_defaults_to_sqlite_when_no_database_url():
|
||||||
assert db.IS_POSTGRES is False
|
assert db.IS_POSTGRES is False
|
||||||
|
|
@ -32,13 +54,18 @@ def test_postgres_pool_sizing_and_sync_timeouts():
|
||||||
"""The Postgres engine-construction kwargs, defined at module scope so they're
|
"""The Postgres engine-construction kwargs, defined at module scope so they're
|
||||||
testable without a real Postgres connection (the engines built from them are
|
testable without a real Postgres connection (the engines built from them are
|
||||||
only actually constructed when IS_POSTGRES, exercised by the docker-compose
|
only actually constructed when IS_POSTGRES, exercised by the docker-compose
|
||||||
stack -- see the module docstring). pool_size=1/max_overflow=1 (the old
|
stack -- see the module docstring). pool_size=1/max_overflow=1 (the original
|
||||||
setting) let a 3rd concurrent request per worker wait out pool_timeout for a
|
setting) let a 3rd concurrent request per worker wait out pool_timeout for a
|
||||||
slot; both async engines need real headroom, and the sync engine (the
|
slot, so the steady-state pool needs real headroom; the sync engine (the
|
||||||
notifier's, driven off-event-loop) must bound connect + statement time so a
|
notifier's, driven off-event-loop) must bound connect + statement time so a
|
||||||
wedged Postgres can't hang it forever under its leader flock."""
|
wedged Postgres can't hang it forever under its leader flock."""
|
||||||
assert db._ASYNC_POOL_KWARGS["pool_size"] >= 5
|
assert db._ASYNC_POOL_KWARGS["pool_size"] >= 5
|
||||||
assert db._ASYNC_POOL_KWARGS["max_overflow"] >= 5
|
|
||||||
|
# The overflow is the part nobody budgets for -- it is multiplied by two async
|
||||||
|
# engines x uvicorn workers x replicas, against a ceiling shared with beta.
|
||||||
|
# Prod saturated max_connections on 2026-07-26 with this at 5. Keep it well
|
||||||
|
# under pool_size; the budget test below is the real constraint.
|
||||||
|
assert db._ASYNC_POOL_KWARGS["max_overflow"] <= 2
|
||||||
|
|
||||||
assert db._SYNC_POOL_KWARGS["pool_size"] < db._ASYNC_POOL_KWARGS["pool_size"]
|
assert db._SYNC_POOL_KWARGS["pool_size"] < db._ASYNC_POOL_KWARGS["pool_size"]
|
||||||
connect_args = db._SYNC_POOL_KWARGS["connect_args"]
|
connect_args = db._SYNC_POOL_KWARGS["connect_args"]
|
||||||
|
|
@ -53,6 +80,27 @@ def test_postgres_pool_sizing_and_sync_timeouts():
|
||||||
engine.dispose()
|
engine.dispose()
|
||||||
|
|
||||||
|
|
||||||
|
def test_pool_budget_fits_under_max_connections():
|
||||||
|
"""At today's replica counts the whole estate's configured worst case must fit
|
||||||
|
inside the server's max_connections, with margin -- prod web + prod worker +
|
||||||
|
beta web + beta worker all draw on ONE Postgres instance. This is the test
|
||||||
|
that would have failed before 2026-07-26, when the configured worst case was
|
||||||
|
170 against a ceiling of 100.
|
||||||
|
|
||||||
|
Note this counts *configured maxima*: every pool at full overflow at the same
|
||||||
|
instant. Prod web autoscaling to its max of 3 replicas is deliberately NOT in
|
||||||
|
this sum -- that peak still does not fit, and closing it needs a connection
|
||||||
|
pooler or a lower autoscale ceiling rather than a bigger number here."""
|
||||||
|
per_process = _async_ceiling_per_process()
|
||||||
|
prod = per_process * PROD_WEB_WORKERS + (per_process + _sync_ceiling())
|
||||||
|
beta = per_process * BETA_WEB_WORKERS + (per_process + _sync_ceiling())
|
||||||
|
|
||||||
|
assert prod + beta <= MAX_CONNECTIONS * 0.8, (
|
||||||
|
f"configured worst case {prod + beta} leaves too little room under "
|
||||||
|
f"max_connections={MAX_CONNECTIONS}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_read_session_yields_a_working_session():
|
def test_read_session_yields_a_working_session():
|
||||||
# On SQLite the read session is fully usable (no read-only enforcement); this
|
# On SQLite the read session is fully usable (no read-only enforcement); this
|
||||||
# just confirms the dependency wiring resolves to a live AsyncSession.
|
# just confirms the dependency wiring resolves to a live AsyncSession.
|
||||||
|
|
|
||||||
351
backend/tests/accounts/test_oauth.py
Normal file
351
backend/tests/accounts/test_oauth.py
Normal file
|
|
@ -0,0 +1,351 @@
|
||||||
|
"""Sign in with an external provider: account resolution, session issuance, and the
|
||||||
|
guards that keep the login flow from becoming an account-takeover path.
|
||||||
|
|
||||||
|
Parametrised over every configured provider, so Discord and Google are held to the
|
||||||
|
same rules rather than one being tested and the other assumed. Provider HTTP is
|
||||||
|
mocked; reuses the throwaway accounts DB from conftest.
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from web import app as appmod
|
||||||
|
from accounts import db, oauth
|
||||||
|
|
||||||
|
V2 = "/thermograph/api/v2"
|
||||||
|
PW = "supersecret123"
|
||||||
|
|
||||||
|
# (provider, callback path, profile builder). Discord's callback is grandfathered
|
||||||
|
# to the URL registered in its developer portal; anything newer uses the canonical
|
||||||
|
# /oauth/{provider}/callback.
|
||||||
|
PROVIDERS = [
|
||||||
|
("discord", f"{V2}/discord/link/callback",
|
||||||
|
lambda sub, email, ver: {"id": sub, "email": email, "verified": ver}),
|
||||||
|
("google", f"{V2}/oauth/google/callback",
|
||||||
|
lambda sub, email, ver: {"sub": sub, "email": email, "email_verified": ver}),
|
||||||
|
]
|
||||||
|
IDS = [p[0] for p in PROVIDERS]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module", autouse=True)
|
||||||
|
def _tables():
|
||||||
|
db.Base.metadata.create_all(db.sync_engine)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _configured(monkeypatch):
|
||||||
|
for name in oauth.PROVIDERS:
|
||||||
|
monkeypatch.setitem(oauth.CREDENTIALS, name, (f"{name}-id", f"{name}-secret"))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(params=PROVIDERS, ids=IDS)
|
||||||
|
def prov(request):
|
||||||
|
"""A provider under test: .name, .callback, .profile(sub, email, verified)."""
|
||||||
|
name, callback, builder = request.param
|
||||||
|
|
||||||
|
class P:
|
||||||
|
pass
|
||||||
|
p = P()
|
||||||
|
p.name, p.callback, p.profile = name, callback, builder
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
class _Resp:
|
||||||
|
def __init__(self, code, data): self.status_code = code; self._data = data
|
||||||
|
def json(self): return self._data
|
||||||
|
|
||||||
|
|
||||||
|
def _mock(monkeypatch, profile):
|
||||||
|
"""Stand in for httpx.AsyncClient: token exchange, then userinfo -> profile."""
|
||||||
|
class _Client:
|
||||||
|
def __init__(self, *a, **k): pass
|
||||||
|
async def __aenter__(self): return self
|
||||||
|
async def __aexit__(self, *a): return False
|
||||||
|
async def post(self, url, **k): return _Resp(200, {"access_token": "tok"})
|
||||||
|
async def get(self, url, **k): return _Resp(200, profile)
|
||||||
|
monkeypatch.setattr(oauth.httpx, "AsyncClient", _Client)
|
||||||
|
|
||||||
|
|
||||||
|
def _register(client, email):
|
||||||
|
assert client.post(f"{V2}/auth/register",
|
||||||
|
json={"email": email, "password": PW}).status_code in (201, 400)
|
||||||
|
|
||||||
|
|
||||||
|
def _login(client, email):
|
||||||
|
_register(client, email)
|
||||||
|
assert client.post(f"{V2}/auth/login",
|
||||||
|
data={"username": email, "password": PW}).status_code == 204
|
||||||
|
|
||||||
|
|
||||||
|
def _cb(client, prov, state):
|
||||||
|
return client.get(f"{prov.callback}?code=abc&state={state}", follow_redirects=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _me(client):
|
||||||
|
return client.get(f"{V2}/users/me").json()
|
||||||
|
|
||||||
|
|
||||||
|
# --- config -------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_config_lists_every_provider_and_its_enabled_state(monkeypatch):
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
monkeypatch.setitem(oauth.CREDENTIALS, "google", ("", ""))
|
||||||
|
by_name = {p["name"]: p for p in c.get(f"{V2}/oauth/config").json()["providers"]}
|
||||||
|
assert by_name["discord"]["enabled"] is True
|
||||||
|
assert by_name["google"]["enabled"] is False # half-configured is inert
|
||||||
|
assert by_name["google"]["label"] == "Google"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_provider_is_404():
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
assert c.get(f"{V2}/oauth/gitlab/login/start",
|
||||||
|
follow_redirects=False).status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
# --- start --------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_login_start_needs_no_session_and_asks_for_an_email_scope(prov):
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
r = c.get(f"{V2}/oauth/{prov.name}/login/start", follow_redirects=False)
|
||||||
|
assert r.status_code == 303
|
||||||
|
loc = r.headers["location"]
|
||||||
|
assert loc.startswith(oauth.PROVIDERS[prov.name].authorize_url)
|
||||||
|
# Whatever the provider calls it, the login scope must be able to yield an
|
||||||
|
# address — without one there is no account to resolve or create.
|
||||||
|
assert "email" in loc and "state=" in loc
|
||||||
|
|
||||||
|
|
||||||
|
def test_login_start_while_signed_in_links_instead_of_switching_account(prov):
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
_login(c, f"already-in-{prov.name}@example.com")
|
||||||
|
r = c.get(f"{V2}/oauth/{prov.name}/login/start", follow_redirects=False)
|
||||||
|
state = r.headers["location"].split("state=")[1].split("&")[0]
|
||||||
|
assert oauth._verify_state(state, "link", prov.name) == _me(c)["id"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_start_when_unconfigured_bounces_back(prov, monkeypatch):
|
||||||
|
monkeypatch.setitem(oauth.CREDENTIALS, prov.name, ("", ""))
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
r = c.get(f"{V2}/oauth/{prov.name}/login/start", follow_redirects=False)
|
||||||
|
assert r.status_code == 303 and "oauth=unavailable" in r.headers["location"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- signed state -------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_state_purposes_and_providers_do_not_cross_over():
|
||||||
|
login = oauth._sign_state(None, "login", "google")
|
||||||
|
link = oauth._sign_state("user-123", "link", "google")
|
||||||
|
# A login state carries no user id but verifies as "" — distinct from reject.
|
||||||
|
assert oauth._verify_state(login, "login", "google") == ""
|
||||||
|
assert oauth._verify_state(login, "link", "google") is None
|
||||||
|
assert oauth._verify_state(link, "link", "google") == "user-123"
|
||||||
|
assert oauth._verify_state(link, "login", "google") is None
|
||||||
|
# And a state minted for one provider is worthless at another's callback.
|
||||||
|
assert oauth._verify_state(link, "link", "discord") is None
|
||||||
|
assert oauth._verify_state(login, "login", "discord") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_state_rejects_tampering_and_expiry(monkeypatch):
|
||||||
|
s = oauth._sign_state("u", "link", "google")
|
||||||
|
payload, mac = s.split(".", 1)
|
||||||
|
assert oauth._verify_state(f"{payload}.{'0' * len(mac)}", "link", "google") is None
|
||||||
|
assert oauth._verify_state(payload, "link", "google") is None
|
||||||
|
assert oauth._verify_state("garbage", "link", "google") is None
|
||||||
|
monkeypatch.setattr(oauth, "_STATE_TTL", -1)
|
||||||
|
assert oauth._verify_state(oauth._sign_state("u", "link", "google"),
|
||||||
|
"link", "google") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_state_is_secret_dependent(monkeypatch):
|
||||||
|
s = oauth._sign_state("u", "link", "google")
|
||||||
|
monkeypatch.setattr(oauth, "SECRET", oauth.SECRET + "-different")
|
||||||
|
assert oauth._verify_state(s, "link", "google") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_login_state_cannot_link_the_signed_in_account(prov, monkeypatch):
|
||||||
|
"""Forced-linking guard. A login state carries no user id, so it is replayable
|
||||||
|
against whoever is signed in; if the callback treated it as a link, an attacker
|
||||||
|
could attach their own identity to a victim's account and then sign in as them.
|
||||||
|
It must complete as a plain login instead."""
|
||||||
|
attacker = f"attacker-{prov.name}@example.com"
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-cross", attacker, True))
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
_login(c, f"crossover-{prov.name}@example.com")
|
||||||
|
r = _cb(c, prov, oauth._sign_state(_me(c)["id"], "login", prov.name))
|
||||||
|
assert "oauth=created" in r.headers["location"]
|
||||||
|
assert _me(c)["email"] == attacker
|
||||||
|
# The account that had been signed in is untouched.
|
||||||
|
victim = TestClient(appmod.app)
|
||||||
|
_login(victim, f"crossover-{prov.name}@example.com")
|
||||||
|
assert _me(victim)["oauth_providers"] == []
|
||||||
|
|
||||||
|
|
||||||
|
# --- login resolves an account ------------------------------------------------
|
||||||
|
|
||||||
|
def test_login_creates_an_account_and_signs_in(prov, monkeypatch):
|
||||||
|
# Mixed case on purpose: the address must be normalised before it is matched or
|
||||||
|
# stored, or the same person gets two accounts depending on how they typed it.
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-new",
|
||||||
|
f"New.User.{prov.name}@Example.com", True))
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
r = _cb(c, prov, oauth._sign_state(None, "login", prov.name))
|
||||||
|
assert r.status_code == 303 and "oauth=created" in r.headers["location"]
|
||||||
|
me = _me(c)
|
||||||
|
assert me["email"] == f"new.user.{prov.name}@example.com" # normalised
|
||||||
|
assert me["is_verified"] is True # the provider vouched
|
||||||
|
assert me["oauth_only"] is True
|
||||||
|
assert me["oauth_providers"] == [prov.name]
|
||||||
|
|
||||||
|
|
||||||
|
def test_login_recognises_a_previously_linked_account(prov, monkeypatch):
|
||||||
|
"""The subject is the durable key: it resolves the account with no email, which
|
||||||
|
is what keeps a login working after someone changes their address upstream."""
|
||||||
|
email = f"link-me-{prov.name}@example.com"
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-ret", email, True))
|
||||||
|
linker = TestClient(appmod.app)
|
||||||
|
_login(linker, email)
|
||||||
|
uid = _me(linker)["id"]
|
||||||
|
assert "oauth=linked" in _cb(
|
||||||
|
linker, prov, oauth._sign_state(uid, "link", prov.name)).headers["location"]
|
||||||
|
# A fresh client, no session, and the provider now returns no email at all.
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-ret", "", False))
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
r = _cb(c, prov, oauth._sign_state(None, "login", prov.name))
|
||||||
|
assert r.status_code == 303 and "oauth=signedin" in r.headers["location"]
|
||||||
|
assert _me(c)["id"] == uid
|
||||||
|
|
||||||
|
|
||||||
|
def test_login_connects_a_verified_email_to_an_existing_account(prov, monkeypatch):
|
||||||
|
"""The 'connect the account' path: an existing password account picks up the
|
||||||
|
identity and is signed in."""
|
||||||
|
email = f"existing-{prov.name}@example.com"
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
_register(c, email)
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-exist", email, True))
|
||||||
|
fresh = TestClient(appmod.app)
|
||||||
|
r = _cb(fresh, prov, oauth._sign_state(None, "login", prov.name))
|
||||||
|
assert r.status_code == 303 and "oauth=signedin" in r.headers["location"]
|
||||||
|
me = _me(fresh)
|
||||||
|
assert me["email"] == email and me["oauth_providers"] == [prov.name]
|
||||||
|
# It kept its password, so it is not OAuth-only and may unlink.
|
||||||
|
assert me["oauth_only"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_login_refuses_an_unverified_email(prov, monkeypatch):
|
||||||
|
"""The provider's verified flag is the only evidence the person owns the
|
||||||
|
address. Without it, this would hand over any account whose email was typed in."""
|
||||||
|
email = f"victim-{prov.name}@example.com"
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
_register(c, email)
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-attacker", email, False))
|
||||||
|
fresh = TestClient(appmod.app)
|
||||||
|
r = _cb(fresh, prov, oauth._sign_state(None, "login", prov.name))
|
||||||
|
assert r.status_code == 303 and "oauth=unverified" in r.headers["location"]
|
||||||
|
assert fresh.get(f"{V2}/users/me").status_code == 401 # no session issued
|
||||||
|
_login(c, email)
|
||||||
|
assert _me(c)["oauth_providers"] == [] # target untouched
|
||||||
|
|
||||||
|
|
||||||
|
def test_login_without_an_email_cannot_create_an_account(prov, monkeypatch):
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-noemail", "", True))
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
r = _cb(c, prov, oauth._sign_state(None, "login", prov.name))
|
||||||
|
assert r.status_code == 303 and "oauth=noemail" in r.headers["location"]
|
||||||
|
assert c.get(f"{V2}/users/me").status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_login_will_not_move_an_account_between_identities(prov, monkeypatch):
|
||||||
|
email = f"owned-{prov.name}@example.com"
|
||||||
|
owner = TestClient(appmod.app)
|
||||||
|
_login(owner, email)
|
||||||
|
uid = _me(owner)["id"]
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-owner", email, True))
|
||||||
|
assert "oauth=linked" in _cb(
|
||||||
|
owner, prov, oauth._sign_state(uid, "link", prov.name)).headers["location"]
|
||||||
|
# A second provider account claiming the same address.
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-interloper", email, True))
|
||||||
|
fresh = TestClient(appmod.app)
|
||||||
|
r = _cb(fresh, prov, oauth._sign_state(None, "login", prov.name))
|
||||||
|
assert r.status_code == 303 and "oauth=mismatch" in r.headers["location"]
|
||||||
|
assert fresh.get(f"{V2}/users/me").status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_linking_an_identity_someone_else_owns_is_refused(prov, monkeypatch):
|
||||||
|
first = TestClient(appmod.app)
|
||||||
|
_login(first, f"first-owner-{prov.name}@example.com")
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-contested",
|
||||||
|
f"first-owner-{prov.name}@example.com", True))
|
||||||
|
assert "oauth=linked" in _cb(
|
||||||
|
first, prov,
|
||||||
|
oauth._sign_state(_me(first)["id"], "link", prov.name)).headers["location"]
|
||||||
|
second = TestClient(appmod.app)
|
||||||
|
_login(second, f"second-owner-{prov.name}@example.com")
|
||||||
|
r = _cb(second, prov, oauth._sign_state(_me(second)["id"], "link", prov.name))
|
||||||
|
assert r.status_code == 303 and "oauth=taken" in r.headers["location"]
|
||||||
|
assert _me(second)["oauth_providers"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_relinking_the_same_identity_is_idempotent(prov, monkeypatch):
|
||||||
|
email = f"relink-{prov.name}@example.com"
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
_login(c, email)
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-relink", email, True))
|
||||||
|
uid = _me(c)["id"]
|
||||||
|
for _ in range(2):
|
||||||
|
r = _cb(c, prov, oauth._sign_state(uid, "link", prov.name))
|
||||||
|
assert "oauth=linked" in r.headers["location"]
|
||||||
|
assert _me(c)["oauth_providers"] == [prov.name] # not duplicated
|
||||||
|
|
||||||
|
|
||||||
|
# --- unlink guards ------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_last_provider_cannot_be_unlinked_into_a_lockout(prov, monkeypatch):
|
||||||
|
_mock(monkeypatch, prov.profile(f"{prov.name}-onlyway",
|
||||||
|
f"onlyway-{prov.name}@example.com", True))
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
assert "oauth=created" in _cb(
|
||||||
|
c, prov, oauth._sign_state(None, "login", prov.name)).headers["location"]
|
||||||
|
r = c.post(f"{V2}/oauth/{prov.name}/unlink")
|
||||||
|
assert r.status_code == 409 and "no password" in r.json()["detail"]
|
||||||
|
assert _me(c)["oauth_providers"] == [prov.name]
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_oauth_only_account_may_unlink_once_a_second_provider_is_linked(monkeypatch):
|
||||||
|
"""The lockout rule is about the *last* credential, not about Discord. With two
|
||||||
|
providers linked, dropping one still leaves a way in."""
|
||||||
|
_mock(monkeypatch, {"sub": "g-two", "email": "two@example.com",
|
||||||
|
"email_verified": True})
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
assert "oauth=created" in _cb(
|
||||||
|
c, PROV_GOOGLE, oauth._sign_state(None, "login", "google")).headers["location"]
|
||||||
|
uid = _me(c)["id"]
|
||||||
|
# Now connect Discord to the same, still password-less, account.
|
||||||
|
_mock(monkeypatch, {"id": "d-two", "email": "two@example.com", "verified": True})
|
||||||
|
assert "oauth=linked" in _cb(
|
||||||
|
c, PROV_DISCORD, oauth._sign_state(uid, "link", "discord")).headers["location"]
|
||||||
|
assert _me(c)["oauth_providers"] == ["discord", "google"]
|
||||||
|
# Either one may now go.
|
||||||
|
assert c.post(f"{V2}/oauth/google/unlink").status_code == 204
|
||||||
|
assert _me(c)["oauth_providers"] == ["discord"]
|
||||||
|
# But the survivor is once again the only way in.
|
||||||
|
assert c.post(f"{V2}/oauth/discord/unlink").status_code == 409
|
||||||
|
|
||||||
|
|
||||||
|
def test_unlink_requires_auth(prov):
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
assert c.post(f"{V2}/oauth/{prov.name}/unlink").status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_link_start_requires_auth(prov):
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
assert c.get(f"{V2}/oauth/{prov.name}/link/start",
|
||||||
|
follow_redirects=False).status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
# Bare provider handles for the cross-provider test above, which needs both at once.
|
||||||
|
class _P:
|
||||||
|
def __init__(self, name, callback):
|
||||||
|
self.name, self.callback = name, callback
|
||||||
|
|
||||||
|
|
||||||
|
PROV_GOOGLE = _P("google", f"{V2}/oauth/google/callback")
|
||||||
|
PROV_DISCORD = _P("discord", f"{V2}/discord/link/callback")
|
||||||
|
|
@ -7,8 +7,8 @@ from fastapi.testclient import TestClient
|
||||||
|
|
||||||
from web import app as appmod
|
from web import app as appmod
|
||||||
from accounts import db
|
from accounts import db
|
||||||
|
from accounts import oauth
|
||||||
from notifications import discord
|
from notifications import discord
|
||||||
from notifications import discord_link as dl
|
|
||||||
from notifications import notify
|
from notifications import notify
|
||||||
|
|
||||||
V2 = "/thermograph/api/v2"
|
V2 = "/thermograph/api/v2"
|
||||||
|
|
@ -161,14 +161,15 @@ class _MockClient:
|
||||||
|
|
||||||
|
|
||||||
def test_linking_opts_in_and_toggle_flips(monkeypatch):
|
def test_linking_opts_in_and_toggle_flips(monkeypatch):
|
||||||
monkeypatch.setattr(dl, "CLIENT_ID", "app")
|
monkeypatch.setitem(oauth.CREDENTIALS, "discord", ("app", "sec"))
|
||||||
monkeypatch.setattr(dl, "CLIENT_SECRET", "sec")
|
monkeypatch.setattr(oauth.httpx, "AsyncClient", _MockClient)
|
||||||
monkeypatch.setattr(dl.httpx, "AsyncClient", _MockClient)
|
|
||||||
c = TestClient(appmod.app)
|
c = TestClient(appmod.app)
|
||||||
_login(c, "dm-toggle@example.com")
|
_login(c, "dm-toggle@example.com")
|
||||||
uid = c.get(f"{V2}/users/me").json()["id"]
|
uid = c.get(f"{V2}/users/me").json()["id"]
|
||||||
# Link -> opted in by default.
|
# Link -> opted in by default. discord_id is a delivery address, so the link
|
||||||
r = c.get(f"{V2}/discord/link/callback?code=x&state={dl._sign_state(uid)}", follow_redirects=False)
|
# has to write it as well as the oauth_account row.
|
||||||
|
state = oauth._sign_state(uid, "link", "discord")
|
||||||
|
r = c.get(f"{V2}/discord/link/callback?code=x&state={state}", follow_redirects=False)
|
||||||
assert r.status_code == 303
|
assert r.status_code == 303
|
||||||
me = c.get(f"{V2}/users/me").json()
|
me = c.get(f"{V2}/users/me").json()
|
||||||
assert me["discord_id"] == "discord-777" and me["discord_dm"] is True
|
assert me["discord_id"] == "discord-777" and me["discord_dm"] is True
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,20 @@
|
||||||
"""Discord account linking: signed-state integrity, OAuth callback (Discord HTTP
|
"""The legacy /discord/* routes.
|
||||||
mocked), unlink, and auth gating. Reuses the throwaway accounts DB from conftest."""
|
|
||||||
|
These are not duplicate coverage of tests/accounts/test_oauth.py — they pin the two
|
||||||
|
compatibility promises that outlive the refactor:
|
||||||
|
|
||||||
|
* **/discord/link/callback is registered in Discord's developer portal.** If it
|
||||||
|
stops answering, every Discord login breaks until an operator edits the portal.
|
||||||
|
* The other /discord/* paths are what the deployed frontend calls, and frontend and
|
||||||
|
backend deploy independently, so a newer backend must keep serving an older one.
|
||||||
|
|
||||||
|
The flow itself lives in accounts/oauth.py and is tested there.
|
||||||
|
"""
|
||||||
import pytest
|
import pytest
|
||||||
from fastapi.testclient import TestClient
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
from web import app as appmod
|
from web import app as appmod
|
||||||
from accounts import db
|
from accounts import db, oauth
|
||||||
from notifications import discord_link as dl
|
|
||||||
|
|
||||||
V2 = "/thermograph/api/v2"
|
V2 = "/thermograph/api/v2"
|
||||||
PW = "supersecret123"
|
PW = "supersecret123"
|
||||||
|
|
@ -16,72 +25,10 @@ def _tables():
|
||||||
db.Base.metadata.create_all(db.sync_engine)
|
db.Base.metadata.create_all(db.sync_engine)
|
||||||
|
|
||||||
|
|
||||||
def _login(client, email):
|
@pytest.fixture(autouse=True)
|
||||||
r = client.post(f"{V2}/auth/register", json={"email": email, "password": PW})
|
def _configured(monkeypatch):
|
||||||
assert r.status_code in (201, 400)
|
monkeypatch.setitem(oauth.CREDENTIALS, "discord", ("app123", "secret456"))
|
||||||
assert client.post(f"{V2}/auth/login", data={"username": email, "password": PW}).status_code == 204
|
monkeypatch.setattr(oauth.httpx, "AsyncClient", _MockClient)
|
||||||
|
|
||||||
|
|
||||||
# --- signed state ------------------------------------------------------------
|
|
||||||
|
|
||||||
def test_state_round_trips_and_rejects_tampering():
|
|
||||||
s = dl._sign_state("user-123")
|
|
||||||
assert dl._verify_state(s) == "user-123"
|
|
||||||
payload, mac = s.split(".", 1)
|
|
||||||
assert dl._verify_state(f"{payload}.{'0' * len(mac)}") is None # bad mac
|
|
||||||
assert dl._verify_state(payload) is None # no mac
|
|
||||||
assert dl._verify_state("garbage") is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_state_expires(monkeypatch):
|
|
||||||
monkeypatch.setattr(dl, "_STATE_TTL", -1) # already expired
|
|
||||||
assert dl._verify_state(dl._sign_state("u")) is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_state_is_secret_dependent(monkeypatch):
|
|
||||||
s = dl._sign_state("u")
|
|
||||||
monkeypatch.setattr(dl, "SECRET", dl.SECRET + "-different")
|
|
||||||
assert dl._verify_state(s) is None # signed under the old secret
|
|
||||||
|
|
||||||
|
|
||||||
# --- routes ------------------------------------------------------------------
|
|
||||||
|
|
||||||
def test_config_reports_enabled_state(monkeypatch):
|
|
||||||
c = TestClient(appmod.app)
|
|
||||||
monkeypatch.setattr(dl, "CLIENT_ID", "")
|
|
||||||
monkeypatch.setattr(dl, "CLIENT_SECRET", "")
|
|
||||||
r = c.get(f"{V2}/discord/config") # no auth required
|
|
||||||
assert r.status_code == 200 and r.json() == {"enabled": False}
|
|
||||||
monkeypatch.setattr(dl, "CLIENT_ID", "app123")
|
|
||||||
monkeypatch.setattr(dl, "CLIENT_SECRET", "secret456")
|
|
||||||
assert c.get(f"{V2}/discord/config").json() == {"enabled": True}
|
|
||||||
|
|
||||||
|
|
||||||
def test_link_requires_auth():
|
|
||||||
c = TestClient(appmod.app)
|
|
||||||
assert c.get(f"{V2}/discord/link/start", follow_redirects=False).status_code == 401
|
|
||||||
assert c.post(f"{V2}/discord/unlink").status_code == 401
|
|
||||||
|
|
||||||
|
|
||||||
def test_start_redirects_to_discord(monkeypatch):
|
|
||||||
monkeypatch.setattr(dl, "CLIENT_ID", "app123")
|
|
||||||
monkeypatch.setattr(dl, "CLIENT_SECRET", "secret456")
|
|
||||||
c = TestClient(appmod.app)
|
|
||||||
_login(c, "start@example.com")
|
|
||||||
r = c.get(f"{V2}/discord/link/start", follow_redirects=False)
|
|
||||||
assert r.status_code == 303
|
|
||||||
loc = r.headers["location"]
|
|
||||||
assert loc.startswith("https://discord.com/api/oauth2/authorize")
|
|
||||||
assert "client_id=app123" in loc and "scope=identify" in loc and "state=" in loc
|
|
||||||
|
|
||||||
|
|
||||||
def test_start_when_unconfigured_bounces_back(monkeypatch):
|
|
||||||
monkeypatch.setattr(dl, "CLIENT_ID", "")
|
|
||||||
monkeypatch.setattr(dl, "CLIENT_SECRET", "")
|
|
||||||
c = TestClient(appmod.app)
|
|
||||||
_login(c, "noconf@example.com")
|
|
||||||
r = c.get(f"{V2}/discord/link/start", follow_redirects=False)
|
|
||||||
assert r.status_code == 303 and "discord=unavailable" in r.headers["location"]
|
|
||||||
|
|
||||||
|
|
||||||
class _Resp:
|
class _Resp:
|
||||||
|
|
@ -90,47 +37,88 @@ class _Resp:
|
||||||
|
|
||||||
|
|
||||||
class _MockClient:
|
class _MockClient:
|
||||||
"""Stands in for httpx.AsyncClient: token exchange then /users/@me."""
|
|
||||||
def __init__(self, *a, **k): pass
|
def __init__(self, *a, **k): pass
|
||||||
async def __aenter__(self): return self
|
async def __aenter__(self): return self
|
||||||
async def __aexit__(self, *a): return False
|
async def __aexit__(self, *a): return False
|
||||||
async def post(self, url, **k): return _Resp(200, {"access_token": "tok"})
|
async def post(self, url, **k): return _Resp(200, {"access_token": "tok"})
|
||||||
async def get(self, url, **k): return _Resp(200, {"id": "discord-999", "username": "someone"})
|
async def get(self, url, **k):
|
||||||
|
return _Resp(200, {"id": "discord-999", "email": "legacy@example.com",
|
||||||
|
"verified": True})
|
||||||
|
|
||||||
|
|
||||||
def test_callback_links_the_account(monkeypatch):
|
def _login(client, email):
|
||||||
monkeypatch.setattr(dl, "CLIENT_ID", "app123")
|
assert client.post(f"{V2}/auth/register",
|
||||||
monkeypatch.setattr(dl, "CLIENT_SECRET", "secret456")
|
json={"email": email, "password": PW}).status_code in (201, 400)
|
||||||
monkeypatch.setattr(dl.httpx, "AsyncClient", _MockClient)
|
assert client.post(f"{V2}/auth/login",
|
||||||
|
data={"username": email, "password": PW}).status_code == 204
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_config_still_reports_enabled_state(monkeypatch):
|
||||||
c = TestClient(appmod.app)
|
c = TestClient(appmod.app)
|
||||||
_login(c, "callback@example.com")
|
monkeypatch.setitem(oauth.CREDENTIALS, "discord", ("", ""))
|
||||||
|
assert c.get(f"{V2}/discord/config").json() == {"enabled": False}
|
||||||
|
monkeypatch.setitem(oauth.CREDENTIALS, "discord", ("app123", "secret456"))
|
||||||
|
assert c.get(f"{V2}/discord/config").json() == {"enabled": True}
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_link_start_still_requires_auth():
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
assert c.get(f"{V2}/discord/link/start", follow_redirects=False).status_code == 401
|
||||||
|
assert c.post(f"{V2}/discord/unlink").status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_start_redirects_to_discord():
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
_login(c, "legacy-start@example.com")
|
||||||
|
r = c.get(f"{V2}/discord/link/start", follow_redirects=False)
|
||||||
|
assert r.status_code == 303
|
||||||
|
loc = r.headers["location"]
|
||||||
|
assert loc.startswith("https://discord.com/api/oauth2/authorize")
|
||||||
|
assert "client_id=app123" in loc and "scope=identify" in loc and "state=" in loc
|
||||||
|
# The redirect_uri it asks Discord to call back on must be the registered one.
|
||||||
|
assert "discord%2Flink%2Fcallback" in loc
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_registered_callback_url_still_completes_a_link():
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
_login(c, "legacy-cb@example.com")
|
||||||
uid = c.get(f"{V2}/users/me").json()["id"]
|
uid = c.get(f"{V2}/users/me").json()["id"]
|
||||||
state = dl._sign_state(uid)
|
state = oauth._sign_state(uid, "link", "discord")
|
||||||
r = c.get(f"{V2}/discord/link/callback?code=abc&state={state}", follow_redirects=False)
|
r = c.get(f"{V2}/discord/link/callback?code=abc&state={state}",
|
||||||
assert r.status_code == 303 and "discord=linked" in r.headers["location"]
|
|
||||||
# /users/me now reports the linked id, and unlink clears it.
|
|
||||||
assert c.get(f"{V2}/users/me").json()["discord_id"] == "discord-999"
|
|
||||||
assert c.post(f"{V2}/discord/unlink").status_code == 204
|
|
||||||
assert c.get(f"{V2}/users/me").json()["discord_id"] is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_callback_rejects_a_forged_state(monkeypatch):
|
|
||||||
monkeypatch.setattr(dl, "CLIENT_ID", "app123")
|
|
||||||
monkeypatch.setattr(dl, "CLIENT_SECRET", "secret456")
|
|
||||||
monkeypatch.setattr(dl.httpx, "AsyncClient", _MockClient)
|
|
||||||
c = TestClient(appmod.app)
|
|
||||||
_login(c, "forged@example.com")
|
|
||||||
# State signed for a different user id must not link this session.
|
|
||||||
r = c.get(f"{V2}/discord/link/callback?code=abc&state={dl._sign_state('someone-else')}",
|
|
||||||
follow_redirects=False)
|
follow_redirects=False)
|
||||||
assert r.status_code == 303 and "discord=error" in r.headers["location"]
|
assert r.status_code == 303 and "discord=linked" in r.headers["location"]
|
||||||
assert c.get(f"{V2}/users/me").json()["discord_id"] is None
|
me = c.get(f"{V2}/users/me").json()
|
||||||
|
assert me["discord_id"] == "discord-999" # the DM delivery address
|
||||||
|
assert me["oauth_providers"] == ["discord"] # and the identity row
|
||||||
|
# Legacy unlink clears both.
|
||||||
|
assert c.post(f"{V2}/discord/unlink").status_code == 204
|
||||||
|
me = c.get(f"{V2}/users/me").json()
|
||||||
|
assert me["discord_id"] is None and me["oauth_providers"] == []
|
||||||
|
|
||||||
|
|
||||||
def test_callback_without_code_is_cancelled(monkeypatch):
|
def test_the_callback_still_emits_the_discord_query_param():
|
||||||
monkeypatch.setattr(dl, "CLIENT_ID", "app123")
|
"""The deployed frontend reads ?discord=<status>, not ?oauth=. Dropping it would
|
||||||
monkeypatch.setattr(dl, "CLIENT_SECRET", "secret456")
|
silently stop it showing any outcome at all."""
|
||||||
c = TestClient(appmod.app)
|
c = TestClient(appmod.app)
|
||||||
_login(c, "cancel@example.com")
|
|
||||||
r = c.get(f"{V2}/discord/link/callback?error=access_denied", follow_redirects=False)
|
r = c.get(f"{V2}/discord/link/callback?error=access_denied", follow_redirects=False)
|
||||||
assert r.status_code == 303 and "discord=cancelled" in r.headers["location"]
|
loc = r.headers["location"]
|
||||||
|
assert "discord=cancelled" in loc and "oauth=cancelled" in loc
|
||||||
|
|
||||||
|
|
||||||
|
def test_users_me_still_carries_the_deprecated_discord_only_field(monkeypatch):
|
||||||
|
"""Renamed to oauth_only, but the deployed frontend still reads the old name."""
|
||||||
|
monkeypatch.setattr(oauth.httpx, "AsyncClient", _MockClient)
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
r = c.get(f"{V2}/discord/link/callback?code=abc"
|
||||||
|
f"&state={oauth._sign_state(None, 'login', 'discord')}",
|
||||||
|
follow_redirects=False)
|
||||||
|
assert "discord=created" in r.headers["location"]
|
||||||
|
me = c.get(f"{V2}/users/me").json()
|
||||||
|
assert me["oauth_only"] is True and me["discord_only"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_login_start_is_still_mounted():
|
||||||
|
c = TestClient(appmod.app)
|
||||||
|
r = c.get(f"{V2}/discord/login/start", follow_redirects=False)
|
||||||
|
assert r.status_code == 303
|
||||||
|
assert r.headers["location"].startswith("https://discord.com/api/oauth2/authorize")
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,7 @@ from data import climate
|
||||||
from notifications import digest
|
from notifications import digest
|
||||||
from notifications import discord_interactions as discord_interactions_mod
|
from notifications import discord_interactions as discord_interactions_mod
|
||||||
from notifications import discord_link
|
from notifications import discord_link
|
||||||
from accounts import db
|
from accounts import db, oauth
|
||||||
from data import grid
|
from data import grid
|
||||||
from core import metrics
|
from core import metrics
|
||||||
from notifications import notify
|
from notifications import notify
|
||||||
|
|
@ -988,7 +988,12 @@ app.include_router(
|
||||||
)
|
)
|
||||||
# Subscriptions + notifications (authed, user-scoped).
|
# Subscriptions + notifications (authed, user-scoped).
|
||||||
app.include_router(api_accounts.router, prefix=f"{BASE}/api/v2")
|
app.include_router(api_accounts.router, prefix=f"{BASE}/api/v2")
|
||||||
# Discord account linking (OAuth2, authed).
|
# Sign in with / connect an external provider (Discord, Google).
|
||||||
|
app.include_router(oauth.router, prefix=f"{BASE}/api/v2")
|
||||||
|
# Discord's legacy /discord/* paths: the callback URL registered in Discord's
|
||||||
|
# portal, plus what the currently-deployed frontend calls. Mounted AFTER the
|
||||||
|
# generic router so neither shadows the other — the paths are disjoint, and
|
||||||
|
# /oauth/{provider}/... would otherwise be a candidate match for /discord/....
|
||||||
app.include_router(discord_link.router, prefix=f"{BASE}/api/v2")
|
app.include_router(discord_link.router, prefix=f"{BASE}/api/v2")
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -55,40 +55,59 @@ and `PAYLOAD_VER` is the lever that moves it.** See
|
||||||
|
|
||||||
## The estate
|
## The estate
|
||||||
|
|
||||||
Four machines on a WireGuard mesh (`10.10.0.0/24`):
|
Hosts are named by **role**, not by environment, plus the operator's desktop —
|
||||||
|
all on a WireGuard mesh (`10.10.0.0/24`):
|
||||||
|
|
||||||
| Host | Mesh IP | Public | Runs |
|
| Host | Mesh IP | Public | Runs |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| **prod** | `10.10.0.1` | `169.58.46.181`, thermograph.org | Docker **Swarm** app stack, TimescaleDB, Postfix, backups, Centralis |
|
| **vps1** | `10.10.0.2` | `75.119.132.91`, git.thermograph.org, dashboard.thermograph.org | Forgejo (git + CI + registry), Grafana + Loki + Alloy, the `emigriffith.dev` portfolio, and **dev** — its own Postgres, plain compose, **mesh-only** (no public DNS, no Caddy site, no TLS) |
|
||||||
| **beta** | `10.10.0.2` | `75.119.132.91`, beta.thermograph.org | **compose** app stack, Forgejo (git + CI + registry), Grafana + Loki |
|
| **vps2** | `10.10.0.1` | `169.58.46.181`, thermograph.org, beta.thermograph.org | **prod** and **beta** as two separate Docker **Swarm** stacks, Centralis, Postfix, backups |
|
||||||
| **desktop** | `10.10.0.3` | — | LAN dev stack, the Forgejo Actions runner, your editor |
|
| **desktop** | `10.10.0.3` | — | AI-model hosting (voice-to-text, an upcoming-feature LLM) + flex Swarm capacity. Hosts **no** Thermograph environment — `make dev-up` there is a laptop convenience only |
|
||||||
| phone | — | — | alerts |
|
| phone | — | — | alerts |
|
||||||
|
|
||||||
|
vps2 runs prod and beta **co-resident on one box**: one TimescaleDB instance
|
||||||
|
serves both, on separate databases and separate roles (`thermograph` /
|
||||||
|
`thermograph_beta`, the latter `NOSUPERUSER`/`NOCREATEDB`, `CONNECT` revoked
|
||||||
|
from `PUBLIC`); separate checkouts, env files and loopback LB ports; beta's
|
||||||
|
Swarm service names are prefixed (`beta-web`, not `web`) because Swarm
|
||||||
|
registers a service's short name as a DNS alias on every network it joins, and
|
||||||
|
beta shares prod's network to reach the database. Beta rehearses prod's actual
|
||||||
|
orchestrator this way — that's the point of putting it next to prod rather
|
||||||
|
than next to dev.
|
||||||
|
|
||||||
|
The mesh IPs did not move when this topology landed — vps1 is the box that used
|
||||||
|
to be called "beta", vps2 the one that used to be called "prod". What moved is
|
||||||
|
which environment lives where. `infra/deploy/env-topology.sh` is the single
|
||||||
|
source of truth for every per-environment path, port and role; `THERMOGRAPH_ENV`
|
||||||
|
(`dev`/`beta`/`prod`) is the input a deploy leg passes, since vps2 can no longer
|
||||||
|
tell beta and prod apart by which host it's running on.
|
||||||
|
|
||||||
Two consequences you will hit within the first week:
|
Two consequences you will hit within the first week:
|
||||||
|
|
||||||
- **Forgejo is mesh-only.** `git.thermograph.org` resolves publicly to beta's
|
- **Forgejo is mesh-only.** `git.thermograph.org` resolves publicly to vps1's
|
||||||
IP, but beta's Caddy rejects `/v2/*` (the registry API) from outside the mesh.
|
IP, but vps1's Caddy rejects `/v2/*` (the registry API) from outside the
|
||||||
Any host that pulls images needs `10.10.0.2 git.thermograph.org` in
|
mesh. Any host that pulls images needs `10.10.0.2 git.thermograph.org` in
|
||||||
`/etc/hosts`.
|
`/etc/hosts`.
|
||||||
- **Prod and beta run different orchestrators.** Prod is Swarm
|
- **Beta and prod run the same orchestrator now — Swarm, as two stacks on one
|
||||||
(`infra/deploy/stack/`), beta and LAN dev are compose
|
box.** Dev is the only environment on compose (`infra/docker-compose.yml`),
|
||||||
(`infra/docker-compose.yml`). `deploy.sh` routes between them by reading
|
and it lives alone on vps1. Most of the time you don't care which stack
|
||||||
`/etc/thermograph/deploy-mode` on the box. Most of the time you don't care —
|
you're reading — but when you're reading logs or naming containers, you do
|
||||||
but when you're reading logs or naming containers, you do
|
(see [observability](09-observability.md)), and on vps2 you additionally have
|
||||||
(see [observability](09-observability.md)).
|
to say *which* environment: `deploy.sh` there routes on the explicit
|
||||||
|
`THERMOGRAPH_ENV` a caller passes, not on which host it's running on.
|
||||||
|
|
||||||
## How a change travels
|
## How a change travels
|
||||||
|
|
||||||
```
|
```
|
||||||
PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box
|
PR ──(required check: `gate`)──▶ dev ──▶ dev (vps1, mesh-only)
|
||||||
│
|
│
|
||||||
promotion PR
|
promotion PR
|
||||||
▼
|
▼
|
||||||
main ──▶ beta.thermograph.org
|
main ──▶ beta.thermograph.org (vps2)
|
||||||
│
|
│
|
||||||
promotion PR ← the owner's call
|
promotion PR ← the owner's call
|
||||||
▼
|
▼
|
||||||
release ──▶ thermograph.org
|
release ──▶ thermograph.org (vps2)
|
||||||
```
|
```
|
||||||
|
|
||||||
`dev`, `main` and `release` are **protected** — no direct pushes, for humans or
|
`dev`, `main` and `release` are **protected** — no direct pushes, for humans or
|
||||||
|
|
|
||||||
|
|
@ -172,11 +172,18 @@ ends fail closed. With Discord unconfigured it logs once and runs cron-only.
|
||||||
```bash
|
```bash
|
||||||
cd infra
|
cd infra
|
||||||
make dev-up # docker-compose.yml + docker-compose.dev.yml overlay:
|
make dev-up # docker-compose.yml + docker-compose.dev.yml overlay:
|
||||||
# uncapped CPU, backend published on 0.0.0.0:8137 for the LAN
|
# uncapped CPU, backend published on 0.0.0.0:8137 for your LAN
|
||||||
make dev-down
|
make dev-down
|
||||||
```
|
```
|
||||||
|
|
||||||
The dev overlay exports `COMPOSE_PROJECT_NAME=thermograph-dev` so the LAN stack
|
This is a **laptop convenience**, not the hosted dev environment: the actual
|
||||||
|
`dev` environment now runs on vps1 with its own Postgres, deployed by CI, and
|
||||||
|
bound only to the WireGuard mesh (`10.10.0.2:8137`) — never `0.0.0.0`, since
|
||||||
|
vps1 is a public box. See [Infra and secrets](08-infra-secrets.md). A local
|
||||||
|
`make dev-up` is fine on `0.0.0.0` because it's your own machine's LAN, not the
|
||||||
|
public internet.
|
||||||
|
|
||||||
|
The dev overlay exports `COMPOSE_PROJECT_NAME=thermograph-dev` so this stack
|
||||||
keeps volumes separate from anything else. **Do not remove either half of the
|
keeps volumes separate from anything else. **Do not remove either half of the
|
||||||
project-name pinning** — `infra/docker-compose.yml` pins `name: thermograph`,
|
project-name pinning** — `infra/docker-compose.yml` pins `name: thermograph`,
|
||||||
and without it running compose from `infra/` derives the project name `infra`,
|
and without it running compose from `infra/` derives the project name `infra`,
|
||||||
|
|
@ -216,7 +223,7 @@ it will be fresher than this page.
|
||||||
|
|
||||||
| Hook | When | What |
|
| Hook | When | What |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `prod-guard.sh` | before Bash / live-host MCP calls | Classifies by **allowlist**: only positively-recognised read-only commands pass; everything else asks. Beta is guarded as strictly as prod — it hosts Forgejo, Grafana *and* beta.thermograph.org. |
|
| `prod-guard.sh` | before Bash / live-host MCP calls | Classifies by **allowlist**: only positively-recognised read-only commands pass; everything else asks. vps1 is guarded as strictly as vps2 — it hosts Forgejo, Grafana *and* the mesh-only `dev` environment, so a destructive command there takes out git, CI and the registry at once, not just a dev sandbox. |
|
||||||
| `secrets-guard.sh` | before Write/Edit | Denies any direct write to `infra/deploy/secrets/*.yaml`. Use `sops edit`. |
|
| `secrets-guard.sh` | before Write/Edit | Denies any direct write to `infra/deploy/secrets/*.yaml`. Use `sops edit`. |
|
||||||
| `lint-after-edit.sh` | after Write/Edit | shellchecks an edited `*.sh` and feeds findings straight back. Exits quietly if shellcheck is missing. |
|
| `lint-after-edit.sh` | after Write/Edit | shellchecks an edited `*.sh` and feeds findings straight back. Exits quietly if shellcheck is missing. |
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -114,24 +114,36 @@ Deep dive: [05-frontend.md](05-frontend.md).
|
||||||
## `infra/` — how and where images run
|
## `infra/` — how and where images run
|
||||||
|
|
||||||
```
|
```
|
||||||
docker-compose.yml the compose stack (beta + LAN dev):
|
docker-compose.yml the compose stack — dev's, on vps1 (and a local
|
||||||
db, backend, lake, daemon, frontend.
|
`make dev-up`): db, backend, lake, daemon, frontend.
|
||||||
`name: thermograph` is PINNED — see below.
|
`name: thermograph` is PINNED — see below.
|
||||||
docker-compose.dev.yml LAN overlay: uncapped, backend on 0.0.0.0:8137
|
docker-compose.dev.yml dev overlay: uncapped, backend bound to
|
||||||
|
`${DEV_BIND_ADDR:-127.0.0.1}` — vps1's mesh IP
|
||||||
|
(10.10.0.2) at deploy time, never 0.0.0.0
|
||||||
docker-compose.openmeteo.yml self-hosted Open-Meteo overlay (prod-only)
|
docker-compose.openmeteo.yml self-hosted Open-Meteo overlay (prod-only)
|
||||||
Makefile compose orchestration only
|
Makefile compose orchestration only
|
||||||
deploy/
|
deploy/
|
||||||
deploy.sh ★ the single entry point for beta and prod
|
env-topology.sh ★ single source of truth: env → host, checkout,
|
||||||
render-secrets.sh renders /etc/thermograph.env from the SOPS vault
|
branch, deploy mode, stack name, env file, LB
|
||||||
|
ports, DB role/name, service prefix
|
||||||
|
deploy.sh ★ the single entry point for dev, beta and prod
|
||||||
|
deploy-dev.sh thin wrapper: routes deploy.sh at dev's compose
|
||||||
|
overlay and secrets policy
|
||||||
|
render-secrets.sh renders the env file for a given environment from
|
||||||
|
the SOPS vault
|
||||||
secrets/ the vault: common.yaml, prod/beta/dev.yaml,
|
secrets/ the vault: common.yaml, prod/beta/dev.yaml,
|
||||||
centralis.prod.yaml, example.yaml
|
centralis.prod.yaml, example.yaml
|
||||||
stack/ the SWARM path (live on prod):
|
stack/ the SWARM path — both live on vps2:
|
||||||
thermograph-stack.yml, deploy-stack.sh,
|
thermograph-stack.yml (prod),
|
||||||
autoscale.sh, the LB
|
thermograph-beta-stack.yml (beta, prefixed
|
||||||
|
services, no `db` of its own), deploy-stack.sh,
|
||||||
|
autoscale.sh, lb/ (per-environment Caddyfiles)
|
||||||
|
db/provision-env-db.sh creates each environment's database + role on the
|
||||||
|
shared TimescaleDB instance
|
||||||
|
provision-dev.sh one-time bootstrap of the dev environment on vps1
|
||||||
swarm/, forgejo/ mesh + Forgejo/CI-runner provisioning
|
swarm/, forgejo/ mesh + Forgejo/CI-runner provisioning
|
||||||
Caddyfile the reverse proxy (path-splits FE vs BE)
|
Caddyfile the reverse proxy (path-splits FE vs BE)
|
||||||
provision-*.sh host bootstrap: agent access, dev LAN, mail, secrets
|
provision-*.sh host bootstrap: agent access, mail, secrets
|
||||||
db/init/ TimescaleDB init + tuning
|
|
||||||
migrations/ hand-run SQL migrations
|
migrations/ hand-run SQL migrations
|
||||||
ops/
|
ops/
|
||||||
dbq.sh read-only psql into any environment's db container
|
dbq.sh read-only psql into any environment's db container
|
||||||
|
|
@ -141,26 +153,26 @@ ACCESS.md, DEPLOY.md, DEPLOY-DEV.md (partly stale — see traps)
|
||||||
```
|
```
|
||||||
|
|
||||||
**Why the project name is pinned.** Compose creates volumes
|
**Why the project name is pinned.** Compose creates volumes
|
||||||
`thermograph_pgdata` / `_appdata` / `_applogs`, and the Swarm stack declares
|
`thermograph_pgdata` / `_appdata` / `_applogs`, and the Swarm stacks declare
|
||||||
those exact names as `external: true` at the same mount paths. Running compose
|
those exact names as `external: true` at the same mount paths. Running compose
|
||||||
from `infra/` without `name: thermograph` derives project `infra`, which makes a
|
from `infra/` without `name: thermograph` derives project `infra`, which makes a
|
||||||
whole new stack with empty volumes next to the running one. `deploy-dev.sh`
|
whole new stack with empty volumes next to the running one. `deploy-dev.sh`
|
||||||
exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to
|
exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to
|
||||||
keep LAN dev separate on purpose. Keep both halves.
|
keep dev separate on purpose. Keep both halves.
|
||||||
|
|
||||||
Deep dive: [08-infra-secrets.md](08-infra-secrets.md).
|
Deep dive: [08-infra-secrets.md](08-infra-secrets.md).
|
||||||
|
|
||||||
## `observability/` — the logging stack
|
## `observability/` — the logging stack
|
||||||
|
|
||||||
```
|
```
|
||||||
docker-compose.yml Loki + Grafana (runs on beta)
|
docker-compose.yml Loki + Grafana (runs on vps1)
|
||||||
loki/config.yml mesh-only, filesystem storage, 30-day retention
|
loki/config.yml mesh-only, filesystem storage, 30-day retention
|
||||||
grafana/provisioning/ datasource + dashboard provider (auto-loaded)
|
grafana/provisioning/ datasource + dashboard provider (auto-loaded)
|
||||||
grafana/provisioning/alerting/ rules, the Discord contact point, the policy
|
grafana/provisioning/alerting/ rules, the Discord contact point, the policy
|
||||||
grafana/dashboards/*.json the fleet-logs dashboard
|
grafana/dashboards/*.json the fleet-logs dashboard
|
||||||
alloy/config.alloy the per-node shipper
|
alloy/config.alloy the per-node shipper
|
||||||
alloy/docker-compose.agent.yml runs Alloy on a node (ALLOY_NODE per host)
|
alloy/docker-compose.agent.yml runs Alloy on a node (ALLOY_NODE per host)
|
||||||
caddy-grafana.conf beta's Grafana vhost, kept for reference
|
caddy-grafana.conf vps1's Grafana vhost, kept for reference
|
||||||
```
|
```
|
||||||
|
|
||||||
No build, no deploy automation — it ships by hand. **The repo is the only
|
No build, no deploy automation — it ships by hand. **The repo is the only
|
||||||
|
|
@ -177,10 +189,10 @@ Deep dive: [09-observability.md](09-observability.md).
|
||||||
| `pr-build.yml` | PR → `dev`/`main` | The `gate` required check. Diffs the PR, builds only touched domains. Deliberately **not** path-filtered. |
|
| `pr-build.yml` | PR → `dev`/`main` | The `gate` required check. Diffs the PR, builds only touched domains. Deliberately **not** path-filtered. |
|
||||||
| `build.yml` | `workflow_call` | Build one domain's image; run the backend suite **inside** the built image. |
|
| `build.yml` | `workflow_call` | Build one domain's image; run the backend suite **inside** the built image. |
|
||||||
| `build-push.yml` | push to `dev`/`main`/`release` touching an app domain, or a `v*.*.*` tag | Builds + pushes `sha-<12hex>` images. A version tag builds **both**. |
|
| `build-push.yml` | push to `dev`/`main`/`release` touching an app domain, or a `v*.*.*` tag | Builds + pushes `sha-<12hex>` images. A version tag builds **both**. |
|
||||||
| `deploy.yml` | push to `main`/`release` touching an app domain | Branch selects environment; matrix covers services; SSHes and runs `deploy.sh`. |
|
| `deploy.yml` | push to `dev`/`main`/`release` touching an app domain | Branch selects environment — three legs: `dev`→vps1, `main`→beta on vps2, `release`→prod on vps2; matrix covers services; SSHes and runs `deploy.sh` with `THERMOGRAPH_ENV` set. |
|
||||||
| `infra-sync.yml` | push to `main` touching `infra/**` | Fast-forwards each host's checkout and re-renders secrets. Rolls **no** service. |
|
| `infra-sync.yml` | push to `main` touching `infra/**` | Fast-forwards beta's and prod's checkouts (both on vps2) and re-renders their secrets. Rolls **no** service. |
|
||||||
| `observability-validate.yml` | push touching `observability/**`, or `workflow_call` | Parses every artifact; validates Alloy config with the pinned binary; strict alerting checks. |
|
| `observability-validate.yml` | push touching `observability/**`, or `workflow_call` | Parses every artifact; validates Alloy config with the pinned binary; strict alerting checks. |
|
||||||
| `ops-cron.yml` | daily 03:00 UTC | **THE prod backup** (pg_dump) + IndexNow. Uses `PROD_SSH_*`. |
|
| `ops-cron.yml` | daily 03:00 UTC | **THE prod backup** (pg_dump) + IndexNow, against prod on vps2. Uses `VPS2_SSH_*`. |
|
||||||
| `secrets-guard.yml` | every PR and push | Fails if any `infra/deploy/secrets/*.yaml` isn't SOPS-encrypted. |
|
| `secrets-guard.yml` | every PR and push | Fails if any `infra/deploy/secrets/*.yaml` isn't SOPS-encrypted. |
|
||||||
| `shell-lint.yml` | every PR and push | shellcheck (pinned v0.11.0 + sha256) over every `*.sh`. |
|
| `shell-lint.yml` | every PR and push | shellcheck (pinned v0.11.0 + sha256) over every `*.sh`. |
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -17,7 +17,7 @@ them is impossible.
|
||||||
| **web** | `THERMOGRAPH_ROLE=web` | Serves requests. Never starts the notifier, even if it would win the election — so the web tier scales to N replicas without also scaling background sweeps. |
|
| **web** | `THERMOGRAPH_ROLE=web` | Serves requests. Never starts the notifier, even if it would win the election — so the web tier scales to N replicas without also scaling background sweeps. |
|
||||||
| **worker** | `THERMOGRAPH_ROLE=worker` | Owns the subscription notifier. Still serves requests today. |
|
| **worker** | `THERMOGRAPH_ROLE=worker` | Owns the subscription notifier. Still serves requests today. |
|
||||||
| **all** | default | Both — the single-process dev default. |
|
| **all** | default | Both — the single-process dev default. |
|
||||||
| **lake** | `THERMOGRAPH_ROLE=lake` | `deploy/entrypoint.sh` execs `uvicorn lake_app:app` on port 8141 instead. No database, no migrations. Prod only. |
|
| **lake** | `THERMOGRAPH_ROLE=lake` | `deploy/entrypoint.sh` execs `uvicorn lake_app:app` on port 8141 instead. No database, no migrations. Runs in every environment — its own Swarm service on prod and beta (`lake` / `beta-lake`), a compose service on dev. |
|
||||||
| **daemon** | compose/Swarm sets the command to `/usr/local/bin/thermograph-daemon` | The Go binary: Discord gateway + cron timers. |
|
| **daemon** | compose/Swarm sets the command to `/usr/local/bin/thermograph-daemon` | The Go binary: Discord gateway + cron timers. |
|
||||||
|
|
||||||
`deploy/entrypoint.sh` runs `alembic upgrade head` (retried — a fresh Postgres
|
`deploy/entrypoint.sh` runs `alembic upgrade head` (retried — a fresh Postgres
|
||||||
|
|
@ -169,7 +169,7 @@ warming, also warm-only.
|
||||||
|
|
||||||
**The catch-all proxy** — the frontend owns every page and asset. In prod and
|
**The catch-all proxy** — the frontend owns every page and asset. In prod and
|
||||||
beta, Caddy path-splits directly, so this proxy is never exercised. It exists as
|
beta, Caddy path-splits directly, so this proxy is never exercised. It exists as
|
||||||
the fallback for environments with no Caddy in front (LAN dev, bare-metal). It
|
the fallback for environments with no Caddy in front (dev, bare-metal). It
|
||||||
forwards `X-Forwarded-Host`/`-Proto` so the frontend can build correct absolute
|
forwards `X-Forwarded-Host`/`-Proto` so the frontend can build correct absolute
|
||||||
URLs instead of resolving to the internal hop's own address. Because it's
|
URLs instead of resolving to the internal hop's own address. Because it's
|
||||||
registered dead last, every real backend route wins first —
|
registered dead last, every real backend route wins first —
|
||||||
|
|
@ -236,7 +236,8 @@ never overlaps a job with itself, and drops ticks that fire mid-run.
|
||||||
|
|
||||||
## The lake (`lake_app.py` + `data/era5lake.py`)
|
## The lake (`lake_app.py` + `data/era5lake.py`)
|
||||||
|
|
||||||
Prod-only Swarm service, same image, `THERMOGRAPH_ROLE=lake`.
|
Runs as a Swarm service on both prod and beta (`lake` / `beta-lake`, same
|
||||||
|
image, `THERMOGRAPH_ROLE=lake`), and as a compose service on dev.
|
||||||
|
|
||||||
The lake is parquet in an S3-compatible bucket (Contabo, `era5-thermograph`),
|
The lake is parquet in an S3-compatible bucket (Contabo, `era5-thermograph`),
|
||||||
extracted once from the public Earthmover ERA5 Icechunk archive by
|
extracted once from the public Earthmover ERA5 Icechunk archive by
|
||||||
|
|
|
||||||
|
|
@ -80,7 +80,7 @@ The static mount is registered last so the explicit routes win.
|
||||||
|
|
||||||
The shell HTML files in `static/` carry an `__ORIGIN__` placeholder for the
|
The shell HTML files in `static/` carry an `__ORIGIN__` placeholder for the
|
||||||
link-preview/Open Graph tags — preview crawlers need absolute URLs and the host
|
link-preview/Open Graph tags — preview crawlers need absolute URLs and the host
|
||||||
differs between LAN and prod. The substituted HTML and its ETag are **memoized
|
differs between dev and prod. The substituted HTML and its ETag are **memoized
|
||||||
per origin**, not recomputed per request.
|
per origin**, not recomputed per request.
|
||||||
|
|
||||||
Origin resolution prefers `X-Forwarded-Host` over `Host`. That matters when the
|
Origin resolution prefers `X-Forwarded-Host` over `Host`. That matters when the
|
||||||
|
|
|
||||||
|
|
@ -3,15 +3,15 @@
|
||||||
## The branch model
|
## The branch model
|
||||||
|
|
||||||
```
|
```
|
||||||
PR ──(required check: `gate`)──▶ dev ──▶ LAN dev box
|
PR ──(required check: `gate`)──▶ dev ──▶ dev (vps1, mesh-only)
|
||||||
│
|
│
|
||||||
promotion PR
|
promotion PR
|
||||||
▼
|
▼
|
||||||
main ──▶ beta.thermograph.org
|
main ──▶ beta.thermograph.org (vps2)
|
||||||
│
|
│
|
||||||
promotion PR ← the owner's call, always
|
promotion PR ← the owner's call, always
|
||||||
▼
|
▼
|
||||||
release ──▶ thermograph.org
|
release ──▶ thermograph.org (vps2)
|
||||||
```
|
```
|
||||||
|
|
||||||
`dev`, `main` and `release` are protected — direct pushes are blocked for
|
`dev`, `main` and `release` are protected — direct pushes are blocked for
|
||||||
|
|
@ -40,9 +40,10 @@ for the separate hotfix path.
|
||||||
**Commit counts are not deliverable content.** Before promoting, compare the two
|
**Commit counts are not deliverable content.** Before promoting, compare the two
|
||||||
tips' *trees*. Identical trees mean the same changes are already on the target
|
tips' *trees*. Identical trees mean the same changes are already on the target
|
||||||
under different SHAs — promoting there merges nothing while still firing the
|
under different SHAs — promoting there merges nothing while still firing the
|
||||||
target's deploy workflows, including `infra-sync` re-rendering
|
target's deploy workflows, including `infra-sync` re-rendering prod's and
|
||||||
`/etc/thermograph.env` on prod and beta. The ahead/behind numbers will look like
|
beta's env files (`/etc/thermograph.env` and `/etc/thermograph-beta.env`, both
|
||||||
real work. `promote` refuses this outright.
|
on vps2). The ahead/behind numbers will look like real work. `promote` refuses
|
||||||
|
this outright.
|
||||||
|
|
||||||
**Expect `dev`, `main` and `release` to be mutually divergent, and expect
|
**Expect `dev`, `main` and `release` to be mutually divergent, and expect
|
||||||
`--ff-only` to fail.** Every promotion merges the source into the target,
|
`--ff-only` to fail.** Every promotion merges the source into the target,
|
||||||
|
|
@ -111,16 +112,17 @@ Triggers on pushes to `dev`/`main`/`release` touching `backend/**` or
|
||||||
- The registry is mesh-only, so the runner host needs
|
- The registry is mesh-only, so the runner host needs
|
||||||
`10.10.0.2 git.thermograph.org` in `/etc/hosts`.
|
`10.10.0.2 git.thermograph.org` in `/etc/hosts`.
|
||||||
|
|
||||||
### `deploy.yml` — one file, two environments, two services
|
### `deploy.yml` — one file, three environments, two services
|
||||||
|
|
||||||
Formerly six near-identical files. Branch selects the environment
|
Formerly six near-identical files. Branch selects the environment
|
||||||
(`main → beta`, `release → prod`); a matrix covers the services.
|
(`dev → dev`, `main → beta`, `release → prod`) — **three legs now, not two** —
|
||||||
|
and a matrix covers the services.
|
||||||
|
|
||||||
The style is **deliberately boring**: no dynamic `fromJSON` matrix, no
|
The style is **deliberately boring**: no dynamic `fromJSON` matrix, no
|
||||||
`cond && secrets.A || secrets.B` ternary. Those are GitHub idioms a Forgejo/act
|
`cond && secrets.A || secrets.B` ternary. Those are GitHub idioms a Forgejo/act
|
||||||
runner may evaluate differently, and the failure mode here is "production does
|
runner may evaluate differently, and the failure mode here is "production does
|
||||||
not deploy" or, worse, "deploys with an empty SSH host". So the two environments
|
not deploy" or, worse, "deploys with an empty SSH host". So each environment
|
||||||
get two explicit, mutually exclusive steps and the tag is computed in shell.
|
gets its own explicit, mutually exclusive step and the tag is computed in shell.
|
||||||
|
|
||||||
Preserved from the originals, all load-bearing:
|
Preserved from the originals, all load-bearing:
|
||||||
|
|
||||||
|
|
@ -129,11 +131,34 @@ Preserved from the originals, all load-bearing:
|
||||||
- 12-hex truncation matching `build-push.yml` exactly.
|
- 12-hex truncation matching `build-push.yml` exactly.
|
||||||
- Per-service, per-environment concurrency with **`cancel-in-progress: false`** —
|
- Per-service, per-environment concurrency with **`cancel-in-progress: false`** —
|
||||||
a half-finished deploy must never be cancelled by a newer one.
|
a half-finished deploy must never be cancelled by a newer one.
|
||||||
- **Separate `PROD_SSH_*` credentials**, so a beta credential leak cannot reach
|
|
||||||
prod.
|
|
||||||
- `appleboy/ssh-action` referenced by full URL; it isn't mirrored in Forgejo's
|
- `appleboy/ssh-action` referenced by full URL; it isn't mirrored in Forgejo's
|
||||||
default action registry.
|
default action registry.
|
||||||
|
|
||||||
|
**SSH credentials are keyed by HOST, not by environment**: `VPS1_SSH_*` and
|
||||||
|
`VPS2_SSH_*`, replacing the old `SSH_*` (which actually meant beta) and
|
||||||
|
`PROD_SSH_*` (which actually meant prod). That old naming used to read as a
|
||||||
|
security boundary — "separate `PROD_SSH_*` credentials, so a beta credential
|
||||||
|
leak cannot reach prod" — and on the old topology it happened to be true,
|
||||||
|
because beta and prod were also on separate hosts. They no longer are: beta and
|
||||||
|
prod are now two co-resident Swarm stacks on vps2, so a `VPS2_SSH_*` credential
|
||||||
|
reaches both by construction, and there is no host-level SSH boundary between
|
||||||
|
them to preserve. The boundary that replaces it lives at two other levels
|
||||||
|
instead: the **database** (beta connects as its own `NOSUPERUSER` role to its
|
||||||
|
own database, `CONNECT` revoked from `PUBLIC` on prod's) and the **filesystem**
|
||||||
|
(separate checkouts, separate rendered env files, separate deploy locks —
|
||||||
|
`env-topology.sh` derives all of it so nothing is shared by accident).
|
||||||
|
|
||||||
|
The credential boundary that *is* still real, and still matters, is
|
||||||
|
**vps1-vs-vps2**: vps1 runs Forgejo, its own CI, and whatever's currently on
|
||||||
|
`dev` — unreviewed by definition, since `dev` is exactly the branch PRs land on
|
||||||
|
before review gates them into `main`. That is precisely why the `dev`
|
||||||
|
environment renders `dev.yaml` alone and never layers `common.yaml` (the
|
||||||
|
fleet's shared production credentials, including a read-write S3 keypair and
|
||||||
|
the VAPID push-signing key) — see [Infra and secrets](08-infra-secrets.md). A
|
||||||
|
`VPS1_SSH_*` leak should never be able to reach a production credential, which
|
||||||
|
is a property `common.yaml`'s dev exclusion buys independently of which SSH key
|
||||||
|
was used to get there.
|
||||||
|
|
||||||
There's also a per-leg refinement: the workflow-level `paths` filter only says
|
There's also a per-leg refinement: the workflow-level `paths` filter only says
|
||||||
*backend or frontend moved*, so each leg re-checks whether **this** push touched
|
*backend or frontend moved*, so each leg re-checks whether **this** push touched
|
||||||
**this** domain. Without it a backend-only push would also roll the frontend,
|
**this** domain. Without it a backend-only push would also roll the frontend,
|
||||||
|
|
@ -144,17 +169,20 @@ skipping silently strands a change.
|
||||||
|
|
||||||
### `infra-sync.yml` — infra's own pipeline
|
### `infra-sync.yml` — infra's own pipeline
|
||||||
|
|
||||||
Push to `main` touching `infra/**` → SSH to beta **and** prod, fast-forward
|
Push to `main` touching `infra/**` → SSH to vps2, fast-forward **both**
|
||||||
`/opt/thermograph`, re-render `/etc/thermograph.env` from the vault.
|
checkouts there (`/opt/thermograph` for prod, `/opt/thermograph-beta` for
|
||||||
|
beta), re-render each one's own env file from the vault.
|
||||||
|
|
||||||
**Rolls no service.** Image tags are the app domains' axis, not infra's. A
|
**Rolls no service.** Image tags are the app domains' axis, not infra's. A
|
||||||
compose change that must recreate containers takes effect on the next app
|
compose or stack change that must recreate containers takes effect on the next
|
||||||
deploy, or on a by-hand `SERVICE=all … infra/deploy/deploy.sh`.
|
app deploy, or on a by-hand `SERVICE=all … infra/deploy/deploy.sh` (or
|
||||||
|
`deploy-dev.sh` for dev).
|
||||||
|
|
||||||
Note the asymmetry: app code *is* environment-staged (`dev`→`main`→`release`
|
Note the asymmetry: app code *is* environment-staged (`dev`→`main`→`release`
|
||||||
maps to LAN→beta→prod via image tags); **infra is not** — both hosts track infra
|
maps to dev(vps1)→beta→prod(vps2) via image tags); **infra is not** — beta's and
|
||||||
via `main`. Prod's *app images* are staged by `release`, but its checkout
|
prod's checkouts both track infra via `main`, while dev's checkout tracks `dev`
|
||||||
follows `main`.
|
itself. Prod's *app images* are staged by `release`, but its checkout follows
|
||||||
|
`main`.
|
||||||
|
|
||||||
### `observability-validate.yml`
|
### `observability-validate.yml`
|
||||||
|
|
||||||
|
|
@ -171,13 +199,17 @@ silent no-op is exactly what this check exists to prevent.
|
||||||
|
|
||||||
### `ops-cron.yml` — the prod backup
|
### `ops-cron.yml` — the prod backup
|
||||||
|
|
||||||
Daily at 03:00 UTC: `pg_dump` on prod, plus an IndexNow ping. Both SSH into prod
|
Daily at 03:00 UTC: `pg_dump` on prod, plus an IndexNow ping. Both SSH into
|
||||||
and run inside the already-running stack.
|
vps2 and run inside prod's already-running stack — specifically prod's, not
|
||||||
|
beta's, even though both now live on the same box.
|
||||||
|
|
||||||
**It uses `PROD_SSH_*`, not `SSH_*`.** An earlier revision reused `SSH_*` — so
|
**It uses `VPS2_SSH_*`, not a beta-flavoured secret.** An earlier revision (on
|
||||||
the "prod" backup was silently dumping *beta*, and prod had no backup at all. If
|
the old topology) reused the credential meant for beta's host — so the "prod"
|
||||||
you touch this file, verify a dump actually lands in
|
backup was silently dumping *beta*, and prod had no backup at all. On vps2's
|
||||||
`agent@prod:~/thermograph-backups/`.
|
co-residency, the host-keyed secret alone doesn't disambiguate prod from beta
|
||||||
|
the way it used to; the script also needs to target prod's checkout and `db`
|
||||||
|
specifically. If you touch this file, verify a dump actually lands in
|
||||||
|
`agent@vps2:~/thermograph-backups/` and that it's prod's data, not beta's.
|
||||||
|
|
||||||
### `secrets-guard.yml` and `shell-lint.yml` — the backstops
|
### `secrets-guard.yml` and `shell-lint.yml` — the backstops
|
||||||
|
|
||||||
|
|
@ -199,7 +231,8 @@ guard.
|
||||||
|
|
||||||
## Runner facts
|
## Runner facts
|
||||||
|
|
||||||
- Jobs run on the `docker` label — the LAN box's Forgejo Actions runner.
|
- Jobs run on the `docker` label — vps1's Forgejo Actions runner (vps1 is where
|
||||||
|
Forgejo itself lives; the desktop no longer runs a runner).
|
||||||
- Job containers get the **host's docker.sock** automounted
|
- Job containers get the **host's docker.sock** automounted
|
||||||
(Docker-outside-of-Docker, no privileged mode). The job image
|
(Docker-outside-of-Docker, no privileged mode). The job image
|
||||||
(`node:20-bookworm`) ships no docker CLI, which is why several workflows
|
(`node:20-bookworm`) ships no docker CLI, which is why several workflows
|
||||||
|
|
|
||||||
|
|
@ -7,23 +7,36 @@ Read [`infra/CLAUDE.md`](../../infra/CLAUDE.md) alongside this.
|
||||||
|
|
||||||
## The machines
|
## The machines
|
||||||
|
|
||||||
| Host | Public | Mesh | Orchestrator | Login |
|
Two VPS boxes named by **role**, not by environment, plus the operator's
|
||||||
|
desktop:
|
||||||
|
|
||||||
|
| Host | Public | Mesh | Runs | Login |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| **prod** | `169.58.46.181` / thermograph.org | `10.10.0.1` | Docker **Swarm** | `agent`, passwordless sudo |
|
| **vps1** | `75.119.132.91` / git.thermograph.org, dashboard.thermograph.org | `10.10.0.2` | Forgejo, Grafana + Loki + Alloy, `emigriffith.dev`, and **dev** (own Postgres, compose, mesh-only) | `agent`, passwordless sudo |
|
||||||
| **beta** | `75.119.132.91` / beta.thermograph.org | `10.10.0.2` | **compose** | `agent`, passwordless sudo |
|
| **vps2** | `169.58.46.181` / thermograph.org, beta.thermograph.org | `10.10.0.1` | **prod** and **beta**, as two co-resident Docker **Swarm** stacks; Centralis; Postfix; backups | `agent`, passwordless sudo |
|
||||||
| **desktop** | — | `10.10.0.3` | compose (LAN dev) | it's your box |
|
| **desktop** | — | `10.10.0.3` | AI-model hosting + flex Swarm capacity. No Thermograph environment; `make dev-up` is a laptop convenience there | it's your box |
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # prod
|
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # vps2 (prod + beta)
|
||||||
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # beta
|
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # vps1 (Forgejo + Grafana + dev)
|
||||||
```
|
```
|
||||||
|
|
||||||
Beta also hosts **Forgejo** (git + CI + registry) and **Grafana + Loki**, which
|
The mesh IPs did not move when this topology landed — vps1 is the box that used
|
||||||
is why beta is guarded as strictly as prod: a destructive command there takes
|
to be called "beta" (still mesh `10.10.0.2`), vps2 the one that used to be
|
||||||
out git, CI and the registry at once. The LAN dev box is deliberately
|
called "prod" (still mesh `10.10.0.1`). What moved is which environment runs on
|
||||||
unguarded.
|
which box: beta moved from vps1 onto vps2 to sit next to prod, and dev moved
|
||||||
|
onto vps1.
|
||||||
|
|
||||||
Every root-effective command on the VPSes is logged by `auditd`
|
**vps1 is guarded as strictly as vps2**, not more loosely because it "used to
|
||||||
|
be beta" or because dev now lives there. It hosts Forgejo, its CI runner and
|
||||||
|
the registry, plus Grafana/Loki — a destructive command there takes out git, CI
|
||||||
|
and the registry at once. Dev being mesh-only and a public VPS's tenant, rather
|
||||||
|
than someone's desktop, is exactly why it gets the same guard as everything
|
||||||
|
else on that box: it shares a kernel with the estate's source of truth for code
|
||||||
|
and the monitoring stack, sudo there is passwordless, and it runs whatever
|
||||||
|
branch is currently in flight.
|
||||||
|
|
||||||
|
Every root-effective command on both VPSes is logged by `auditd`
|
||||||
(`ausearch -k agentcmd`). That's a feature — fixes are attributable.
|
(`ausearch -k agentcmd`). That's a feature — fixes are attributable.
|
||||||
|
|
||||||
Prefer Centralis for routine work (`run_on_host`, `fleet_status`,
|
Prefer Centralis for routine work (`run_on_host`, `fleet_status`,
|
||||||
|
|
@ -31,20 +44,36 @@ Prefer Centralis for routine work (`run_on_host`, `fleet_status`,
|
||||||
|
|
||||||
## The two orchestrators
|
## The two orchestrators
|
||||||
|
|
||||||
Which path a host takes is decided by **`/etc/thermograph/deploy-mode`**: the
|
Which path an environment takes is decided by **`infra/deploy/env-topology.sh`**
|
||||||
string `stack` makes `deploy.sh` exec `deploy/stack/deploy-stack.sh`; anything
|
(`thermograph_topology <env>`, keyed on `THERMOGRAPH_ENV=dev|beta|prod`): it
|
||||||
else is compose. The workflows never need to know which mode a host runs.
|
derives the checkout, branch, deploy mode, stack/compose name, env file, LB
|
||||||
|
ports and DB role for that environment, and `deploy.sh` execs
|
||||||
|
`deploy/stack/deploy-stack.sh` when the mode is `stack`, or rolls compose
|
||||||
|
otherwise. The old host-wide `/etc/thermograph/deploy-mode` marker still exists
|
||||||
|
as a **fallback** for a by-hand run on a box that predates this file, but it
|
||||||
|
cannot describe vps2 on its own — vps2 runs two environments, so "which mode
|
||||||
|
does this host use" is no longer a well-formed question there.
|
||||||
|
|
||||||
| | compose (beta, LAN dev) | Swarm (prod) |
|
| | compose (dev) | Swarm (prod, beta) |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| File | `infra/docker-compose.yml` | `infra/deploy/stack/thermograph-stack.yml` |
|
| Host | vps1 | vps2, both stacks co-resident |
|
||||||
| Services | `db`, `backend`, `lake`, `daemon`, `frontend` | `db`, `web`, `worker`, `lake`, `daemon`, `frontend`, `autoscaler`, `autoscaler-lake` |
|
| File | `infra/docker-compose.yml` | `infra/deploy/stack/thermograph-stack.yml` (prod), `thermograph-beta-stack.yml` (beta) |
|
||||||
|
| Services | `db`, `backend`, `lake`, `daemon`, `frontend` | prod: `db`, `web`, `worker`, `lake`, `daemon`, `frontend`, `autoscaler`, `autoscaler-lake`. beta: `beta-web`, `beta-worker`, `beta-lake`, `beta-daemon`, `beta-frontend` — prefixed, no `db`, no autoscalers |
|
||||||
| Rolling | `up -d --no-deps <targets>` | start-first, health-gated, auto-rollback |
|
| Rolling | `up -d --no-deps <targets>` | start-first, health-gated, auto-rollback |
|
||||||
| Tag file | `deploy/.image-tags.env` | `deploy/.stack-image-tags.env` |
|
| Tag file | `deploy/.image-tags.env` | `deploy/.stack-image-tags.env` |
|
||||||
|
|
||||||
Note prod splits `backend` into **`web`** and **`worker`** (the
|
Note prod splits `backend` into **`web`** and **`worker`** (the
|
||||||
`THERMOGRAPH_ROLE` split from [04](04-backend.md)); compose runs one `backend`
|
`THERMOGRAPH_ROLE` split from [04](04-backend.md)); dev's compose stack runs one
|
||||||
service in role `all`.
|
`backend` service in role `all`. Beta's Swarm services are prefixed
|
||||||
|
(`beta-web`, not `web`) because Swarm registers a service's short name as a DNS
|
||||||
|
alias on every network it joins — two stacks both naming a service `web` on
|
||||||
|
the shared `data` network would make `web` ambiguous, and beta joins that
|
||||||
|
network (declared `external` in its stack file) purely to reach prod's `db`.
|
||||||
|
Beta keeps its own `internal` overlay for beta-to-beta traffic and has no `db`
|
||||||
|
service of its own: **one TimescaleDB instance serves both environments**, on
|
||||||
|
separate databases (`thermograph` / `thermograph_beta`) and separate roles
|
||||||
|
(beta's is `NOSUPERUSER`/`NOCREATEDB`/`NOCREATEROLE`, `CONNECT` revoked from
|
||||||
|
`PUBLIC` on prod's database).
|
||||||
|
|
||||||
`STACK_TEST=1` rehearses the entire Swarm deploy under stack name
|
`STACK_TEST=1` rehearses the entire Swarm deploy under stack name
|
||||||
`thermograph-test` on throwaway volumes and ports `18137`/`18080`. Leftover
|
`thermograph-test` on throwaway volumes and ports `18137`/`18080`. Leftover
|
||||||
|
|
@ -58,11 +87,15 @@ coupling is why `name: thermograph` is pinned in the compose file — running
|
||||||
compose from `infra/` without it derives project `infra`, silently creating a
|
compose from `infra/` without it derives project `infra`, silently creating a
|
||||||
whole new stack with empty volumes beside the running one. `deploy-dev.sh`
|
whole new stack with empty volumes beside the running one. `deploy-dev.sh`
|
||||||
exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to
|
exports `COMPOSE_PROJECT_NAME=thermograph-dev` (env wins over the file key) to
|
||||||
keep LAN dev separate on purpose. **Keep both halves.**
|
keep dev separate on purpose. **Keep both halves.**
|
||||||
|
|
||||||
## `deploy/deploy.sh` — read this before you touch a deploy
|
## `deploy/deploy.sh` — read this before you touch a deploy
|
||||||
|
|
||||||
Single entry point for beta and prod. Contract:
|
Single entry point for dev, beta and prod — the same script for all three;
|
||||||
|
`infra/deploy/env-topology.sh` is what tells it which checkout, branch, stack
|
||||||
|
and ports apply. Contract, run from that environment's own checkout
|
||||||
|
(`/opt/thermograph` for prod, `/opt/thermograph-beta` for beta, both on vps2;
|
||||||
|
`/opt/thermograph-dev` on vps1 via the `deploy-dev.sh` wrapper):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/infra/deploy/deploy.sh
|
SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/infra/deploy/deploy.sh
|
||||||
|
|
@ -76,13 +109,17 @@ What it does, in order, and why each step is the way it is:
|
||||||
the whole run. Backend and frontend deploys can fire for the same push
|
the whole run. Backend and frontend deploys can fire for the same push
|
||||||
seconds apart and both SSH into one checkout; concurrent runs race the git
|
seconds apart and both SSH into one checkout; concurrent runs race the git
|
||||||
reset, the compose project and the tag file. `-w 600` bounds the wait.
|
reset, the compose project and the tag file. `-w 600` bounds the wait.
|
||||||
2. **Render secrets** from the SOPS vault into `/etc/thermograph.env`, then
|
2. **Render secrets** from the SOPS vault into that environment's own env file
|
||||||
source it, so a by-hand run interpolates the same as the systemd unit does.
|
(`/etc/thermograph.env` for prod, `/etc/thermograph-beta.env` for beta,
|
||||||
Guarded on the helper's existence so the very deploy that *introduces*
|
likewise `/etc/thermograph.env` on dev's separate host), then source it, so
|
||||||
`render-secrets.sh` is safe.
|
a by-hand run interpolates the same as CI's does. Guarded on the helper's
|
||||||
|
existence so the very deploy that *introduces* `render-secrets.sh` is safe.
|
||||||
3. **`git reset --hard origin/$BRANCH`** on the checkout root (`BRANCH` defaults
|
3. **`git reset --hard origin/$BRANCH`** on the checkout root (`BRANCH` defaults
|
||||||
to `main`). ⚠️ **Uncommitted edits in `/opt/thermograph` evaporate here.**
|
to `main` for beta/prod, `dev` for dev). ⚠️ **Uncommitted edits in that
|
||||||
The one exception is `deploy/.image-tags.env`, untracked on purpose.
|
checkout evaporate here** — on vps2 that's `/opt/thermograph` for a prod
|
||||||
|
deploy or `/opt/thermograph-beta` for a beta one; each reset only ever
|
||||||
|
touches its own checkout, never the sibling's. The one exception is
|
||||||
|
`deploy/.image-tags.env`, untracked on purpose.
|
||||||
4. **Read `.image-tags.env`** so a single-service roll re-renders compose with
|
4. **Read `.image-tags.env`** so a single-service roll re-renders compose with
|
||||||
*both* services' real tags and never accidentally recreates or downgrades the
|
*both* services' real tags and never accidentally recreates or downgrades the
|
||||||
sibling.
|
sibling.
|
||||||
|
|
@ -115,12 +152,15 @@ What it does, in order, and why each step is the way it is:
|
||||||
|
|
||||||
Rollback is redeploying the previous image tag. But **do not treat image
|
Rollback is redeploying the previous image tag. But **do not treat image
|
||||||
retention as a rollback guarantee**: both `deploy.sh` and `deploy-stack.sh`
|
retention as a rollback guarantee**: both `deploy.sh` and `deploy-stack.sh`
|
||||||
*end* by deleting images outside the running pair. On beta that succeeds, so
|
*end* by deleting images outside the running pair. Beta now runs Swarm exactly
|
||||||
beta typically holds **no local rollback target at all**; on prod it usually
|
like prod (it used to be compose, on the old topology, where this pruning
|
||||||
fails only because Swarm's stopped task containers still hold references, which
|
reliably left no rollback target at all); on both Swarm stacks it usually
|
||||||
is why prod keeps a handful. Verify the target tag is actually present before
|
fails only because Swarm's stopped task containers still hold references,
|
||||||
promising a rollback — Centralis's `rollback_to(dry_run: true)` checks both host
|
which is why prod and beta both tend to keep a handful. Dev, the one remaining
|
||||||
and registry.
|
compose environment, is the one where this prune still reliably succeeds and
|
||||||
|
leaves nothing to roll back to. Verify the target tag is actually present
|
||||||
|
before promising a rollback — Centralis's `rollback_to(dry_run: true)` checks
|
||||||
|
both host and registry.
|
||||||
|
|
||||||
`docker system prune -a` on a live box is forbidden for the same reason.
|
`docker system prune -a` on a live box is forbidden for the same reason.
|
||||||
`docker image prune -f` (dangling only) is the safe form.
|
`docker image prune -f` (dangling only) is the safe form.
|
||||||
|
|
@ -129,55 +169,81 @@ and registry.
|
||||||
|
|
||||||
**The single source of truth is `infra/deploy/secrets/*.yaml`**, committed
|
**The single source of truth is `infra/deploy/secrets/*.yaml`**, committed
|
||||||
encrypted (values only — keys stay readable so diffs mean something) and
|
encrypted (values only — keys stay readable so diffs mean something) and
|
||||||
rendered into `/etc/thermograph.env` at deploy time by
|
rendered at deploy time by `deploy/render-secrets.sh` into each environment's
|
||||||
`deploy/render-secrets.sh`.
|
own env file (`/etc/thermograph.env` for prod, `/etc/thermograph-beta.env` for
|
||||||
|
beta, `/etc/thermograph.env` again on dev's separate host).
|
||||||
|
|
||||||
**Cycling a key is: edit → commit → deploy.** No SSH, no hand-edited root file,
|
**Cycling a key is: edit → commit → deploy.** No SSH, no hand-edited root file,
|
||||||
no per-host duplication.
|
no per-host duplication.
|
||||||
|
|
||||||
| File | Holds |
|
| File | Holds |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `common.yaml` | the 16 values identical on prod **and** beta — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config |
|
| `common.yaml` | the 16 values identical on prod **and** beta — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. **Never layered under `dev`** — see below |
|
||||||
| `prod.yaml` | prod's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL`, Discord + mail credentials that exist nowhere else |
|
| `prod.yaml` | prod's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL`, Discord + mail credentials that exist nowhere else |
|
||||||
| `beta.yaml` | beta's own — the three held-back credentials, sizing, base URL |
|
| `beta.yaml` | beta's own — the three held-back credentials, sizing, base URL |
|
||||||
| `dev.yaml` | the LAN box's own, **self-contained**, 12 values, no production credential |
|
| `dev.yaml` | dev's own, **self-contained**, 12 values, no production credential |
|
||||||
| `centralis.prod.yaml` | Centralis's nine variables → `/etc/centralis.env`, its own renderer |
|
| `centralis.prod.yaml` | Centralis's nine variables → `/etc/centralis.env`, its own renderer |
|
||||||
| `example.yaml` | format reference / CI fixture (fake values) |
|
| `example.yaml` | format reference / CI fixture (fake values) |
|
||||||
| `../../.sops.yaml` | which age recipients files encrypt to (plaintext config) |
|
| `../../.sops.yaml` | which age recipients files encrypt to (plaintext config) |
|
||||||
|
|
||||||
The renderer concatenates `common.yaml` then `<env>.yaml`, so a **host value
|
The renderer concatenates `common.yaml` then `<env>.yaml`, so a **host value
|
||||||
wins** (last occurrence of a duplicate key). `<env>` comes from
|
wins** (last occurrence of a duplicate key). `<env>` is passed explicitly as
|
||||||
`/etc/thermograph/secrets-env` on the box.
|
`THERMOGRAPH_ENV` by the deploy caller — required now that vps2 alone hosts two
|
||||||
|
environments and a host-wide marker can't tell them apart; the marker
|
||||||
|
(`/etc/thermograph/secrets-env`) survives only as the fallback for a by-hand
|
||||||
|
run on a single-environment box.
|
||||||
|
|
||||||
### Two design decisions worth understanding
|
### Two design decisions worth understanding — re-derived for vps1/vps2
|
||||||
|
|
||||||
**Three credentials are deliberately *not* in `common.yaml`** —
|
**Three credentials are deliberately *not* in `common.yaml`** —
|
||||||
`POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET`, `THERMOGRAPH_DATABASE_URL`.
|
`POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET`, `THERMOGRAPH_DATABASE_URL`.
|
||||||
They hold identical values on prod and beta today, so by the mechanical rule
|
They hold identical *values* on prod and beta today (beta was seeded from
|
||||||
they belong there. They're kept per-host anyway because these are the
|
prod), so by the mechanical rule they'd belong in `common.yaml`. They're kept
|
||||||
credentials that let one environment *act as* another: with them, a foothold on
|
per-host anyway, and the reason used to be phrased as a host-isolation
|
||||||
beta (the more exposed box — public Forgejo and Grafana) is a foothold on prod's
|
argument — "a foothold on beta is a foothold on prod's database, and beta is
|
||||||
database and prod's session signing. They match because beta was seeded from
|
the more exposed box." **That framing no longer holds**: beta and prod are now
|
||||||
prod, not because the two are meant to be one system. Keeping them per-host
|
two Swarm stacks co-resident on vps2, so a foothold on the host reaches both
|
||||||
costs one extra line and buys the ability to diverge.
|
regardless of which file a credential lives in. What these three credentials
|
||||||
|
still buy, correctly stated, is a **database-level** and **file-level**
|
||||||
|
boundary, not a host-level one: beta's `THERMOGRAPH_DATABASE_URL` names its own
|
||||||
|
`NOSUPERUSER` role (`thermograph_beta`) against its own database on the shared
|
||||||
|
TimescaleDB instance, with `CONNECT` revoked from `PUBLIC` on prod's database,
|
||||||
|
and each environment's `POSTGRES_PASSWORD`/`THERMOGRAPH_AUTH_SECRET` live in a
|
||||||
|
separate file so rotating prod's stops implying "and beta's too." Keeping them
|
||||||
|
per-host costs one extra line and buys the ability to diverge — that part is
|
||||||
|
unchanged. What changed is what "diverge" is defending against: not a beta
|
||||||
|
compromise reaching a separate prod host, but a beta compromise reaching
|
||||||
|
prod's role and session-signing key on the host they now already share.
|
||||||
|
|
||||||
**`dev` renders `dev.yaml` alone** — `deploy-dev.sh` exports
|
**The credential boundary that actually maps to hosts now is vps1-vs-vps2, not
|
||||||
`THERMOGRAPH_SECRETS_SKIP_COMMON=1`. Eleven of `common.yaml`'s sixteen values
|
beta-vs-prod** — which is exactly why `dev` renders `dev.yaml` **alone**,
|
||||||
are live production credentials, including a **read-write** object-storage
|
never layering `common.yaml` (the fleet's shared production credentials):
|
||||||
keypair on the bucket that also holds prod's database backups, and the VAPID
|
`deploy-dev.sh` exports `THERMOGRAPH_SECRETS_SKIP_COMMON=1`. Eleven of
|
||||||
private key that signs push to real subscribers. The render is plaintext
|
`common.yaml`'s sixteen values are live production credentials, including a
|
||||||
concatenation, so layering it would put all of those into every container in the
|
**read-write** object-storage keypair on the bucket that also holds prod's
|
||||||
dev stack — on the operator's desktop, which is also the CI runner whose
|
database backups, and the VAPID private key that signs push to real
|
||||||
`docker`-labelled jobs get the host socket, running unreviewed `dev` code. An
|
subscribers. The render is plaintext concatenation, so layering it would put
|
||||||
|
all of those into every container in the dev stack — on vps1, the box that
|
||||||
|
also runs Forgejo and its CI runner, whose `docker`-labelled jobs get the host
|
||||||
|
socket, running whatever branch is currently `dev` (unreviewed by definition:
|
||||||
|
that's the branch PRs land on before the gate promotes them to `main`). An
|
||||||
override in `dev.yaml` wouldn't help: last-wins governs *consumers*, but the
|
override in `dev.yaml` wouldn't help: last-wins governs *consumers*, but the
|
||||||
production value is still physically a line in the file.
|
production value is still physically a line in the file. This is the load-
|
||||||
|
bearing boundary in the vault today, not a per-environment credential split
|
||||||
|
that co-residency has already dissolved at the host level.
|
||||||
|
|
||||||
Dev works fine without them: no S3 keys means the `lake` service answers 503 and
|
Dev works fine without `common.yaml`'s values: no S3 keys means the `lake`
|
||||||
history falls through to the Open-Meteo archive (an accelerator, never a
|
service answers 503 and history falls through to the Open-Meteo archive (an
|
||||||
dependency); VAPID and IndexNow both self-generate and persist to the `appdata`
|
accelerator, never a dependency); VAPID and IndexNow both self-generate and
|
||||||
volume; with no metrics token `/api/v2/metrics` is direct-loopback-only, which
|
persist to the `appdata` volume; with no metrics token `/api/v2/metrics` is
|
||||||
is right on a LAN box. One `common.yaml` value is simply *wrong* for dev —
|
direct-loopback-only, which is right for a mesh-only box. One `common.yaml`
|
||||||
`THERMOGRAPH_COOKIE_SECURE=1` silently breaks login over plain HTTP.
|
value is simply *wrong* for dev — `THERMOGRAPH_COOKIE_SECURE=1` silently breaks
|
||||||
|
login over dev's plain-HTTP mesh URL.
|
||||||
|
|
||||||
|
None of this makes dev unimportant or unguarded. It's a public VPS's tenant
|
||||||
|
now, sharing a box with Forgejo and the monitoring stack — not somebody's
|
||||||
|
desktop — which is precisely why it gets its own exclusion from `common.yaml`
|
||||||
|
rather than being treated as beneath the vault's concern.
|
||||||
|
|
||||||
### Working with the vault
|
### Working with the vault
|
||||||
|
|
||||||
|
|
@ -223,12 +289,14 @@ infra/ops/iceberg.sh prod -c "select count(*) from era5_daily"
|
||||||
```
|
```
|
||||||
|
|
||||||
Both exec **into the container**. No database is exposed over TCP — each listens
|
Both exec **into the container**. No database is exposed over TCP — each listens
|
||||||
only on its private docker network, and prod's is a Swarm **overlay the prod
|
only on its private docker network, and prod's and beta's are both Swarm
|
||||||
host itself cannot route to**, so `ssh -L` works for beta and is *impossible*
|
**overlays the vps2 host itself cannot route to**, so `ssh -L` is *impossible*
|
||||||
for prod. Exec works identically everywhere with no ports, no tunnels, no infra
|
for either. (Dev's, on vps1, is plain compose, so `ssh -L` would actually work
|
||||||
changes. The prod container name is a Swarm task name that changes on every
|
there — but exec is used uniformly across environments anyway, so it doesn't
|
||||||
redeploy, so it's resolved at call time via `docker ps --filter name=`, never
|
matter which one happens to allow the shortcut.) Exec works identically
|
||||||
hardcoded.
|
everywhere with no ports, no tunnels, no infra changes. The prod and beta
|
||||||
|
container names are Swarm task names that change on every redeploy, so they're
|
||||||
|
resolved at call time via `docker ps --filter name=`, never hardcoded.
|
||||||
|
|
||||||
Queries connect as **`thermograph_ro`** — `NOSUPERUSER`, granted only
|
Queries connect as **`thermograph_ro`** — `NOSUPERUSER`, granted only
|
||||||
`pg_read_all_data`. Read-only is enforced by Postgres, not by convention:
|
`pg_read_all_data`. Read-only is enforced by Postgres, not by convention:
|
||||||
|
|
@ -244,8 +312,9 @@ confirmation — on prod, know what you are doing.
|
||||||
|
|
||||||
## Backups
|
## Backups
|
||||||
|
|
||||||
The nightly `ops-cron.yml` `pg_dump` into `agent@prod:~/thermograph-backups/`
|
The nightly `ops-cron.yml` `pg_dump` into `agent@vps2:~/thermograph-backups/`
|
||||||
is it. Known gaps, stated plainly: **a single copy on the same box as the
|
(prod's data specifically — see [CI and release](07-ci-and-release.md)) is it.
|
||||||
|
Known gaps, stated plainly: **a single copy on the same box as the
|
||||||
database, no offsite yet**, and restores must handle TimescaleDB's
|
database, no offsite yet**, and restores must handle TimescaleDB's
|
||||||
`continuous_agg` circular-FK warning (`--disable-triggers`). The `backups/`
|
`continuous_agg` circular-FK warning (`--disable-triggers`). The `backups/`
|
||||||
prefix in the object-storage bucket belongs to those jobs — don't write outside
|
prefix in the object-storage bucket belongs to those jobs — don't write outside
|
||||||
|
|
|
||||||
|
|
@ -1,26 +1,34 @@
|
||||||
# 9. Observability
|
# 9. Observability
|
||||||
|
|
||||||
Fleet-wide log aggregation: **Loki + Grafana on beta**, fed by a **Grafana
|
Fleet-wide log aggregation: **Loki + Grafana on vps1**, fed by a **Grafana
|
||||||
Alloy** agent on every node, all over the WireGuard mesh.
|
Alloy** agent on every node, all over the WireGuard mesh. vps2 runs two
|
||||||
|
environments (beta, prod) behind that one Alloy agent; vps1 runs Alloy
|
||||||
|
alongside the things it's shipping logs *from* (Forgejo, Grafana itself, dev).
|
||||||
|
|
||||||
```
|
```
|
||||||
prod (10.10.0.1) beta (10.10.0.2) desktop / LAN dev (10.10.0.3)
|
vps2 (10.10.0.1) vps1 (10.10.0.2) desktop (10.10.0.3)
|
||||||
|
prod + beta Loki, Grafana, dev
|
||||||
┌────────────┐ ┌──────────────────┐ ┌────────────┐
|
┌────────────┐ ┌──────────────────┐ ┌────────────┐
|
||||||
│ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌───│ Alloy agent│
|
│ Alloy agent│──wg0──┐ │ Loki ◀── Alloy │ ┌───│ Alloy agent│
|
||||||
└────────────┘ └──▶│ Grafana (Caddy) │◀─┘ └────────────┘
|
└────────────┘ └──▶│ Grafana (Caddy) │◀─┘ └────────────┘
|
||||||
docker+caddy+app └──────────────────┘ docker+caddy+app
|
docker+caddy+app └──────────────────┘ docker (AI models)
|
||||||
```
|
```
|
||||||
|
|
||||||
- **Loki** — filesystem storage, **30-day retention**, listening on the mesh IP
|
- **Loki** — filesystem storage, **30-day retention**, listening on the mesh IP
|
||||||
`10.10.0.2:3100`. Never public.
|
`10.10.0.2:3100`. Never public.
|
||||||
- **Grafana** — the UI, fronted by beta's Caddy at
|
- **Grafana** — the UI, fronted by vps1's Caddy at
|
||||||
**`dashboard.thermograph.org`** (Google SSO, pre-provisioned accounts only,
|
**`dashboard.thermograph.org`** (Google SSO, pre-provisioned accounts only,
|
||||||
`allow_sign_up=false`, plus a break-glass local admin). Use that hostname
|
`allow_sign_up=false`, plus a break-glass local admin). Use that hostname
|
||||||
everywhere; **never** `grafana.thermograph.org`.
|
everywhere; **never** `grafana.thermograph.org`.
|
||||||
- **Alloy** — on each node, shipping three sources: every Docker container's
|
- **Alloy** — one agent per *host*, not per environment: vps2's single agent
|
||||||
stdout/stderr, Caddy's host access logs, and the app's structured JSON logs
|
ships logs for both beta and prod, vps1's ships Forgejo/Grafana/dev, and the
|
||||||
(`errors`/`access`/`audit` `*.jsonl`, parsed so `level`/`tag`/`phase` become
|
desktop's ships whatever runs there now. Each ships three sources: every
|
||||||
labels). Every line is tagged `host = prod|beta|dev`.
|
Docker container's stdout/stderr, Caddy's host access logs, and the app's
|
||||||
|
structured JSON logs (`errors`/`access`/`audit` `*.jsonl`, parsed so
|
||||||
|
`level`/`tag`/`phase` become labels). Every app log line is tagged
|
||||||
|
`host = prod|beta|dev` — that label is the *environment*, not the box, which
|
||||||
|
matters on vps2 where one Alloy agent ships lines carrying two different
|
||||||
|
`host` values.
|
||||||
|
|
||||||
There is **no build and no deploy automation** for this domain — it ships by
|
There is **no build and no deploy automation** for this domain — it ships by
|
||||||
hand. But `observability-validate.yml` parses every artifact that ships to a
|
hand. But `observability-validate.yml` parses every artifact that ships to a
|
||||||
|
|
@ -42,20 +50,24 @@ Start wide, then narrow:
|
||||||
2. `logs_overview since="1h"` — where the volume is, and what has gone quiet.
|
2. `logs_overview since="1h"` — where the volume is, and what has gone quiet.
|
||||||
3. `logs_query service=… host=…` — the actual lines.
|
3. `logs_query service=… host=…` — the actual lines.
|
||||||
|
|
||||||
### ⚠ The prod trap
|
### ⚠ The prod (and now beta) trap
|
||||||
|
|
||||||
Prod runs a Swarm stack, and its `job="docker"` streams carry **no `service`
|
Prod and beta both run Swarm stacks — co-resident on vps2 — and their
|
||||||
label**. A raw `{host="prod", service="backend"}` query matches nothing and
|
`job="docker"` streams carry **no `service` label**. A raw
|
||||||
reads as *"no logs"* when it actually means *"wrong selector"*.
|
`{host="prod", service="backend"}` query matches nothing and reads as *"no
|
||||||
|
logs"* when it actually means *"wrong selector"*. This used to be a prod-only
|
||||||
|
trap when beta ran compose; it now applies to both environments on vps2, and
|
||||||
|
dev (compose, on vps1) is the one environment this does *not* affect.
|
||||||
|
|
||||||
Use Centralis's `logs_query` `service` argument, which rewrites it to a
|
Use Centralis's `logs_query` `service` argument, which rewrites it to a
|
||||||
container name-regex with the churning task suffix wildcarded. Prod service
|
container name-regex with the churning task suffix wildcarded. Prod service
|
||||||
names are `thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`,
|
names are `thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`,
|
||||||
`_lake`.
|
`_lake`; beta's are the same names prefixed `beta-` (`beta-web`, `beta-worker`,
|
||||||
|
…) with no `_db` or autoscaler equivalents.
|
||||||
|
|
||||||
Related: **prod container names change on every redeploy** (the Swarm task
|
Related: **prod and beta container names both change on every redeploy** (the
|
||||||
suffix). Never hardcode one; resolve via `docker ps --filter name=` at call
|
Swarm task suffix). Never hardcode one; resolve via `docker ps --filter name=`
|
||||||
time.
|
at call time.
|
||||||
|
|
||||||
## The dashboard
|
## The dashboard
|
||||||
|
|
||||||
|
|
@ -81,16 +93,16 @@ Three reasons, all learned the hard way:
|
||||||
- Grafana's factory default pointed at the literal string
|
- Grafana's factory default pointed at the literal string
|
||||||
`<example@email.com>` — a paging path that had never delivered a message to
|
`<example@email.com>` — a paging path that had never delivered a message to
|
||||||
anyone.
|
anyone.
|
||||||
- Beta's Grafana relays SMTP through **prod's** Postfix. Email alerts therefore
|
- vps1's Grafana relays SMTP through **vps2's** Postfix (prod's mail service).
|
||||||
travel through the box most likely to be on fire, and vanish exactly when they
|
Email alerts therefore travel through the box most likely to be on fire, and
|
||||||
matter.
|
vanish exactly when they matter.
|
||||||
- Discord is off-estate, works when prod is dead, and reaches a phone.
|
- Discord is off-estate, works when prod is dead, and reaches a phone.
|
||||||
|
|
||||||
`#ops-alerts` is dedicated. `#dev` / `#uat` / `#prod` are the *product's* alert
|
`#ops-alerts` is dedicated. `#dev` / `#uat` / `#prod` are the *product's* alert
|
||||||
subscription output — real deliveries to real subscribers — so routing ops noise
|
subscription output — real deliveries to real subscribers — so routing ops noise
|
||||||
there would corrupt the evidence.
|
there would corrupt the evidence.
|
||||||
|
|
||||||
The webhook URL is a secret and lives only in beta's gitignored `.env` as
|
The webhook URL is a secret and lives only in vps1's gitignored `.env` as
|
||||||
`DISCORD_ALERT_WEBHOOK_URL`. Grafana refuses to start if it's unset. A literal
|
`DISCORD_ALERT_WEBHOOK_URL`. Grafana refuses to start if it's unset. A literal
|
||||||
webhook URL committed to the repo is a **hard CI failure**.
|
webhook URL committed to the repo is a **hard CI failure**.
|
||||||
|
|
||||||
|
|
@ -135,7 +147,7 @@ pages nobody. **The only proof is a message actually arriving in
|
||||||
## Deploying an observability change
|
## Deploying an observability change
|
||||||
|
|
||||||
> ⚠️ **Merging is not deploying — and this one has already bitten.**
|
> ⚠️ **Merging is not deploying — and this one has already bitten.**
|
||||||
> `/opt/observability` on beta and prod is a checkout of the **archived**
|
> `/opt/observability` on vps1 and vps2 is a checkout of the **archived**
|
||||||
> `emi/thermograph-observability` repo. It can never `git pull` again; the
|
> `emi/thermograph-observability` repo. It can never `git pull` again; the
|
||||||
> content now lives here under `observability/`. A previous observability PR
|
> content now lives here under `observability/`. A previous observability PR
|
||||||
> merged and had **zero live effect** for exactly this reason.
|
> merged and had **zero live effect** for exactly this reason.
|
||||||
|
|
@ -146,13 +158,14 @@ hand: back up the live file with a UTC-timestamped suffix, `scp` the new one up,
|
||||||
|
|
||||||
Two gotchas in that sequence:
|
Two gotchas in that sequence:
|
||||||
|
|
||||||
- Use `sudo docker restart <container>` on **prod** — not `docker compose`,
|
- Use `sudo docker restart <container>` on **vps2** — not `docker compose`,
|
||||||
which there demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to
|
which there demands a `GF_SECURITY_ADMIN_PASSWORD` it has no `.env` to
|
||||||
interpolate from. Beta's stack *does* have an `.env`, so compose works there.
|
interpolate from. vps1's stack *does* have an `.env`, so compose works there.
|
||||||
- **`docker restart` will not pick up a new environment variable.** Any change
|
- **`docker restart` will not pick up a new environment variable.** Any change
|
||||||
that introduces one needs `docker compose up -d grafana` on beta.
|
that introduces one needs `docker compose up -d grafana` on vps1.
|
||||||
|
|
||||||
Alerting lives on **beta only** — prod and dev run Alloy agents, not Grafana.
|
Alerting lives on **vps1 only** — vps2 (prod and beta alike) and dev run Alloy
|
||||||
|
agents, not Grafana.
|
||||||
|
|
||||||
Full step-by-step, including rollback, is in
|
Full step-by-step, including rollback, is in
|
||||||
[`observability/README.md`](../../observability/README.md).
|
[`observability/README.md`](../../observability/README.md).
|
||||||
|
|
@ -177,9 +190,9 @@ Full step-by-step, including rollback, is in
|
||||||
- **Grafana cannot alert on its own death.** `AlertingWatchdog` is a *manual*
|
- **Grafana cannot alert on its own death.** `AlertingWatchdog` is a *manual*
|
||||||
dead-man's switch — it relies on somebody noticing the daily message stopped.
|
dead-man's switch — it relies on somebody noticing the daily message stopped.
|
||||||
An external uptime pinger from off-estate would close this properly.
|
An external uptime pinger from off-estate would close this properly.
|
||||||
- **Beta runs a `docker-compose.override.yml` that is not in this repo**, wiring
|
- **vps1 runs a `docker-compose.override.yml` that is not in this repo**, wiring
|
||||||
Grafana's SMTP to prod's Postfix. It should be mirrored into the tracked
|
Grafana's SMTP to vps2's Postfix (prod's). It should be mirrored into the
|
||||||
compose file or deleted; alerting no longer depends on it.
|
tracked compose file or deleted; alerting no longer depends on it.
|
||||||
|
|
||||||
## When something breaks
|
## When something breaks
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -79,14 +79,17 @@ git commit && push && open a PR
|
||||||
```
|
```
|
||||||
|
|
||||||
Then it reaches the hosts by either route: an `infra/**` push to `main` fires
|
Then it reaches the hosts by either route: an `infra/**` push to `main` fires
|
||||||
`infra-sync.yml` (re-renders `/etc/thermograph.env` on beta and prod, rolls
|
`infra-sync.yml` (re-renders prod's and beta's env files — `/etc/thermograph.env`
|
||||||
nothing), or the next app deploy renders it as step 2 of `deploy.sh`.
|
and `/etc/thermograph-beta.env`, both on vps2 — rolls nothing), or the next app
|
||||||
|
deploy renders it as step 2 of `deploy.sh`. Dev's vault (`dev.yaml`) is never
|
||||||
|
part of that sync — it's rendered only by dev's own deploy on vps1, and it
|
||||||
|
never layers `common.yaml` regardless.
|
||||||
|
|
||||||
Decide **which file** first:
|
Decide **which file** first:
|
||||||
|
|
||||||
- identical on prod *and* beta, and not a "lets one environment act as another"
|
- identical on prod *and* beta, and not a "lets one environment act as another"
|
||||||
credential → `common.yaml`;
|
credential → `common.yaml` (never reaches dev, on vps1, at all);
|
||||||
- per-host, or one of `POSTGRES_PASSWORD` / `THERMOGRAPH_AUTH_SECRET` /
|
- per-environment, or one of `POSTGRES_PASSWORD` / `THERMOGRAPH_AUTH_SECRET` /
|
||||||
`THERMOGRAPH_DATABASE_URL` → `prod.yaml` / `beta.yaml`;
|
`THERMOGRAPH_DATABASE_URL` → `prod.yaml` / `beta.yaml`;
|
||||||
- dev → `dev.yaml`, and give it **its own** value. Never a copy of prod's.
|
- dev → `dev.yaml`, and give it **its own** value. Never a copy of prod's.
|
||||||
|
|
||||||
|
|
@ -209,9 +212,11 @@ SERVICE=backend BACKEND_IMAGE_TAG=sha-<previous12hex> \
|
||||||
```
|
```
|
||||||
|
|
||||||
**Check the tag actually exists first.** Both deploy scripts *end* by deleting
|
**Check the tag actually exists first.** Both deploy scripts *end* by deleting
|
||||||
images outside the running pair, so beta typically holds no local rollback
|
images outside the running pair. Beta now runs Swarm exactly like prod
|
||||||
target at all. `rollback_to(dry_run: true)` checks host and registry and will
|
(co-resident on vps2), so both usually keep a handful of stopped-task
|
||||||
tell you when it's in neither.
|
references; dev, the one remaining compose environment (on vps1), is the one
|
||||||
|
that typically holds no local rollback target at all. `rollback_to(dry_run:
|
||||||
|
true)` checks host and registry and will tell you when it's in neither.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -48,19 +48,31 @@ walkthrough as superseded — but the drift goes further than that note admits.
|
||||||
The **key-management** sections (Part 1) and the `/etc/hosts` registry note are
|
The **key-management** sections (Part 1) and the `/etc/hosts` registry note are
|
||||||
still accurate and useful.
|
still accurate and useful.
|
||||||
|
|
||||||
### `infra/DEPLOY-DEV.md` and `infra/ACCESS.md` reference deleted workflows
|
### `infra/DEPLOY-DEV.md` and `infra/ACCESS.md` reference deleted workflows — and predate dev becoming a real environment
|
||||||
|
|
||||||
Both talk about `.forgejo/workflows/deploy-dev.yml`. It does not exist. The two
|
Both talk about `.forgejo/workflows/deploy-dev.yml`. It does not exist under
|
||||||
LAN-dev deploy workflows were deleted rather than ported at the CI
|
that name. The two original LAN-dev deploy workflows were deleted rather than
|
||||||
consolidation — they were documented as inert (they targeted a monorepo layout
|
ported at the CI consolidation — at the time they were genuinely inert (they
|
||||||
at `~/thermograph-dev` on a box still holding a split-era checkout). **LAN dev
|
targeted a monorepo layout at `~/thermograph-dev` on a box still holding a
|
||||||
is a local `make dev-up` concern, not a CI environment.**
|
split-era checkout), which is where the old claim **"LAN dev is a local `make
|
||||||
|
dev-up` concern, not a CI environment"** came from.
|
||||||
|
|
||||||
|
**That claim is now false, and worth flagging precisely because it used to be
|
||||||
|
true.** `dev` is a first-class hosted environment on vps1 — its own Postgres
|
||||||
|
container, mesh-only exposure, deployed by CI like beta and prod, as the third
|
||||||
|
leg of `deploy.yml` (see [07](07-ci-and-release.md) and
|
||||||
|
[08](08-infra-secrets.md)). The desktop's `make dev-up` is still a real,
|
||||||
|
useful thing — a laptop-local rehearsal — but it is no longer the *only* way
|
||||||
|
`dev` runs, and neither doc has been updated to say so.
|
||||||
|
|
||||||
`ACCESS.md` §3c also lists `{build,pr-build,deploy-dev,deploy}.yml` as the
|
`ACCESS.md` §3c also lists `{build,pr-build,deploy-dev,deploy}.yml` as the
|
||||||
workflow set; there are now nine files and the deploy/build ones were collapsed.
|
workflow set; there are now nine files and the deploy/build ones were collapsed.
|
||||||
|
It also still describes the host table from before the vps1/vps2 rename — see
|
||||||
|
`CUTOVER-NOTES.md`'s vps1/vps2 section for the current one.
|
||||||
|
|
||||||
Everything else in `ACCESS.md` — the host table, the agent-access model, the key
|
Everything else in `ACCESS.md` — the agent-access model, the key rotation
|
||||||
rotation procedure, the Swarm/Forgejo tracks — is current and worth reading.
|
procedure, the Swarm/Forgejo tracks — is current and worth reading; just
|
||||||
|
mentally translate host names.
|
||||||
|
|
||||||
### `backend/README.md` is written for the split-repo era
|
### `backend/README.md` is written for the split-repo era
|
||||||
|
|
||||||
|
|
@ -93,7 +105,7 @@ it — that's the standing instruction in the root `CLAUDE.md`.
|
||||||
|
|
||||||
### `observability/README.md`: merging is not deploying
|
### `observability/README.md`: merging is not deploying
|
||||||
|
|
||||||
`/opt/observability` on beta and prod is a checkout of the **archived**
|
`/opt/observability` on vps1 and vps2 is a checkout of the **archived**
|
||||||
`emi/thermograph-observability` repo. It can never `git pull` again. A previous
|
`emi/thermograph-observability` repo. It can never `git pull` again. A previous
|
||||||
observability PR merged and had **zero live effect** for exactly this reason.
|
observability PR merged and had **zero live effect** for exactly this reason.
|
||||||
The README documents the hand-ship procedure; until the checkouts are re-pointed
|
The README documents the hand-ship procedure; until the checkouts are re-pointed
|
||||||
|
|
@ -153,7 +165,7 @@ test run.
|
||||||
`infra/docker-compose.yml` pins `name: thermograph`. Without it, running compose
|
`infra/docker-compose.yml` pins `name: thermograph`. Without it, running compose
|
||||||
from `infra/` derives project `infra` — a silently **new** stack with empty
|
from `infra/` derives project `infra` — a silently **new** stack with empty
|
||||||
volumes beside the running one. `deploy-dev.sh` exports
|
volumes beside the running one. `deploy-dev.sh` exports
|
||||||
`COMPOSE_PROJECT_NAME=thermograph-dev` to keep LAN dev separate on purpose.
|
`COMPOSE_PROJECT_NAME=thermograph-dev` to keep dev separate on purpose.
|
||||||
Keep both halves.
|
Keep both halves.
|
||||||
|
|
||||||
### `deploy.sh` hard-resets the host checkout
|
### `deploy.sh` hard-resets the host checkout
|
||||||
|
|
@ -167,11 +179,13 @@ roll the sibling onto `local`.
|
||||||
### The deploy scripts eat your rollback image
|
### The deploy scripts eat your rollback image
|
||||||
|
|
||||||
Both `deploy.sh` and `deploy-stack.sh` **end** by deleting images outside the
|
Both `deploy.sh` and `deploy-stack.sh` **end** by deleting images outside the
|
||||||
running pair. On beta that succeeds, so beta typically holds **no local rollback
|
running pair. Beta now runs Swarm co-resident with prod on vps2, so both
|
||||||
target at all**; on prod it usually fails only because Swarm's stopped task
|
usually fail to fully prune only because Swarm's stopped task containers still
|
||||||
containers still hold references. `docker system prune -a` is forbidden on a
|
hold references — dev, the one remaining compose environment (on vps1), is the
|
||||||
live box for the same reason — but the deploy script is the thing that actually
|
one that typically holds **no local rollback target at all**. `docker system
|
||||||
eats the image, not prune. Verify a tag exists before promising a rollback.
|
prune -a` is forbidden on a live box for the same reason — but the deploy
|
||||||
|
script is the thing that actually eats the image, not prune. Verify a tag
|
||||||
|
exists before promising a rollback.
|
||||||
|
|
||||||
### `daemon` and `lake` are never deploy targets on their own
|
### `daemon` and `lake` are never deploy targets on their own
|
||||||
|
|
||||||
|
|
@ -189,13 +203,14 @@ line, in file order. A `| head -1` therefore grabbed `db`'s image, and
|
||||||
and dropped the daemon from *every* backend deploy. The script now builds the
|
and dropped the daemon from *every* backend deploy. The script now builds the
|
||||||
image reference from the same variables compose interpolates.
|
image reference from the same variables compose interpolates.
|
||||||
|
|
||||||
### Prod's Loki streams carry no `service` label
|
### Prod's (and beta's) Loki streams carry no `service` label
|
||||||
|
|
||||||
Prod runs Swarm; its `job="docker"` streams have no `service` label. A raw
|
Prod and beta both run Swarm, co-resident on vps2; their `job="docker"` streams
|
||||||
`{host="prod", service="backend"}` query matches nothing and reads as "no logs"
|
have no `service` label. A raw `{host="prod", service="backend"}` query matches
|
||||||
when it means "wrong selector". Use `logs_query`'s `service` argument. Prod
|
nothing and reads as "no logs" when it means "wrong selector". Use
|
||||||
container names also change on every redeploy (the task suffix) — resolve via
|
`logs_query`'s `service` argument. Prod and beta container names also change on
|
||||||
`docker ps --filter name=` at call time, never hardcode.
|
every redeploy (the task suffix) — resolve via `docker ps --filter name=` at
|
||||||
|
call time, never hardcode. Dev, on compose, doesn't have this problem.
|
||||||
|
|
||||||
### A Grafana alert rule can provision cleanly and never fire
|
### A Grafana alert rule can provision cleanly and never fire
|
||||||
|
|
||||||
|
|
@ -212,9 +227,9 @@ It's executable documentation until state is bootstrapped.
|
||||||
|
|
||||||
### The registry is mesh-only
|
### The registry is mesh-only
|
||||||
|
|
||||||
`git.thermograph.org` resolves publicly to beta's IP, but beta's Caddy rejects
|
`git.thermograph.org` resolves publicly to vps1's IP, but vps1's Caddy rejects
|
||||||
`/v2/*` from outside the WireGuard mesh. Any host that pulls needs
|
`/v2/*` from outside the WireGuard mesh. Any host that pulls needs
|
||||||
`10.10.0.2 git.thermograph.org` in `/etc/hosts`. Beta itself doesn't — it *is*
|
`10.10.0.2 git.thermograph.org` in `/etc/hosts`. vps1 itself doesn't — it *is*
|
||||||
the box.
|
the box.
|
||||||
|
|
||||||
### Forgejo runners only have the labels they registered with
|
### Forgejo runners only have the labels they registered with
|
||||||
|
|
|
||||||
|
|
@ -42,12 +42,13 @@ readings.
|
||||||
the superseded original — see [traps](11-traps.md).)
|
the superseded original — see [traps](11-traps.md).)
|
||||||
- **`infra/`** — compose and Swarm files, deploy scripts, the SOPS secrets
|
- **`infra/`** — compose and Swarm files, deploy scripts, the SOPS secrets
|
||||||
vault, Terraform, ops query tooling.
|
vault, Terraform, ops query tooling.
|
||||||
- **`observability/`** — Loki + Grafana on beta, an Alloy agent per node.
|
- **`observability/`** — Loki + Grafana on vps1, an Alloy agent per node.
|
||||||
|
|
||||||
Branches stage environments: PR → `dev` (LAN dev) → `main` (beta) → `release`
|
Branches stage environments: PR → `dev` (vps1, mesh-only) → `main` (beta, vps2)
|
||||||
(prod). The two app domains build and deploy **independently** — that
|
→ `release` (prod, vps2). The two app domains build and deploy
|
||||||
independence is the whole reason the split-then-reunify history exists, and
|
**independently** — that independence is the whole reason the split-then-reunify
|
||||||
[contracts](06-contracts.md) is the list of things that keep it safe.
|
history exists, and [contracts](06-contracts.md) is the list of things that
|
||||||
|
keep it safe.
|
||||||
|
|
||||||
## Your first day
|
## Your first day
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -175,7 +175,10 @@
|
||||||
<p><a href="{{.Base}}/climate" data-event="home.nav_cities">Browse all cities →</a></p>
|
<p><a href="{{.Base}}/climate" data-event="home.nav_cities">Browse all cities →</a></p>
|
||||||
</section>
|
</section>
|
||||||
{{template "base_body_end" .}} <script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
|
{{template "base_body_end" .}} <script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
|
||||||
<script type="module" src="{{.AssetBase}}/app.js"></script>{{/* The records strip renders real temperatures server-side as .temp spans, so
|
<script type="module" src="{{.AssetBase}}/app.js"></script>{{/* Bookmarked locations: a star toggle in the results share row + a "Saved"
|
||||||
|
dropdown by the Find button. Loaded as its own module alongside app.js
|
||||||
|
(not merged into it) — see bookmarks-ui.js's header comment for why. */}}
|
||||||
|
<script type="module" src="{{.AssetBase}}/bookmarks-ui.js"></script>{{/* The records strip renders real temperatures server-side as .temp spans, so
|
||||||
it needs the same converter the SEO pages use or they'd stay in °F while
|
it needs the same converter the SEO pages use or they'd stay in °F while
|
||||||
the toggle says °C. Both import units.js, which the module loader dedupes. */}}
|
the toggle says °C. Both import units.js, which the module loader dedupes. */}}
|
||||||
<script type="module" src="{{.AssetBase}}/climate.js"></script>
|
<script type="module" src="{{.AssetBase}}/climate.js"></script>
|
||||||
|
|
|
||||||
|
|
@ -10,8 +10,8 @@
|
||||||
// units.js is (imported by each page's entry module).
|
// units.js is (imported by each page's entry module).
|
||||||
|
|
||||||
let currentUser = null; // {id, email, display_name} or null
|
let currentUser = null; // {id, email, display_name} or null
|
||||||
let discordEnabled = false; // is Discord linking configured on the server?
|
let providers = []; // [{name, label, enabled}] from /oauth/config
|
||||||
let discordChecked = false; // have we asked yet? (once per page load)
|
let providersChecked = false; // have we asked yet? (once per page load)
|
||||||
const authCbs = []; // notified on login/logout so pages can re-gate
|
const authCbs = []; // notified on login/logout so pages can re-gate
|
||||||
|
|
||||||
// backend's own origin+base (e.g. "https://thermograph.org/thermograph", or
|
// backend's own origin+base (e.g. "https://thermograph.org/thermograph", or
|
||||||
|
|
@ -89,19 +89,27 @@ async function refreshUser() {
|
||||||
const res = await apiFetch(uv("users/me"));
|
const res = await apiFetch(uv("users/me"));
|
||||||
currentUser = res.ok ? await res.json() : null;
|
currentUser = res.ok ? await res.json() : null;
|
||||||
} catch (e) { currentUser = null; }
|
} catch (e) { currentUser = null; }
|
||||||
// Learn once whether Discord linking is configured, so the account menu only
|
// Learn once whether Discord is configured, so we only offer "Link Discord" and
|
||||||
// offers "Link Discord" when it will actually work (checked lazily, and only
|
// "Continue with Discord" when they'll actually work. Asked regardless of sign-in
|
||||||
// for a signed-in user — the menu never shows it to anyone else).
|
// state: the sign-in modal needs the answer precisely when nobody is signed in.
|
||||||
if (currentUser && !discordChecked) {
|
if (!providersChecked) {
|
||||||
discordChecked = true;
|
providersChecked = true;
|
||||||
try {
|
try {
|
||||||
const r = await apiFetch(uv("discord/config"));
|
const r = await apiFetch(uv("oauth/config"));
|
||||||
if (r.ok) discordEnabled = (await r.json()).enabled === true;
|
if (r.ok) providers = ((await r.json()).providers || []).filter((p) => p.enabled);
|
||||||
} catch (e) { /* leave it hidden */ }
|
} catch (e) { /* leave them hidden */ }
|
||||||
}
|
}
|
||||||
return currentUser;
|
return currentUser;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Providers the server can complete a flow with. Empty until refreshUser() has
|
||||||
|
// run, which is why the sign-in modal re-renders its provider row on every open
|
||||||
|
// rather than only when it is first built.
|
||||||
|
function enabledProviders() { return providers; }
|
||||||
|
function isLinked(name) {
|
||||||
|
return !!(currentUser && (currentUser.oauth_providers || []).includes(name));
|
||||||
|
}
|
||||||
|
|
||||||
// --- auth calls --------------------------------------------------------------
|
// --- auth calls --------------------------------------------------------------
|
||||||
async function login(email, password) {
|
async function login(email, password) {
|
||||||
// fastapi-users login is an OAuth2 form: username=email, password.
|
// fastapi-users login is an OAuth2 form: username=email, password.
|
||||||
|
|
@ -126,6 +134,27 @@ export async function logout() {
|
||||||
// --- auth modal --------------------------------------------------------------
|
// --- auth modal --------------------------------------------------------------
|
||||||
let modal = null, mode = "login";
|
let modal = null, mode = "login";
|
||||||
|
|
||||||
|
// Discord's brand mark ("Clyde"), inline like the other icons here so it needs no
|
||||||
|
// extra request and inherits currentColor.
|
||||||
|
const DISCORD_IC = `<svg viewBox="0 0 24 18" fill="currentColor" aria-hidden="true"><path d="M20.32 1.51A19.79 19.79 0 0 0 15.43 0a13.9 13.9 0 0 0-.63 1.28 18.4 18.4 0 0 0-5.6 0A13.9 13.9 0 0 0 8.57 0 19.74 19.74 0 0 0 3.68 1.51C.57 6.15-.28 10.68.14 15.14a19.9 19.9 0 0 0 6.05 3.05c.49-.66.92-1.37 1.3-2.11a12.9 12.9 0 0 1-2.05-.98c.17-.13.34-.26.5-.4a14.2 14.2 0 0 0 12.12 0c.16.14.33.27.5.4-.65.38-1.34.71-2.05.98.37.74.81 1.45 1.3 2.11a19.87 19.87 0 0 0 6.05-3.05c.49-5.17-.84-9.67-3.54-13.63ZM8.02 12.4c-1.18 0-2.15-1.08-2.15-2.41S6.82 7.58 8.02 7.58s2.17 1.09 2.15 2.41c0 1.33-.95 2.41-2.15 2.41Zm7.96 0c-1.18 0-2.15-1.08-2.15-2.41s.95-2.41 2.15-2.41 2.17 1.09 2.15 2.41c0 1.33-.95 2.41-2.15 2.41Z"/></svg>`;
|
||||||
|
|
||||||
|
// Google's mark is four fixed brand colours, so unlike the others it does not take
|
||||||
|
// currentColor — hence the white button beneath it, which is also what Google's
|
||||||
|
// branding terms require.
|
||||||
|
const GOOGLE_IC = `<svg viewBox="0 0 18 18" aria-hidden="true"><path fill="#4285F4" d="M17.64 9.2c0-.64-.06-1.25-.16-1.84H9v3.48h4.84a4.14 4.14 0 0 1-1.8 2.72v2.26h2.92c1.7-1.57 2.68-3.88 2.68-6.62Z"/><path fill="#34A853" d="M9 18c2.43 0 4.47-.8 5.96-2.18l-2.92-2.26c-.8.54-1.83.86-3.04.86-2.34 0-4.32-1.58-5.03-3.7H.96v2.33A9 9 0 0 0 9 18Z"/><path fill="#FBBC05" d="M3.97 10.72a5.4 5.4 0 0 1 0-3.44V4.95H.96a9 9 0 0 0 0 8.1l3-2.33Z"/><path fill="#EA4335" d="M9 3.58c1.32 0 2.5.45 3.44 1.35l2.58-2.58C13.46.89 11.43 0 9 0A9 9 0 0 0 .96 4.95l3.01 2.33C4.68 5.16 6.66 3.58 9 3.58Z"/></svg>`;
|
||||||
|
|
||||||
|
const PROVIDER_IC = { discord: DISCORD_IC, google: GOOGLE_IC };
|
||||||
|
|
||||||
|
// The provider buttons under the sign-in form. Rebuilt on every open because
|
||||||
|
// `providers` is populated asynchronously and may still have been empty when the
|
||||||
|
// modal was first constructed.
|
||||||
|
function providerButtonsHtml() {
|
||||||
|
return enabledProviders().map((p) => `
|
||||||
|
<a class="acct-oauth-login is-${p.name}" href="${uv(`oauth/${p.name}/login/start`)}">
|
||||||
|
${PROVIDER_IC[p.name] || ""}<span>Continue with ${escapeHtml(p.label)}</span>
|
||||||
|
</a>`).join("");
|
||||||
|
}
|
||||||
|
|
||||||
function buildModal() {
|
function buildModal() {
|
||||||
modal = document.createElement("div");
|
modal = document.createElement("div");
|
||||||
modal.className = "mp-overlay acct-overlay";
|
modal.className = "mp-overlay acct-overlay";
|
||||||
|
|
@ -145,6 +174,10 @@ function buildModal() {
|
||||||
</label>
|
</label>
|
||||||
<p class="acct-error" role="alert" hidden></p>
|
<p class="acct-error" role="alert" hidden></p>
|
||||||
<button type="submit" class="acct-submit">Sign in</button>
|
<button type="submit" class="acct-submit">Sign in</button>
|
||||||
|
<div class="acct-alt" hidden>
|
||||||
|
<span class="acct-or">or</span>
|
||||||
|
<div class="acct-oauth-row"></div>
|
||||||
|
</div>
|
||||||
<p class="acct-switch"></p>
|
<p class="acct-switch"></p>
|
||||||
</form>
|
</form>
|
||||||
</div>`;
|
</div>`;
|
||||||
|
|
@ -173,6 +206,11 @@ function setMode(m) {
|
||||||
modal.querySelector(".acct-switch").innerHTML = isLogin
|
modal.querySelector(".acct-switch").innerHTML = isLogin
|
||||||
? 'Need an account? <button type="button">Create one</button>'
|
? 'Need an account? <button type="button">Create one</button>'
|
||||||
: 'Already have an account? <button type="button">Sign in</button>';
|
: 'Already have an account? <button type="button">Sign in</button>';
|
||||||
|
// One label for both modes: the provider signs you in or creates the account,
|
||||||
|
// whichever applies, so making the user pick first would be a false choice.
|
||||||
|
const enabled = enabledProviders();
|
||||||
|
modal.querySelector(".acct-oauth-row").innerHTML = providerButtonsHtml();
|
||||||
|
modal.querySelector(".acct-alt").hidden = enabled.length === 0;
|
||||||
showError("");
|
showError("");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -328,6 +366,30 @@ const USER_IC = `<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stro
|
||||||
|
|
||||||
let acctEl = null;
|
let acctEl = null;
|
||||||
|
|
||||||
|
// The connected-accounts section of the account popover: one entry per provider
|
||||||
|
// the server offers, plus Discord's DM toggle, which is a delivery setting rather
|
||||||
|
// than an auth one and so only appears once Discord is actually linked.
|
||||||
|
function connectedAccountsHtml() {
|
||||||
|
const linkedCount = (currentUser.oauth_providers || []).length;
|
||||||
|
return enabledProviders().map((p) => {
|
||||||
|
if (!isLinked(p.name)) {
|
||||||
|
return `<a href="${uv(`oauth/${p.name}/link/start`)}" class="acct-pop-link">`
|
||||||
|
+ `Link ${escapeHtml(p.label)}</a>`;
|
||||||
|
}
|
||||||
|
const dm = p.name === "discord"
|
||||||
|
? `<button type="button" class="acct-pop-link acct-discord-dm">`
|
||||||
|
+ `${currentUser.discord_dm ? "Discord alerts: on" : "Discord alerts: off"}</button>`
|
||||||
|
: "";
|
||||||
|
// With no password and nothing else linked, this is the only way back in and
|
||||||
|
// the server refuses to unlink it — so don't offer a button that can't work.
|
||||||
|
const stuck = currentUser.oauth_only && linkedCount <= 1;
|
||||||
|
return dm + (stuck
|
||||||
|
? `<span class="acct-pop-note muted">Signed in with ${escapeHtml(p.label)}</span>`
|
||||||
|
: `<button type="button" class="acct-pop-link acct-oauth-unlink" `
|
||||||
|
+ `data-provider="${p.name}">Unlink ${escapeHtml(p.label)}</button>`);
|
||||||
|
}).join("");
|
||||||
|
}
|
||||||
|
|
||||||
function ensureAcctEl() {
|
function ensureAcctEl() {
|
||||||
const brand = document.querySelector(".brand");
|
const brand = document.querySelector(".brand");
|
||||||
if (!brand) return null;
|
if (!brand) return null;
|
||||||
|
|
@ -369,10 +431,7 @@ function renderHeader() {
|
||||||
</button>
|
</button>
|
||||||
<div class="acct-pop" hidden>
|
<div class="acct-pop" hidden>
|
||||||
<a href="${APP_BASE}/alerts" class="acct-pop-link">My alerts</a>
|
<a href="${APP_BASE}/alerts" class="acct-pop-link">My alerts</a>
|
||||||
${currentUser.discord_id
|
${connectedAccountsHtml()}
|
||||||
? `<button type="button" class="acct-pop-link acct-discord-dm">${currentUser.discord_dm ? "Discord alerts: on" : "Discord alerts: off"}</button>`
|
|
||||||
+ '<button type="button" class="acct-pop-link acct-discord-unlink">Unlink Discord</button>'
|
|
||||||
: (discordEnabled ? `<a href="${uv("discord/link/start")}" class="acct-pop-link">Link Discord</a>` : "")}
|
|
||||||
<button type="button" class="acct-pop-link acct-signout">Sign out</button>
|
<button type="button" class="acct-pop-link acct-signout">Sign out</button>
|
||||||
</div>
|
</div>
|
||||||
</div>`;
|
</div>`;
|
||||||
|
|
@ -400,12 +459,25 @@ function renderHeader() {
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
el.querySelector(".acct-signout").addEventListener("click", logout);
|
el.querySelector(".acct-signout").addEventListener("click", logout);
|
||||||
const unlinkBtn = el.querySelector(".acct-discord-unlink");
|
el.querySelectorAll(".acct-oauth-unlink").forEach((unlinkBtn) => {
|
||||||
if (unlinkBtn) unlinkBtn.addEventListener("click", async () => {
|
unlinkBtn.addEventListener("click", async () => {
|
||||||
unlinkBtn.disabled = true;
|
const provider = unlinkBtn.dataset.provider;
|
||||||
try { await apiFetch(uv("discord/unlink"), { method: "POST" }); } catch (e) {}
|
unlinkBtn.disabled = true;
|
||||||
await refreshUser();
|
try {
|
||||||
emitAuth(); // repaint the popover in its unlinked state
|
const res = await apiFetch(uv(`oauth/${provider}/unlink`), { method: "POST" });
|
||||||
|
// 409: the account has no password and this was its last provider, so
|
||||||
|
// unlinking would lock it out. Say why instead of appearing to do nothing.
|
||||||
|
if (res.status === 409) {
|
||||||
|
const body = await res.json().catch(() => null);
|
||||||
|
showToast((body && body.detail)
|
||||||
|
|| "That account can't be unlinked — it's the only way to sign in.", true);
|
||||||
|
unlinkBtn.disabled = false;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
} catch (e) {}
|
||||||
|
await refreshUser();
|
||||||
|
emitAuth(); // repaint the popover in its unlinked state
|
||||||
|
});
|
||||||
});
|
});
|
||||||
const dmBtn = el.querySelector(".acct-discord-dm");
|
const dmBtn = el.querySelector(".acct-discord-dm");
|
||||||
if (dmBtn) dmBtn.addEventListener("click", async () => {
|
if (dmBtn) dmBtn.addEventListener("click", async () => {
|
||||||
|
|
@ -452,6 +524,43 @@ function showToast(msg, isError = false) {
|
||||||
toastTimer = setTimeout(() => { el.hidden = true; }, 6000);
|
toastTimer = setTimeout(() => { el.hidden = true; }, 6000);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// --- OAuth outcomes ----------------------------------------------------------
|
||||||
|
// The link/login callback can only talk back to us through a redirect, so it
|
||||||
|
// lands on ?oauth=<status>&provider=<name>. Each status maps to a message builder
|
||||||
|
// taking the provider's display name; anything unrecognised is treated as a
|
||||||
|
// generic failure rather than shown raw.
|
||||||
|
const OAUTH_STATUS = {
|
||||||
|
linked: [(p) => `${p} connected.`, false],
|
||||||
|
signedin: [(p) => `Signed in with ${p}.`, false],
|
||||||
|
created: [(p) => `Account created with ${p} — welcome to Thermograph.`, false],
|
||||||
|
cancelled: [(p) => `${p} sign-in cancelled.`, false],
|
||||||
|
already: [(p) => `A different ${p} account is already connected here.`, true],
|
||||||
|
taken: [(p) => `That ${p} account is already connected to another Thermograph account.`, true],
|
||||||
|
mismatch: [(p) => `That email belongs to an account connected to a different ${p} `
|
||||||
|
+ "account. Sign in with your password to change it.", true],
|
||||||
|
noemail: [(p) => `${p} didn't share an email address, so there's no account to sign in to.`, true],
|
||||||
|
unverified: [(p) => `Verify your email address with ${p} first, then try again.`, true],
|
||||||
|
inactive: [() => "That account is deactivated.", true],
|
||||||
|
unavailable: [(p) => `${p} sign-in isn't available on this server.`, true],
|
||||||
|
};
|
||||||
|
|
||||||
|
function providerLabel(name) {
|
||||||
|
const known = providers.find((p) => p.name === name);
|
||||||
|
if (known) return known.label;
|
||||||
|
// The config probe may have failed, or the provider may have been switched off
|
||||||
|
// between starting the flow and coming back. Title-case the raw name rather
|
||||||
|
// than showing "undefined" in the toast.
|
||||||
|
return name ? name.charAt(0).toUpperCase() + name.slice(1) : "That provider";
|
||||||
|
}
|
||||||
|
|
||||||
|
function showOAuthStatus(status, provider) {
|
||||||
|
const label = providerLabel(provider);
|
||||||
|
const entry = OAUTH_STATUS[status];
|
||||||
|
if (!entry) return showToast(`Something went wrong with ${label}. Please try again.`, true);
|
||||||
|
const [build, isError] = entry;
|
||||||
|
showToast(build(label), isError);
|
||||||
|
}
|
||||||
|
|
||||||
// --- boot --------------------------------------------------------------------
|
// --- boot --------------------------------------------------------------------
|
||||||
(async function initAccount() {
|
(async function initAccount() {
|
||||||
ensureAcctEl();
|
ensureAcctEl();
|
||||||
|
|
@ -462,10 +571,21 @@ function showToast(msg, isError = false) {
|
||||||
|
|
||||||
const params = new URLSearchParams(location.search);
|
const params = new URLSearchParams(location.search);
|
||||||
const token = params.get("verify_token");
|
const token = params.get("verify_token");
|
||||||
if (token) {
|
// `discord` is the pre-Google spelling: a backend older than this frontend sends
|
||||||
params.delete("verify_token");
|
// only that one, so fall back to it and assume the provider it implies.
|
||||||
|
const oauthStatus = params.get("oauth") || params.get("discord");
|
||||||
|
const oauthProvider = params.get("provider") || (params.get("discord") ? "discord" : "");
|
||||||
|
// These are one-shot handoffs from a redirect; strip them together so a reload
|
||||||
|
// can't replay any of them, and so they never fight over the URL.
|
||||||
|
if (token || oauthStatus) {
|
||||||
|
["verify_token", "oauth", "provider", "discord"].forEach((k) => params.delete(k));
|
||||||
const clean = location.pathname + (params.toString() ? `?${params}` : "") + location.hash;
|
const clean = location.pathname + (params.toString() ? `?${params}` : "") + location.hash;
|
||||||
history.replaceState(null, "", clean);
|
history.replaceState(null, "", clean);
|
||||||
|
}
|
||||||
|
// refreshUser() above already ran, and the callback's redirect carried the new
|
||||||
|
// session cookie, so the header is painted signed-in before this toast lands.
|
||||||
|
if (oauthStatus) showOAuthStatus(oauthStatus, oauthProvider);
|
||||||
|
if (token) {
|
||||||
try {
|
try {
|
||||||
await verifyEmail(token);
|
await verifyEmail(token);
|
||||||
await refreshUser();
|
await refreshUser();
|
||||||
|
|
|
||||||
|
|
@ -152,10 +152,27 @@ async function runGrade() {
|
||||||
updateHash();
|
updateHash();
|
||||||
placeholder.hidden = true;
|
placeholder.hidden = true;
|
||||||
results.hidden = false;
|
results.hidden = false;
|
||||||
// Omit the date when it's today, so the URL (and cache key) matches the prefetch
|
// Always send the target date, even when it's "today" — the backend only
|
||||||
// fired from the other views — a same-tab navigation then reuses the response.
|
// falls back to its own UTC today (api_grade's `date` param default; see
|
||||||
const q = `lat=${selected.lat}&lon=${selected.lon}`;
|
// backend/web/app.py) when the param is omitted or empty, a day behind local
|
||||||
const url = dateInput.value === todayISO() ? uv(`grade?${q}`) : uv(`grade?${q}&date=${dateInput.value}`);
|
// midnight for any viewer east of UTC (reported from Kaunas, UTC+3: asking
|
||||||
|
// for 7/26 rendered 7/25). The comment this replaces claimed omitting the
|
||||||
|
// date kept this URL matching the cross-view prefetch — but that match was
|
||||||
|
// the other half of the bug: cache.js's viewFetches() seeds the grade view's
|
||||||
|
// cache entry under this exact dateless URL too, tagged fresh for the
|
||||||
|
// viewer's local day no matter which day the server actually resolved, so a
|
||||||
|
// same-tab nav that had already prefetched this cell (e.g. via the Day page)
|
||||||
|
// could serve that stale, server-resolved slice straight out of IndexedDB
|
||||||
|
// with no live request ever going out. Sending the real date makes every
|
||||||
|
// load self-consistent with the viewer's own clock instead of aliasing onto
|
||||||
|
// whatever "today" the server or an earlier prefetch resolved.
|
||||||
|
// dateInput.value can also come back empty (a cleared native date field —
|
||||||
|
// day.js guards the same input for the same reason), so fall back to the
|
||||||
|
// viewer's local today rather than ever send `date=` empty, which hits that
|
||||||
|
// same server fallback.
|
||||||
|
const date = dateInput.value || todayISO();
|
||||||
|
const q = `lat=${selected.lat}&lon=${selected.lon}&date=${date}`;
|
||||||
|
const url = uv(`grade?${q}`);
|
||||||
await loadView({
|
await loadView({
|
||||||
url, ttl: TTL.grade, seq: ++gradeSeq, current: () => gradeSeq,
|
url, ttl: TTL.grade, seq: ++gradeSeq, current: () => gradeSeq,
|
||||||
spinner: () => { results.innerHTML = `<p class="spinner">Fetching decades of climate history…</p>`; },
|
spinner: () => { results.innerHTML = `<p class="spinner">Fetching decades of climate history…</p>`; },
|
||||||
|
|
|
||||||
256
frontend/static/bookmarks-ui.js
Normal file
256
frontend/static/bookmarks-ui.js
Normal file
|
|
@ -0,0 +1,256 @@
|
||||||
|
// Map-page bookmark UI: a star toggle in the results "share" row, and a
|
||||||
|
// compact "Saved" pill + dropdown near the Find button for jumping between
|
||||||
|
// saved spots without retyping. Pure DOM glue over bookmarks.js, which is the
|
||||||
|
// only place bookmark data actually lives — this module holds no state of its
|
||||||
|
// own beyond references to the elements it built.
|
||||||
|
//
|
||||||
|
// Loaded as an extra module alongside app.js rather than merged into it: app.js
|
||||||
|
// owns #results' markup (it rewrites results.innerHTML wholesale on every grade
|
||||||
|
// via its own render()), so a MutationObserver on #results is what lets this
|
||||||
|
// module re-attach the star after every re-render without needing a hook
|
||||||
|
// app.js doesn't expose. selectLocation() is module-private to app.js, so
|
||||||
|
// jumping to a saved spot goes through the same hash the app already treats as
|
||||||
|
// authoritative on load (see nav.js: "a hash always wins") via a reload —
|
||||||
|
// the same path opening a shared link takes, so the normal grade flow runs.
|
||||||
|
import * as bookmarks from "./bookmarks.js";
|
||||||
|
|
||||||
|
const STAR_OUTLINE = `<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><polygon points="12 2 15.09 8.26 22 9.27 17 14.14 18.18 21.02 12 17.77 5.82 21.02 7 14.14 2 9.27 8.91 8.26 12 2"/></svg>`;
|
||||||
|
const STAR_FILLED = `<svg viewBox="0 0 24 24" fill="currentColor" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><polygon points="12 2 15.09 8.26 22 9.27 17 14.14 18.18 21.02 12 17.77 5.82 21.02 7 14.14 2 9.27 8.91 8.26 12 2"/></svg>`;
|
||||||
|
const CHEV_IC = `<svg viewBox="0 0 24 24" width="14" height="14" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><polyline points="6 9 12 15 18 9"/></svg>`;
|
||||||
|
const CLOSE_X = `<svg viewBox="0 0 24 24" width="13" height="13" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>`;
|
||||||
|
const PENCIL_IC = `<svg viewBox="0 0 24 24" width="13" height="13" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M12 20h9"/><path d="M16.5 3.5a2.12 2.12 0 0 1 3 3L7 19l-4 1 1-4Z"/></svg>`;
|
||||||
|
|
||||||
|
function esc(s) {
|
||||||
|
return String(s == null ? "" : s).replace(/[&<>"']/g, (c) =>
|
||||||
|
({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c]));
|
||||||
|
}
|
||||||
|
|
||||||
|
function injectStyle() {
|
||||||
|
if (document.getElementById("tg-bookmarks-style")) return;
|
||||||
|
const style = document.createElement("style");
|
||||||
|
style.id = "tg-bookmarks-style";
|
||||||
|
style.textContent = `
|
||||||
|
.bkm-star[aria-pressed="true"] { color: var(--accent); border-color: var(--accent); }
|
||||||
|
.bkm-pill-wrap { position: relative; display: inline-flex; align-self: center; }
|
||||||
|
.bkm-pill {
|
||||||
|
border-radius: 10px; border: 1px solid var(--border); cursor: pointer;
|
||||||
|
background: var(--surface-2); color: var(--text); font-weight: 600; font-size: 13px;
|
||||||
|
padding: 9px 12px; min-height: 40px; display: inline-flex; align-items: center; gap: 6px;
|
||||||
|
}
|
||||||
|
.bkm-pill:hover { border-color: var(--accent); }
|
||||||
|
.bkm-pill[hidden] { display: none; }
|
||||||
|
.bkm-drop {
|
||||||
|
position: absolute; left: 0; top: calc(100% + 6px); z-index: 1100; min-width: 260px;
|
||||||
|
max-width: min(340px, 90vw); max-height: 60vh; overflow-y: auto;
|
||||||
|
display: flex; flex-direction: column; padding: 6px; margin: 0; list-style: none;
|
||||||
|
background: var(--surface); border: 1px solid var(--border); border-radius: 12px;
|
||||||
|
box-shadow: 0 16px 40px rgba(0, 0, 0, .45);
|
||||||
|
}
|
||||||
|
.bkm-drop[hidden] { display: none; }
|
||||||
|
.bkm-item { display: flex; align-items: center; gap: 2px; border-radius: 8px; }
|
||||||
|
.bkm-item:hover { background: var(--surface-2); }
|
||||||
|
.bkm-item-go {
|
||||||
|
flex: 1 1 auto; min-width: 0; text-align: left; background: none; border: 0; cursor: pointer;
|
||||||
|
color: var(--text); font: inherit; font-size: 13.5px; padding: 9px 8px;
|
||||||
|
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
|
||||||
|
}
|
||||||
|
.bkm-item-btn {
|
||||||
|
flex-shrink: 0; background: none; border: 0; color: var(--muted); cursor: pointer;
|
||||||
|
padding: 7px; border-radius: 6px; display: inline-flex;
|
||||||
|
}
|
||||||
|
.bkm-item-btn:hover { color: var(--text); background: var(--border); }
|
||||||
|
.bkm-empty { padding: 10px 8px; font-size: 12.5px; color: var(--muted); }
|
||||||
|
.bkm-import-banner {
|
||||||
|
display: flex; align-items: center; justify-content: space-between; gap: 10px; flex-wrap: wrap;
|
||||||
|
background: var(--surface-2); border: 1px solid var(--border); border-left: 3px solid var(--accent);
|
||||||
|
border-radius: 12px; padding: 10px 14px; margin: 0 0 16px; font-size: 13.5px;
|
||||||
|
}
|
||||||
|
.bkm-import-banner .bkm-import-actions { display: flex; gap: 8px; flex-shrink: 0; }
|
||||||
|
.bkm-import-banner button {
|
||||||
|
border-radius: 8px; border: 1px solid var(--border); cursor: pointer;
|
||||||
|
padding: 7px 12px; font-size: 12.5px; font-weight: 600; background: var(--surface); color: var(--text);
|
||||||
|
}
|
||||||
|
.bkm-import-banner button.bkm-import-yes { background: var(--accent); color: #1a1206; border-color: var(--accent); }
|
||||||
|
@media (max-width: 480px) { .bkm-drop { left: auto; right: 0; } }
|
||||||
|
`;
|
||||||
|
document.head.appendChild(style);
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseHash() {
|
||||||
|
const p = new URLSearchParams(location.hash.slice(1));
|
||||||
|
const lat = parseFloat(p.get("lat")), lon = parseFloat(p.get("lon"));
|
||||||
|
if (isNaN(lat) || isNaN(lon)) return null;
|
||||||
|
return { lat, lon };
|
||||||
|
}
|
||||||
|
|
||||||
|
function jumpTo(lat, lon) {
|
||||||
|
const dateInput = document.getElementById("date-input");
|
||||||
|
const date = dateInput && dateInput.value ? dateInput.value : "";
|
||||||
|
let h = `#lat=${lat.toFixed(5)}&lon=${lon.toFixed(5)}`;
|
||||||
|
if (date) h += `&date=${date}`;
|
||||||
|
location.hash = h;
|
||||||
|
location.reload();
|
||||||
|
}
|
||||||
|
|
||||||
|
function currentLabel() {
|
||||||
|
const h2 = document.querySelector("#results .loc-title h2");
|
||||||
|
return (h2 && h2.textContent.trim()) || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- star toggle in the .share row ----
|
||||||
|
function buildStar() {
|
||||||
|
const btn = document.createElement("button");
|
||||||
|
btn.type = "button";
|
||||||
|
btn.id = "btn-bookmark";
|
||||||
|
btn.className = "bkm-star";
|
||||||
|
btn.addEventListener("click", async () => {
|
||||||
|
const loc = parseHash();
|
||||||
|
if (!loc) return;
|
||||||
|
const existing = bookmarks.find(loc.lat, loc.lon);
|
||||||
|
btn.disabled = true;
|
||||||
|
try {
|
||||||
|
if (existing) await bookmarks.remove(existing.id);
|
||||||
|
else await bookmarks.add(loc.lat, loc.lon, currentLabel());
|
||||||
|
} catch (e) {
|
||||||
|
// A real server rejection (e.g. the per-user cap) — bookmarks.add() no
|
||||||
|
// longer fakes a local save for this, so say why nothing changed.
|
||||||
|
alert(e.message || "Couldn't save this location.");
|
||||||
|
} finally {
|
||||||
|
btn.disabled = false;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return btn;
|
||||||
|
}
|
||||||
|
|
||||||
|
function paintStar() {
|
||||||
|
const share = document.querySelector("#results .share");
|
||||||
|
if (!share) return;
|
||||||
|
let btn = share.querySelector("#btn-bookmark");
|
||||||
|
if (!btn) {
|
||||||
|
btn = buildStar();
|
||||||
|
share.insertBefore(btn, share.firstChild);
|
||||||
|
}
|
||||||
|
const loc = parseHash();
|
||||||
|
const saved = loc ? !!bookmarks.find(loc.lat, loc.lon) : false;
|
||||||
|
btn.setAttribute("aria-pressed", saved ? "true" : "false");
|
||||||
|
btn.title = saved ? "Remove this saved location" : "Save this location";
|
||||||
|
btn.setAttribute("aria-label", btn.title);
|
||||||
|
btn.innerHTML = (saved ? STAR_FILLED : STAR_OUTLINE) + `<span>${saved ? "Saved" : "Save"}</span>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const resultsEl = document.getElementById("results");
|
||||||
|
if (resultsEl) {
|
||||||
|
new MutationObserver(() => paintStar()).observe(resultsEl, { childList: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- "Saved" dropdown near the Find button ----
|
||||||
|
let dropEl = null, pillEl = null;
|
||||||
|
|
||||||
|
function buildSavedControl() {
|
||||||
|
const findBar = document.querySelector(".find-bar");
|
||||||
|
if (!findBar) return;
|
||||||
|
const wrap = document.createElement("div");
|
||||||
|
wrap.className = "bkm-pill-wrap";
|
||||||
|
wrap.innerHTML = `
|
||||||
|
<button type="button" class="bkm-pill" id="bkm-pill" hidden aria-haspopup="true" aria-expanded="false">
|
||||||
|
<span>Saved</span>${CHEV_IC}
|
||||||
|
</button>
|
||||||
|
<ul class="bkm-drop" id="bkm-drop" hidden aria-label="Saved locations"></ul>`;
|
||||||
|
findBar.appendChild(wrap);
|
||||||
|
pillEl = wrap.querySelector("#bkm-pill");
|
||||||
|
dropEl = wrap.querySelector("#bkm-drop");
|
||||||
|
|
||||||
|
pillEl.addEventListener("click", () => {
|
||||||
|
const open = dropEl.hidden;
|
||||||
|
dropEl.hidden = !open;
|
||||||
|
pillEl.setAttribute("aria-expanded", open ? "true" : "false");
|
||||||
|
});
|
||||||
|
document.addEventListener("click", (e) => {
|
||||||
|
if (!wrap.contains(e.target) && !dropEl.hidden) {
|
||||||
|
dropEl.hidden = true;
|
||||||
|
pillEl.setAttribute("aria-expanded", "false");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
document.addEventListener("keydown", (e) => {
|
||||||
|
if (e.key === "Escape" && !dropEl.hidden) {
|
||||||
|
dropEl.hidden = true;
|
||||||
|
pillEl.setAttribute("aria-expanded", "false");
|
||||||
|
pillEl.focus();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderDropdown(list) {
|
||||||
|
if (!pillEl || !dropEl) return;
|
||||||
|
pillEl.hidden = list.length === 0; // new visitors see nothing extra at all
|
||||||
|
if (!list.length) { dropEl.hidden = true; return; }
|
||||||
|
dropEl.innerHTML = list.map((b) => {
|
||||||
|
const label = b.label || `${b.lat.toFixed(3)}, ${b.lon.toFixed(3)}`;
|
||||||
|
return `
|
||||||
|
<li class="bkm-item">
|
||||||
|
<button type="button" class="bkm-item-go" data-id="${esc(b.id)}" title="${esc(label)}">${esc(label)}</button>
|
||||||
|
<button type="button" class="bkm-item-btn bkm-item-rename" data-id="${esc(b.id)}" title="Rename" aria-label="Rename ${esc(label)}">${PENCIL_IC}</button>
|
||||||
|
<button type="button" class="bkm-item-btn bkm-item-remove" data-id="${esc(b.id)}" title="Remove" aria-label="Remove ${esc(label)}">${CLOSE_X}</button>
|
||||||
|
</li>`;
|
||||||
|
}).join("");
|
||||||
|
dropEl.querySelectorAll(".bkm-item-go").forEach((el) => {
|
||||||
|
el.addEventListener("click", () => {
|
||||||
|
const b = list.find((x) => x.id === el.dataset.id);
|
||||||
|
if (b) { dropEl.hidden = true; jumpTo(b.lat, b.lon); }
|
||||||
|
});
|
||||||
|
});
|
||||||
|
dropEl.querySelectorAll(".bkm-item-rename").forEach((el) => {
|
||||||
|
el.addEventListener("click", (e) => {
|
||||||
|
e.stopPropagation();
|
||||||
|
const b = list.find((x) => x.id === el.dataset.id);
|
||||||
|
if (!b) return;
|
||||||
|
const next = window.prompt("Rename this saved location:", b.label || "");
|
||||||
|
if (next && next.trim()) bookmarks.rename(b.id, next.trim());
|
||||||
|
});
|
||||||
|
});
|
||||||
|
dropEl.querySelectorAll(".bkm-item-remove").forEach((el) => {
|
||||||
|
el.addEventListener("click", (e) => {
|
||||||
|
e.stopPropagation();
|
||||||
|
const b = list.find((x) => x.id === el.dataset.id);
|
||||||
|
if (b) bookmarks.remove(b.id);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- import prompt banner (non-blocking, shown once until dismissed) ----
|
||||||
|
let bannerEl = null;
|
||||||
|
function renderBanner() {
|
||||||
|
const n = bookmarks.pendingImportCount();
|
||||||
|
if (!n) { if (bannerEl) bannerEl.hidden = true; return; }
|
||||||
|
const controls = document.querySelector(".controls");
|
||||||
|
if (!bannerEl && controls) {
|
||||||
|
bannerEl = document.createElement("div");
|
||||||
|
bannerEl.className = "bkm-import-banner";
|
||||||
|
controls.insertAdjacentElement("afterend", bannerEl);
|
||||||
|
}
|
||||||
|
if (!bannerEl) return;
|
||||||
|
bannerEl.hidden = false;
|
||||||
|
bannerEl.innerHTML = `
|
||||||
|
<span>Import ${n} saved location${n === 1 ? "" : "s"} into your account?</span>
|
||||||
|
<span class="bkm-import-actions">
|
||||||
|
<button type="button" class="bkm-import-yes">Import</button>
|
||||||
|
<button type="button" class="bkm-import-no">Not now</button>
|
||||||
|
</span>`;
|
||||||
|
bannerEl.querySelector(".bkm-import-yes").onclick = async () => {
|
||||||
|
const btn = bannerEl.querySelector(".bkm-import-yes");
|
||||||
|
btn.disabled = true;
|
||||||
|
try { await bookmarks.importPending(); } catch (e) { btn.disabled = false; }
|
||||||
|
};
|
||||||
|
bannerEl.querySelector(".bkm-import-no").onclick = () => bookmarks.dismissImportPrompt();
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- boot ----
|
||||||
|
injectStyle();
|
||||||
|
buildSavedControl();
|
||||||
|
bookmarks.subscribe((list) => {
|
||||||
|
renderDropdown(list);
|
||||||
|
paintStar();
|
||||||
|
renderBanner();
|
||||||
|
});
|
||||||
|
renderDropdown(bookmarks.list());
|
||||||
|
renderBanner();
|
||||||
199
frontend/static/bookmarks.js
Normal file
199
frontend/static/bookmarks.js
Normal file
|
|
@ -0,0 +1,199 @@
|
||||||
|
// Bookmarked locations: the single source of truth for "places you've saved".
|
||||||
|
// Logged out, this is pure localStorage. Logged in, the server is authoritative
|
||||||
|
// and this module mirrors it into localStorage so the list still renders
|
||||||
|
// instantly (and offline) on the next load. Every consumer (the map page's
|
||||||
|
// star toggle + saved-locations dropdown, the alerts page's Saved locations
|
||||||
|
// section) reads through list()/subscribe() and never touches storage or the
|
||||||
|
// API directly.
|
||||||
|
import { apiFetch, onAuthChange, uv } from "./account.js";
|
||||||
|
|
||||||
|
const LS_KEY = "thermograph:bookmarks";
|
||||||
|
const DISMISS_KEY = "thermograph:bookmarks:import-dismissed";
|
||||||
|
const MAX_LOCAL = 200;
|
||||||
|
|
||||||
|
// A stable local key for a spot: rounded to ~4dp (~11m), so re-picking
|
||||||
|
// "the same place" a pixel off the first pin still matches.
|
||||||
|
export function cellId(lat, lon) {
|
||||||
|
return `${lat.toFixed(4)},${lon.toFixed(4)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function readLocal() {
|
||||||
|
try {
|
||||||
|
const arr = JSON.parse(localStorage.getItem(LS_KEY));
|
||||||
|
return Array.isArray(arr)
|
||||||
|
? arr.filter((b) => b && typeof b.lat === "number" && typeof b.lon === "number")
|
||||||
|
: [];
|
||||||
|
} catch (e) { return []; }
|
||||||
|
}
|
||||||
|
function writeLocal(rows) {
|
||||||
|
try { localStorage.setItem(LS_KEY, JSON.stringify(rows.slice(0, MAX_LOCAL))); } catch (e) { /* private mode, quota, etc. — degrade quietly */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
let cache = readLocal(); // what every consumer renders
|
||||||
|
let pending = []; // local-only bookmarks a signed-in user hasn't imported yet
|
||||||
|
let signedIn = false;
|
||||||
|
const subs = [];
|
||||||
|
|
||||||
|
function notify() { subs.forEach((cb) => { try { cb(cache.slice()); } catch (e) {} }); }
|
||||||
|
|
||||||
|
/** Subscribe to bookmark-list changes. Returns an unsubscribe function. */
|
||||||
|
export function subscribe(cb) {
|
||||||
|
subs.push(cb);
|
||||||
|
return () => { const i = subs.indexOf(cb); if (i > -1) subs.splice(i, 1); };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function list() { return cache.slice(); }
|
||||||
|
export function find(lat, lon) {
|
||||||
|
const id = cellId(lat, lon);
|
||||||
|
return cache.find((b) => cellId(b.lat, b.lon) === id) || null;
|
||||||
|
}
|
||||||
|
export function isBookmarked(lat, lon) { return !!find(lat, lon); }
|
||||||
|
export function pendingImportCount() { return pending.length; }
|
||||||
|
|
||||||
|
export function importDismissed() {
|
||||||
|
try { return localStorage.getItem(DISMISS_KEY) === "1"; } catch (e) { return false; }
|
||||||
|
}
|
||||||
|
export function dismissImportPrompt() {
|
||||||
|
try { localStorage.setItem(DISMISS_KEY, "1"); } catch (e) {}
|
||||||
|
pending = [];
|
||||||
|
notify();
|
||||||
|
}
|
||||||
|
|
||||||
|
function fromServerRow(b) {
|
||||||
|
return {
|
||||||
|
id: b.id != null ? String(b.id) : (b.cell_id || cellId(b.lat, b.lon)),
|
||||||
|
cell_id: b.cell_id || null,
|
||||||
|
lat: b.lat, lon: b.lon,
|
||||||
|
label: b.label || null,
|
||||||
|
created_at: b.created_at || Date.now() / 1000,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Returns the server's list, or null on 401 (signed out) / offline / error —
|
||||||
|
// callers treat null as "can't say, fall back to local" rather than "empty".
|
||||||
|
async function fetchServerList() {
|
||||||
|
try {
|
||||||
|
const res = await apiFetch(uv("bookmarks"));
|
||||||
|
if (res.status === 401) { signedIn = false; return null; }
|
||||||
|
if (!res.ok) return null;
|
||||||
|
signedIn = true;
|
||||||
|
const data = await res.json();
|
||||||
|
return Array.isArray(data.bookmarks) ? data.bookmarks : [];
|
||||||
|
} catch (e) { return null; }
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reconciles cache with the server, diffing against whatever was in
|
||||||
|
// localStorage right before this call so any local-only spots can be offered
|
||||||
|
// up for import instead of silently vanishing behind the server's list.
|
||||||
|
async function syncFromServer(preSyncLocal) {
|
||||||
|
const rows = await fetchServerList();
|
||||||
|
if (rows == null) return; // signed out or offline: leave cache as-is
|
||||||
|
const serverList = rows.map(fromServerRow);
|
||||||
|
const serverIds = new Set(serverList.map((b) => cellId(b.lat, b.lon)));
|
||||||
|
const localOnly = preSyncLocal.filter((b) => !serverIds.has(cellId(b.lat, b.lon)));
|
||||||
|
cache = serverList;
|
||||||
|
writeLocal(cache);
|
||||||
|
pending = importDismissed() ? [] : localOnly;
|
||||||
|
notify();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Save (or re-save with a new label) the given spot. Upserts either way.
|
||||||
|
*
|
||||||
|
* Throws if signed in and the server explicitly rejects the request (e.g. the
|
||||||
|
* 409 per-user cap, a 422 on bad input) — that is a real failure, not the
|
||||||
|
* "offline" case below, and must not be papered over with a local-only
|
||||||
|
* bookmark that doesn't actually exist server-side and would just vanish,
|
||||||
|
* unexplained, on the next sync. A genuine network failure (no response at
|
||||||
|
* all) still falls through to the local-only save so the star visibly
|
||||||
|
* sticks even offline. */
|
||||||
|
export async function add(lat, lon, label) {
|
||||||
|
const id = cellId(lat, lon);
|
||||||
|
if (signedIn) {
|
||||||
|
let res = null;
|
||||||
|
try {
|
||||||
|
res = await apiFetch(uv("bookmarks"), { method: "POST", json: { lat, lon, label: label || undefined } });
|
||||||
|
} catch (e) {
|
||||||
|
// offline / network failure: fall through to the local-only path below.
|
||||||
|
}
|
||||||
|
if (res) {
|
||||||
|
if (res.ok) {
|
||||||
|
const rows = await fetchServerList();
|
||||||
|
if (rows != null) { cache = rows.map(fromServerRow); writeLocal(cache); notify(); return find(lat, lon); }
|
||||||
|
} else {
|
||||||
|
let msg = `Couldn't save this location (${res.status}).`;
|
||||||
|
try { const d = await res.json(); if (typeof d.detail === "string") msg = d.detail; } catch (e) { /* no body */ }
|
||||||
|
throw new Error(msg);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let b = cache.find((x) => cellId(x.lat, x.lon) === id);
|
||||||
|
if (b) { if (label) b.label = label; }
|
||||||
|
else {
|
||||||
|
b = { id, lat, lon, label: label || null, created_at: Date.now() / 1000 };
|
||||||
|
cache = [b, ...cache].slice(0, MAX_LOCAL);
|
||||||
|
}
|
||||||
|
writeLocal(cache);
|
||||||
|
notify();
|
||||||
|
return b;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function remove(id) {
|
||||||
|
if (signedIn) {
|
||||||
|
try { await apiFetch(uv(`bookmarks/${encodeURIComponent(id)}`), { method: "DELETE" }); } catch (e) {}
|
||||||
|
}
|
||||||
|
cache = cache.filter((b) => b.id !== id);
|
||||||
|
writeLocal(cache);
|
||||||
|
notify();
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function rename(id, label) {
|
||||||
|
if (signedIn) {
|
||||||
|
try { await apiFetch(uv(`bookmarks/${encodeURIComponent(id)}`), { method: "PATCH", json: { label } }); }
|
||||||
|
catch (e) { /* still apply locally so the edit shows up */ }
|
||||||
|
}
|
||||||
|
const b = cache.find((x) => x.id === id);
|
||||||
|
if (b) { b.label = label; writeLocal(cache); notify(); }
|
||||||
|
}
|
||||||
|
|
||||||
|
async function runImport(items) {
|
||||||
|
if (!items.length) return { imported: 0, skipped: 0 };
|
||||||
|
const res = await apiFetch(uv("bookmarks/import"), { method: "POST", json: { items } });
|
||||||
|
if (!res.ok) throw new Error(`Import failed (${res.status}).`);
|
||||||
|
const data = await res.json();
|
||||||
|
if (Array.isArray(data.bookmarks)) { cache = data.bookmarks.map(fromServerRow); writeLocal(cache); }
|
||||||
|
pending = [];
|
||||||
|
notify();
|
||||||
|
return data;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Import the local-only bookmarks the login prompt flagged. */
|
||||||
|
export function importPending() {
|
||||||
|
return runImport(pending.map((b) => ({ lat: b.lat, lon: b.lon, label: b.label })));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Import everything currently in localStorage — the explicit "Import from
|
||||||
|
* this device" button on the alerts page, reachable even after the login
|
||||||
|
* prompt was dismissed. */
|
||||||
|
export function importAllLocal() {
|
||||||
|
return runImport(readLocal().map((b) => ({ lat: b.lat, lon: b.lon, label: b.label })));
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- boot + auth transitions -------------------------------------------------
|
||||||
|
(async function boot() {
|
||||||
|
const local = readLocal();
|
||||||
|
cache = local;
|
||||||
|
notify(); // render local state immediately, don't wait on the network
|
||||||
|
await syncFromServer(local);
|
||||||
|
})();
|
||||||
|
|
||||||
|
onAuthChange(async (user) => {
|
||||||
|
if (user) {
|
||||||
|
await syncFromServer(readLocal());
|
||||||
|
} else {
|
||||||
|
// Logged out: fall back to local bookmarks without wiping them.
|
||||||
|
signedIn = false;
|
||||||
|
pending = [];
|
||||||
|
cache = readLocal();
|
||||||
|
notify();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
@ -207,7 +207,7 @@ function seedCache(url, data, etag) {
|
||||||
// lives here next to the bundle fetch — not knowledge of any one page.
|
// lives here next to the bundle fetch — not knowledge of any one page.
|
||||||
function viewFetches(q, today) {
|
function viewFetches(q, today) {
|
||||||
return {
|
return {
|
||||||
grade: [uv(`grade?${q}`), TTL.grade],
|
grade: [uv(`grade?${q}&date=${today}`), TTL.grade],
|
||||||
forecast: [uv(`forecast?${q}`), TTL.forecast],
|
forecast: [uv(`forecast?${q}`), TTL.forecast],
|
||||||
calendar: [uv(`calendar?${q}&months=24`), TTL.calendar],
|
calendar: [uv(`calendar?${q}&months=24`), TTL.calendar],
|
||||||
day: [uv(`day?${q}&date=${today}`), TTL.day],
|
day: [uv(`day?${q}&date=${today}`), TTL.day],
|
||||||
|
|
@ -232,6 +232,12 @@ export function prefetchViews(lat, lon, ownViews) {
|
||||||
if (_prefetched === tag) return;
|
if (_prefetched === tag) return;
|
||||||
_prefetched = tag;
|
_prefetched = tag;
|
||||||
const q = `lat=${lat}&lon=${lon}`;
|
const q = `lat=${lat}&lon=${lon}`;
|
||||||
|
// The client's local "today" — sent explicitly so the server never has to guess
|
||||||
|
// it from its own (UTC) clock. Getting this wrong is exactly what poisoned the
|
||||||
|
// Weekly view's cache for UTC+ visitors between their local midnight and UTC
|
||||||
|
// midnight: the bundle used to be built against the server's date while the
|
||||||
|
// per-view request stated the client's, so the two silently disagreed.
|
||||||
|
const today = localDay();
|
||||||
const own = new Set(ownViews);
|
const own = new Set(ownViews);
|
||||||
// Yield first so the landing view finishes rendering before we fetch the rest.
|
// Yield first so the landing view finishes rendering before we fetch the rest.
|
||||||
setTimeout(async () => {
|
setTimeout(async () => {
|
||||||
|
|
@ -239,17 +245,20 @@ export function prefetchViews(lat, lon, ownViews) {
|
||||||
const ek = "tg-cell-etag:" + q;
|
const ek = "tg-cell-etag:" + q;
|
||||||
let prev = null;
|
let prev = null;
|
||||||
try { prev = sessionStorage.getItem(ek); } catch (e) {}
|
try { prev = sessionStorage.getItem(ek); } catch (e) {}
|
||||||
const res = await fetch(uv(`cell?${q}&neighbors=1`),
|
const res = await fetch(uv(`cell?${q}&date=${today}&neighbors=1`),
|
||||||
{ credentials: "include", ...(prev ? { headers: { "If-None-Match": prev } } : {}) });
|
{ credentials: "include", ...(prev ? { headers: { "If-None-Match": prev } } : {}) });
|
||||||
if (res.ok && res.status !== 304) {
|
if (res.ok && res.status !== 304) {
|
||||||
const bundle = await res.json();
|
const bundle = await res.json();
|
||||||
try { sessionStorage.setItem(ek, res.headers.get("ETag") || ""); } catch (e) {}
|
try { sessionStorage.setItem(ek, res.headers.get("ETag") || ""); } catch (e) {}
|
||||||
const fetches = viewFetches(q, localDay());
|
// Seed under bundle.today (the date the server actually resolved — normally
|
||||||
|
// just an echo of the date= we sent) rather than our own `today`, so the
|
||||||
|
// seeded key stays byte-identical to what a per-view request for that same
|
||||||
|
// resolved date will use even in the rare case the two disagree (e.g. a
|
||||||
|
// request that straddles local midnight).
|
||||||
|
const fetches = viewFetches(q, bundle.today);
|
||||||
for (const [name, slice] of Object.entries(bundle.slices || {})) {
|
for (const [name, slice] of Object.entries(bundle.slices || {})) {
|
||||||
if (own.has(name) || !slice || !slice.data) continue;
|
if (own.has(name) || !slice || !slice.data) continue;
|
||||||
const url = name === "day"
|
const url = (fetches[name] || [])[0];
|
||||||
? uv(`day?${q}&date=${bundle.today}`) // the day slice targets today
|
|
||||||
: (fetches[name] || [])[0];
|
|
||||||
if (url) seedCache(url, slice.data, slice.etag);
|
if (url) seedCache(url, slice.data, slice.etag);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -6,6 +6,9 @@
|
||||||
--text: #e7ecf2;
|
--text: #e7ecf2;
|
||||||
--muted: #9aa6b2;
|
--muted: #9aa6b2;
|
||||||
--accent: #f0803c;
|
--accent: #f0803c;
|
||||||
|
/* Discord's brand blurple. Fixed in both schemes on purpose: it identifies the
|
||||||
|
provider, so it is their colour rather than one of ours to re-theme. */
|
||||||
|
--discord: #5865f2;
|
||||||
|
|
||||||
/* temperature grades: 9-step diverging cold -> green -> hot (colorblind-safe).
|
/* temperature grades: 9-step diverging cold -> green -> hot (colorblind-safe).
|
||||||
rec-* are the "Near Record" danger tiers — deliberately dark/saturated. */
|
rec-* are the "Near Record" danger tiers — deliberately dark/saturated. */
|
||||||
|
|
@ -422,6 +425,39 @@ main { max-width: 1200px; margin: 0 auto; padding: 20px 24px 60px; }
|
||||||
color: var(--text); border: 1px solid var(--rec-hot, #d64545);
|
color: var(--text); border: 1px solid var(--rec-hot, #d64545);
|
||||||
}
|
}
|
||||||
.acct-error[hidden] { display: none; }
|
.acct-error[hidden] { display: none; }
|
||||||
|
/* "or / Continue with Discord" under the submit button. Hidden unless the server
|
||||||
|
reports Discord configured (account.js sets [hidden]). */
|
||||||
|
.acct-alt { display: flex; flex-direction: column; gap: 10px; }
|
||||||
|
.acct-alt[hidden] { display: none; }
|
||||||
|
.acct-or {
|
||||||
|
display: flex; align-items: center; gap: 10px;
|
||||||
|
font-size: 12px; color: var(--muted); text-transform: uppercase; letter-spacing: .06em;
|
||||||
|
}
|
||||||
|
.acct-or::before, .acct-or::after {
|
||||||
|
content: ""; flex: 1; height: 1px; background: var(--border);
|
||||||
|
}
|
||||||
|
.acct-oauth-row { display: flex; flex-direction: column; gap: 8px; }
|
||||||
|
.acct-oauth-login {
|
||||||
|
display: flex; align-items: center; justify-content: center; gap: 9px;
|
||||||
|
padding: 12px 16px; min-height: 44px; border-radius: 10px;
|
||||||
|
font-weight: 700; font-size: 15px; text-decoration: none;
|
||||||
|
border: 1px solid transparent;
|
||||||
|
}
|
||||||
|
.acct-oauth-login:hover { filter: brightness(1.08); }
|
||||||
|
.acct-oauth-login svg { width: 20px; height: 20px; flex: none; }
|
||||||
|
.acct-oauth-login.is-discord {
|
||||||
|
background: var(--discord); color: #fff;
|
||||||
|
}
|
||||||
|
.acct-oauth-login.is-discord svg { height: 15px; }
|
||||||
|
/* Google's branding terms require their mark on white (or their own grey), with a
|
||||||
|
visible border — so this one button is deliberately light in both schemes and
|
||||||
|
does not follow the surface tokens. */
|
||||||
|
.acct-oauth-login.is-google {
|
||||||
|
background: #fff; color: #1f1f1f; border-color: #747775;
|
||||||
|
}
|
||||||
|
.acct-oauth-login.is-google:hover { filter: none; background: #f2f2f2; }
|
||||||
|
|
||||||
|
.acct-pop-note { display: block; padding: 9px 12px; font-size: 13px; }
|
||||||
.acct-switch { margin: 0; font-size: 13px; color: var(--muted); text-align: center; }
|
.acct-switch { margin: 0; font-size: 13px; color: var(--muted); text-align: center; }
|
||||||
.acct-switch button {
|
.acct-switch button {
|
||||||
background: none; border: 0; color: var(--accent); cursor: pointer;
|
background: none; border: 0; color: var(--accent); cursor: pointer;
|
||||||
|
|
|
||||||
|
|
@ -2,10 +2,16 @@
|
||||||
// them add a city (via the shared map picker), tune the percentile threshold and
|
// them add a city (via the shared map picker), tune the percentile threshold and
|
||||||
// watched metrics, toggle active, and remove. Auth-gated: signed-out visitors get
|
// watched metrics, toggle active, and remove. Auth-gated: signed-out visitors get
|
||||||
// a sign-in prompt. All calls go through account.js's cookie-aware apiFetch.
|
// a sign-in prompt. All calls go through account.js's cookie-aware apiFetch.
|
||||||
|
//
|
||||||
|
// Also the closest thing to an account page, so it doubles as the Saved
|
||||||
|
// locations surface: bookmarks.js is the single source of truth for those (see
|
||||||
|
// its header comment), this just lists/renders/edits them the same way it does
|
||||||
|
// alert subscriptions, reusing .sub-list/.sub-card.
|
||||||
import "./nav.js";
|
import "./nav.js";
|
||||||
import { apiFetch, openAuth, onAuthChange, uv } from "./account.js";
|
import { apiFetch, openAuth, onAuthChange, uv } from "./account.js";
|
||||||
import { open as openPicker } from "./mappicker.js";
|
import { open as openPicker } from "./mappicker.js";
|
||||||
import * as pushClient from "./push-client.js";
|
import * as pushClient from "./push-client.js";
|
||||||
|
import * as bookmarks from "./bookmarks.js";
|
||||||
|
|
||||||
// Metric keys a user can watch, with friendly labels. (fmax/fmin exist server-side
|
// Metric keys a user can watch, with friendly labels. (fmax/fmin exist server-side
|
||||||
// but are omitted from the picker to keep it focused; labelled here if present.)
|
// but are omitted from the picker to keep it focused; labelled here if present.)
|
||||||
|
|
@ -24,6 +30,7 @@ METRIC_LABEL.fmin = "Feels-like low";
|
||||||
const DEFAULT_METRICS = ["tmax", "feels", "precip"];
|
const DEFAULT_METRICS = ["tmax", "feels", "precip"];
|
||||||
|
|
||||||
const TRASH_IC = `<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M3 6h18M8 6V4a2 2 0 0 1 2-2h4a2 2 0 0 1 2 2v2m2 0v14a2 2 0 0 1-2 2H7a2 2 0 0 1-2-2V6"/></svg>`;
|
const TRASH_IC = `<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M3 6h18M8 6V4a2 2 0 0 1 2-2h4a2 2 0 0 1 2 2v2m2 0v14a2 2 0 0 1-2 2H7a2 2 0 0 1-2-2V6"/></svg>`;
|
||||||
|
const PENCIL_IC = `<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M12 20h9"/><path d="M16.5 3.5a2.12 2.12 0 0 1 3 3L7 19l-4 1 1-4Z"/></svg>`;
|
||||||
|
|
||||||
const body = document.getElementById("alerts-body");
|
const body = document.getElementById("alerts-body");
|
||||||
let subs = [];
|
let subs = [];
|
||||||
|
|
@ -33,6 +40,25 @@ function esc(s) {
|
||||||
({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c]));
|
({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c]));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Small stylesheet for the bits of the Saved locations section that don't
|
||||||
|
// already have a home in .sub-list/.sub-card — injected once, not appended to
|
||||||
|
// the shared style.css, so this stays self-contained.
|
||||||
|
(function injectBkmStyle() {
|
||||||
|
if (document.getElementById("tg-bkm-alerts-style")) return;
|
||||||
|
const s = document.createElement("style");
|
||||||
|
s.id = "tg-bkm-alerts-style";
|
||||||
|
s.textContent = `
|
||||||
|
.bkm-heading { font-size: 15px; margin: 26px 0 4px; }
|
||||||
|
.bkm-import-row { display: flex; align-items: center; justify-content: space-between; gap: 10px; flex-wrap: wrap; margin: 0 0 12px; }
|
||||||
|
.bkm-import-status { margin: -4px 0 10px; }
|
||||||
|
.bkm-card-actions { display: flex; align-items: center; gap: 2px; flex-shrink: 0; }
|
||||||
|
.bkm-rename { background: none; border: 0; color: var(--muted); cursor: pointer; padding: 7px; border-radius: 6px; display: inline-flex; }
|
||||||
|
.bkm-rename:hover { color: var(--text); background: var(--border); }
|
||||||
|
.bkm-card-loc { color: var(--muted); font-size: 12.5px; }
|
||||||
|
`;
|
||||||
|
document.head.appendChild(s);
|
||||||
|
})();
|
||||||
|
|
||||||
async function readErr(res) {
|
async function readErr(res) {
|
||||||
try {
|
try {
|
||||||
const d = await res.json();
|
const d = await res.json();
|
||||||
|
|
@ -80,17 +106,84 @@ function render() {
|
||||||
<button type="button" class="find-btn" id="add-alert">+ Add a city</button>
|
<button type="button" class="find-btn" id="add-alert">+ Add a city</button>
|
||||||
<p class="alerts-note muted">Each alert sends at most one notification per week.</p>
|
<p class="alerts-note muted">Each alert sends at most one notification per week.</p>
|
||||||
</div>
|
</div>
|
||||||
<ul class="sub-list" id="sub-list"></ul>`;
|
<ul class="sub-list" id="sub-list"></ul>
|
||||||
|
<section class="bkm-section" id="bkm-alerts-section" aria-labelledby="bkm-heading"></section>`;
|
||||||
body.querySelector("#add-alert").onclick = startAdd;
|
body.querySelector("#add-alert").onclick = startAdd;
|
||||||
renderPushBar();
|
renderPushBar();
|
||||||
const list = body.querySelector("#sub-list");
|
const list = body.querySelector("#sub-list");
|
||||||
if (!subs.length) {
|
if (!subs.length) {
|
||||||
list.innerHTML = `<li class="sub-empty muted">No alerts yet. Add a city to get started.</li>`;
|
list.innerHTML = `<li class="sub-empty muted">No alerts yet. Add a city to get started.</li>`;
|
||||||
return;
|
} else {
|
||||||
|
subs.forEach((s) => list.appendChild(subCard(s)));
|
||||||
}
|
}
|
||||||
subs.forEach((s) => list.appendChild(subCard(s)));
|
renderBookmarksSection();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// --- Saved locations (bookmarks) ---------------------------------------------
|
||||||
|
function bkmCard(b) {
|
||||||
|
const li = document.createElement("li");
|
||||||
|
li.className = "sub-card";
|
||||||
|
const label = b.label || `${b.lat.toFixed(3)}, ${b.lon.toFixed(3)}`;
|
||||||
|
li.innerHTML = `
|
||||||
|
<div class="sub-top">
|
||||||
|
<div class="sub-title">
|
||||||
|
<span class="sub-name">${esc(label)}</span>
|
||||||
|
${b.label ? `<span class="bkm-card-loc">${esc(b.lat.toFixed(3))}, ${esc(b.lon.toFixed(3))}</span>` : ""}
|
||||||
|
</div>
|
||||||
|
<span class="bkm-card-actions">
|
||||||
|
<button type="button" class="bkm-rename" title="Rename" aria-label="Rename ${esc(label)}">${PENCIL_IC}</button>
|
||||||
|
<button type="button" class="sub-remove" aria-label="Remove ${esc(label)}">${TRASH_IC}</button>
|
||||||
|
</span>
|
||||||
|
</div>`;
|
||||||
|
li.querySelector(".bkm-rename").onclick = () => {
|
||||||
|
const next = window.prompt("Rename this saved location:", b.label || "");
|
||||||
|
if (next && next.trim()) bookmarks.rename(b.id, next.trim());
|
||||||
|
};
|
||||||
|
li.querySelector(".sub-remove").onclick = () => bookmarks.remove(b.id);
|
||||||
|
return li;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderBookmarksSection() {
|
||||||
|
const el = document.getElementById("bkm-alerts-section");
|
||||||
|
if (!el) return;
|
||||||
|
const list = bookmarks.list();
|
||||||
|
el.innerHTML = `
|
||||||
|
<h2 class="bkm-heading" id="bkm-heading">Saved locations</h2>
|
||||||
|
<div class="bkm-import-row">
|
||||||
|
<p class="alerts-note muted">Places you've starred on the map, kept in sync with your account.</p>
|
||||||
|
<button type="button" class="find-btn" id="bkm-import-btn">Import saved locations from this device</button>
|
||||||
|
</div>
|
||||||
|
<p class="bkm-import-status muted" id="bkm-import-status" hidden></p>
|
||||||
|
<ul class="sub-list" id="bkm-list"></ul>`;
|
||||||
|
const ul = el.querySelector("#bkm-list");
|
||||||
|
if (!list.length) {
|
||||||
|
ul.innerHTML = `<li class="sub-empty muted">No saved locations yet. Star a spot on the map to see it here.</li>`;
|
||||||
|
} else {
|
||||||
|
list.forEach((b) => ul.appendChild(bkmCard(b)));
|
||||||
|
}
|
||||||
|
el.querySelector("#bkm-import-btn").onclick = async () => {
|
||||||
|
const btn = el.querySelector("#bkm-import-btn");
|
||||||
|
const status = el.querySelector("#bkm-import-status");
|
||||||
|
btn.disabled = true;
|
||||||
|
try {
|
||||||
|
const res = await bookmarks.importAllLocal();
|
||||||
|
status.hidden = false;
|
||||||
|
status.textContent = res.imported
|
||||||
|
? `Imported ${res.imported} location${res.imported === 1 ? "" : "s"}${res.skipped ? ` (${res.skipped} already saved)` : ""}.`
|
||||||
|
: "Nothing new to import from this device.";
|
||||||
|
} catch (e) {
|
||||||
|
status.hidden = false;
|
||||||
|
status.textContent = e.message || "Couldn't import right now.";
|
||||||
|
} finally {
|
||||||
|
btn.disabled = false;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Bookmarks can change from outside this render cycle (rename/remove above,
|
||||||
|
// or an import banner) — keep the section live rather than only painting once.
|
||||||
|
bookmarks.subscribe(() => renderBookmarksSection());
|
||||||
|
|
||||||
// --- push notifications toggle (this device) ---------------------------------
|
// --- push notifications toggle (this device) ---------------------------------
|
||||||
// Delivery over OS push is per-device: a subscription lives in each browser, so
|
// Delivery over OS push is per-device: a subscription lives in each browser, so
|
||||||
// this control reflects/toggles *this* device, separate from the account-wide
|
// this control reflects/toggles *this* device, separate from the account-wide
|
||||||
|
|
|
||||||
|
|
@ -198,6 +198,10 @@
|
||||||
{% block body_scripts %}
|
{% block body_scripts %}
|
||||||
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
|
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
|
||||||
<script type="module" src="{{ asset_base }}/app.js"></script>
|
<script type="module" src="{{ asset_base }}/app.js"></script>
|
||||||
|
{# Bookmarked locations: a star toggle in the results share row + a "Saved"
|
||||||
|
dropdown by the Find button. Loaded as its own module alongside app.js
|
||||||
|
(not merged into it) — see bookmarks-ui.js's header comment for why. #}
|
||||||
|
<script type="module" src="{{ asset_base }}/bookmarks-ui.js"></script>
|
||||||
{# The records strip renders real temperatures server-side as .temp spans, so
|
{# The records strip renders real temperatures server-side as .temp spans, so
|
||||||
it needs the same converter the SEO pages use or they'd stay in °F while
|
it needs the same converter the SEO pages use or they'd stay in °F while
|
||||||
the toggle says °C. Both import units.js, which the module loader dedupes. #}
|
the toggle says °C. Both import units.js, which the module loader dedupes. #}
|
||||||
|
|
|
||||||
55
frontend/tests/unit/test_app_js_grade_date.py
Normal file
55
frontend/tests/unit/test_app_js_grade_date.py
Normal file
|
|
@ -0,0 +1,55 @@
|
||||||
|
"""Regression guard for the grade-URL "today" date bug: runGrade() in app.js used
|
||||||
|
to omit `date` from the /api/v2/grade request whenever the selected date equaled
|
||||||
|
the viewer's local today, so the backend fell back to ITS OWN UTC today
|
||||||
|
(api_grade's `date` param default in backend/web/app.py) -- a day behind local
|
||||||
|
midnight for any viewer east of UTC. Reported from Kaunas (UTC+3): asking for
|
||||||
|
the Weekly tab on 7/26 rendered 7/25.
|
||||||
|
|
||||||
|
There is no JS/DOM test harness in this repo to drive app.js and assert on the
|
||||||
|
real fetch() URL it builds -- static/package.json declares zero dependencies, so
|
||||||
|
there is no jest/playwright/node runner wired into CI, and nothing else covers
|
||||||
|
static/*.js runtime behavior. So this can't be a behavioral test. What it CAN
|
||||||
|
do, hermetically, through the same served-asset route test_pages.py already
|
||||||
|
exercises (`GET {B}/app.js`), is treat the real shipped source as data and
|
||||||
|
assert the structural property the fix guarantees: runGrade() builds exactly one
|
||||||
|
grade URL and always includes an explicit `date=`, rather than branching on
|
||||||
|
whether the selected date is "today" and omitting it in that case.
|
||||||
|
|
||||||
|
This fails against the pre-fix source and passes against the fixed source --
|
||||||
|
a source guard, not a substitute for a real behavioral test.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
|
||||||
|
B = "/thermograph" # matches THERMOGRAPH_BASE set in tests/conftest.py
|
||||||
|
|
||||||
|
|
||||||
|
def _run_grade_body(client) -> str:
|
||||||
|
r = client.get(f"{B}/app.js")
|
||||||
|
assert r.status_code == 200
|
||||||
|
src = r.text
|
||||||
|
m = re.search(r"async function runGrade\(\) \{\n(.*?)\n\}\n", src, re.DOTALL)
|
||||||
|
assert m, "runGrade() not found in app.js -- update this test if it moved/was renamed"
|
||||||
|
return m.group(1)
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_grade_always_sends_explicit_date(client):
|
||||||
|
"""The historical bug: `dateInput.value === todayISO() ? grade?q : grade?q&date=...`
|
||||||
|
omitted `date` for "today", so the backend's own UTC-today fallback rendered the
|
||||||
|
previous day for any UTC+ viewer between their local midnight and UTC midnight."""
|
||||||
|
body = _run_grade_body(client)
|
||||||
|
assert not re.search(r"dateInput\.value\s*===\s*todayISO\(\)", body), \
|
||||||
|
"runGrade() still branches on today to decide whether to send date="
|
||||||
|
assert body.count("grade?") == 1, \
|
||||||
|
"runGrade() should build exactly one grade URL, not two divergent branches"
|
||||||
|
assert re.search(r"date=\$\{[^}]+\}", body), \
|
||||||
|
"runGrade() must always build the URL with an explicit date= param"
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_grade_never_sends_an_empty_date(client):
|
||||||
|
"""A cleared native date field reports dateInput.value === "" (day.js guards the
|
||||||
|
same input for the same reason). An empty date= hits the identical server-UTC
|
||||||
|
fallback this fix exists to avoid, so runGrade() must fall back to the viewer's
|
||||||
|
own local today rather than ever send the param empty."""
|
||||||
|
body = _run_grade_body(client)
|
||||||
|
assert re.search(r"dateInput\.value\s*\|\|\s*todayISO\(\)", body), \
|
||||||
|
"runGrade() should fall back to todayISO() when dateInput.value is empty"
|
||||||
|
|
@ -28,12 +28,24 @@ python3 .claude/skills/key-gaps/key_gaps.py \
|
||||||
|
|
||||||
## Audit the LIVE boxes (read-only, key names only)
|
## Audit the LIVE boxes (read-only, key names only)
|
||||||
|
|
||||||
Gather the key names over SSH (never the values), then audit. Hosts/keys per INFRA.md:
|
Gather the key names over SSH (never the values), then audit. Hosts per the
|
||||||
|
root `CLAUDE.md` topology:
|
||||||
|
|
||||||
|
- **vps2** (`169.58.46.181`) hosts BOTH prod and beta as separate Swarm
|
||||||
|
stacks — TWO live env files on the same box: `/etc/thermograph.env` (prod)
|
||||||
|
and `/etc/thermograph-beta.env` (beta). "The live env file per environment"
|
||||||
|
is no longer one path per host; it's one path per environment, and both
|
||||||
|
environments' files live on this one host. Get both in the same SSH round
|
||||||
|
trip, or two separate commands against the same host — never assume one
|
||||||
|
host means one file here.
|
||||||
|
- **vps1** (`75.119.132.91`) hosts dev, at the same `/etc/thermograph.env`
|
||||||
|
path (it's a different host, so no collision with prod's file of the same
|
||||||
|
name).
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
K=~/.ssh/thermograph_agent_ed25519
|
K=~/.ssh/thermograph_agent_ed25519
|
||||||
# sudo: /etc/thermograph.env is root-owned (0640). On prod `agent` can read it
|
# sudo: the env files are root-owned (0640). On these boxes `agent` can often
|
||||||
# directly too, but sudo works uniformly on both boxes.
|
# read them directly too, but sudo works uniformly regardless.
|
||||||
#
|
#
|
||||||
# The character class must allow DIGITS. This was `^[A-Z_]+=` until 2026-07-24,
|
# The character class must allow DIGITS. This was `^[A-Z_]+=` until 2026-07-24,
|
||||||
# which silently skipped every key whose name contains a digit — in this estate
|
# which silently skipped every key whose name contains a digit — in this estate
|
||||||
|
|
@ -44,9 +56,10 @@ K=~/.ssh/thermograph_agent_ed25519
|
||||||
# no audit: the invented finding sends someone to provision a credential that
|
# no audit: the invented finding sends someone to provision a credential that
|
||||||
# already exists, and it was briefly recorded as a root cause of a real bug.
|
# already exists, and it was briefly recorded as a root cause of a real bug.
|
||||||
# The match still stops at the `=`, so no value is ever read.
|
# The match still stops at the `=`, so no value is ever read.
|
||||||
ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/prod.keys # prod
|
ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/prod.keys # vps2: prod
|
||||||
ssh -i $K agent@75.119.132.91 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/beta.keys # beta
|
ssh -i $K agent@169.58.46.181 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph-beta.env' > /tmp/beta.keys # vps2: beta
|
||||||
python3 .claude/skills/key-gaps/key_gaps.py prod=/tmp/prod.keys beta=/tmp/beta.keys
|
ssh -i $K agent@75.119.132.91 'sudo grep -oE "^[A-Z][A-Z0-9_]*=" /etc/thermograph.env' > /tmp/dev.keys # vps1: dev
|
||||||
|
python3 .claude/skills/key-gaps/key_gaps.py prod=/tmp/prod.keys beta=/tmp/beta.keys dev=/tmp/dev.keys
|
||||||
```
|
```
|
||||||
|
|
||||||
## Verify a SOPS cutover matches live
|
## Verify a SOPS cutover matches live
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,11 @@
|
||||||
# Local docker-compose overrides. Copy to .env (gitignored) for `docker compose
|
# Local docker-compose overrides. Copy to .env (gitignored) for `docker compose
|
||||||
# up` / `make up` on a dev box: cp .env.example .env
|
# up` / `make up` on a laptop: cp .env.example .env
|
||||||
#
|
#
|
||||||
# Compose auto-reads a repo-root .env for ${VAR} interpolation in
|
# Compose auto-reads a repo-root .env for ${VAR} interpolation in
|
||||||
# docker-compose.yml. In production these live in /etc/thermograph.env instead
|
# docker-compose.yml. On the fleet (dev/beta/prod), these values live in each
|
||||||
# (loaded by the systemd unit), so this file is only for local runs.
|
# environment's own rendered env file instead (/etc/thermograph.env,
|
||||||
|
# /etc/thermograph-beta.env — see deploy/secrets/README.md), so this file is
|
||||||
|
# only for a local, unmanaged run.
|
||||||
|
|
||||||
# Database password. Compose uses it to initialize the postgres container AND to
|
# Database password. Compose uses it to initialize the postgres container AND to
|
||||||
# build the app's THERMOGRAPH_DATABASE_URL. Change it before first `up`.
|
# build the app's THERMOGRAPH_DATABASE_URL. Change it before first `up`.
|
||||||
|
|
@ -33,8 +35,13 @@ THERMOGRAPH_INTERNAL_TOKEN=
|
||||||
# replicate with another -- see docker-compose.yml's db service comment.
|
# replicate with another -- see docker-compose.yml's db service comment.
|
||||||
# TIMESCALEDB_TAG=latest-pg18
|
# TIMESCALEDB_TAG=latest-pg18
|
||||||
|
|
||||||
# Postgres sizing. Terraform sets these per host in prod/beta (prod DB_MEMORY
|
# Postgres sizing. Only meaningful for an environment that runs its OWN db
|
||||||
# 16g); local/beta default to 8g / 2 CPUs.
|
# service — dev's compose stack (this file) and prod's Swarm stack, which
|
||||||
|
# sizes the ONE shared TimescaleDB instance on vps2 (see
|
||||||
|
# deploy/secrets/prod.yaml, currently DB_MEMORY=16g). Beta shares that same
|
||||||
|
# instance rather than running a second one, so a DB_MEMORY/DB_CPUS value in
|
||||||
|
# beta's own vault file no longer sizes anything — don't be misled by its
|
||||||
|
# presence there. Local/dev default to 8g / 2 CPUs.
|
||||||
# DB_MEMORY=8g
|
# DB_MEMORY=8g
|
||||||
# DB_CPUS=2
|
# DB_CPUS=2
|
||||||
|
|
||||||
|
|
|
||||||
158
infra/ACCESS.md
158
infra/ACCESS.md
|
|
@ -5,28 +5,55 @@ provisioning / Postgres / Terraform work described in `terraform/README.md`
|
||||||
and (for the historical Track A/Track B plan this whole effort grew from,
|
and (for the historical Track A/Track B plan this whole effort grew from,
|
||||||
including the decision to split this repo out of the app monorepo)
|
including the decision to split this repo out of the app monorepo)
|
||||||
`thermograph-docs/runbooks/implementation-handoff.md` and
|
`thermograph-docs/runbooks/implementation-handoff.md` and
|
||||||
`thermograph-docs/architecture/repo-topology-and-infrastructure.md`. Terraform
|
`thermograph-docs/architecture/repo-topology-and-infrastructure.md`.
|
||||||
provisions **prod** (the new 48 GB / 12-core box, `thermograph.org`) and
|
**Note:** `terraform/README.md`'s own host table still describes the
|
||||||
**beta** (the old VPS, `75.119.132.91`) — see `terraform/README.md` /
|
pre-cutover shape (`prod` and `beta` as two single-purpose boxes) and has not
|
||||||
`terraform.tfvars.example`. **None of this touches the app repo** (its
|
yet been updated to the vps1/vps2 split below — that file is out of this
|
||||||
backend/frontend source, `Dockerfile`, or CI) — this repo owns only how and
|
pass's scope; treat this document as the current source of truth for the
|
||||||
where the already-built app image runs; the app repo owns building it.
|
physical topology in the meantime.
|
||||||
|
|
||||||
```
|
```
|
||||||
Track 1: deploy/provision-agent-access.sh — a dedicated full-root login for me
|
Track 1: deploy/provision-agent-access.sh — a dedicated full-root login for me
|
||||||
Track 2: deploy/swarm/ — Swarm cluster spanning THREE nodes
|
Track 2: deploy/swarm/ — Swarm mesh spanning THREE nodes
|
||||||
Track 3: deploy/forgejo/ + .forgejo/ — Forgejo + Forgejo Actions, replacing GitHub
|
Track 3: deploy/forgejo/ + .forgejo/ — Forgejo + Forgejo Actions, replacing GitHub
|
||||||
```
|
```
|
||||||
|
|
||||||
**Three nodes, not two**: prod, beta, and **the desktop** (the LAN dev
|
## The estate (renamed by role, not by environment)
|
||||||
machine — same box that already runs the pre-Forgejo GitHub self-hosted
|
|
||||||
runner). This matches the canonical Track B design in the handoff doc; an
|
The same two VPS boxes as always — **neither public IP nor WireGuard mesh
|
||||||
earlier revision of this whole effort covered just prod+beta and has been
|
address moved** — plus the operator's desktop, now named for what they *do*
|
||||||
realigned.
|
rather than which environment happens to live there:
|
||||||
|
|
||||||
|
| Host | Public IP | Mesh IP | Role |
|
||||||
|
|------|-----------|---------|------|
|
||||||
|
| **vps1** | `75.119.132.91` | `10.10.0.2` | "Operational programs": Forgejo (git + CI + registry, `git.thermograph.org`), Grafana/Loki/Alloy (`dashboard.thermograph.org`), the `emigriffith.dev` portfolio, and the **dev** environment (mesh-only, `10.10.0.2:8137`, no public DNS/Caddy/TLS) |
|
||||||
|
| **vps2** | `169.58.46.181` | `10.10.0.1` | "The deployed environment": **prod** and **beta**, as two separate Docker Swarm stacks, plus Centralis, Postfix, the backups, and the one shared TimescaleDB instance both stacks use |
|
||||||
|
| **desktop** | — | `10.10.0.3` | AI-model hosting (voice-to-text, an upcoming-feature LLM) plus flex Swarm-worker capacity. Hosts **no** Thermograph environment — `make dev-up` still works there, but only as a laptop-local rehearsal |
|
||||||
|
|
||||||
|
This is a **rename**, not a re-provisioning: vps1 is the box that used to be
|
||||||
|
called "beta" (it already ran Forgejo and Grafana before this split), and vps2
|
||||||
|
is the box that used to be called "prod". What moved is the **beta
|
||||||
|
environment** — off vps1 and onto vps2, so a beta green light is evidence
|
||||||
|
about prod (same orchestrator, same Postgres build, same Caddy, same mesh
|
||||||
|
position) — and the **dev environment**, off the desktop and onto vps1, so the
|
||||||
|
desktop can retire from the Thermograph estate entirely. See
|
||||||
|
`deploy/env-topology.sh`'s header comment for the full rationale.
|
||||||
|
|
||||||
|
**Three nodes on the Swarm mesh, not two**: vps2 (manager), vps1 (worker), and
|
||||||
|
the desktop (worker, flex capacity + AI model hosting). One manager, not more:
|
||||||
|
Raft needs 3 nodes for real quorum-based HA, and this cluster only has 3 nodes
|
||||||
|
total, so a second manager would still fall short of real HA while adding
|
||||||
|
split-brain risk. If the manager (vps2) goes down, the workers keep running
|
||||||
|
whatever was already scheduled on them (Forgejo, pinned to vps1) but the
|
||||||
|
cluster can't reschedule anything until vps2 is back — acceptable for a small
|
||||||
|
cluster whose only Swarm-scheduled workload today is Forgejo (prod and beta's
|
||||||
|
app stacks are separate Swarm stacks that also happen to run on vps2, the
|
||||||
|
manager, because their volumes are local to that node — see
|
||||||
|
`deploy/stack/thermograph-stack.yml`'s header).
|
||||||
|
|
||||||
## Track 1 — Agent access
|
## Track 1 — Agent access
|
||||||
|
|
||||||
Run `sudo bash deploy/provision-agent-access.sh` on **prod and beta** (not
|
Run `sudo bash deploy/provision-agent-access.sh` on **vps1 and vps2** (not
|
||||||
the desktop — that's wherever you're already working from, no separate
|
the desktop — that's wherever you're already working from, no separate
|
||||||
access-provisioning step needed there). Creates a dedicated `agent` user (not
|
access-provisioning step needed there). Creates a dedicated `agent` user (not
|
||||||
raw root login — a distinct name gives a clean audit trail) with passwordless
|
raw root login — a distinct name gives a clean audit trail) with passwordless
|
||||||
|
|
@ -42,20 +69,27 @@ Both VPS boxes are provisioned and reachable as of this writing:
|
||||||
|
|
||||||
| Host | Role | Public IP | Login |
|
| Host | Role | Public IP | Login |
|
||||||
|------|------|-----------|-------|
|
|------|------|-----------|-------|
|
||||||
| prod | Swarm manager; Thermograph's live prod home (`release`, thermograph.org) | `169.58.46.181` | `agent` |
|
| vps2 | Swarm manager; prod AND beta live here (`thermograph.org`, `beta.thermograph.org`) | `169.58.46.181` | `agent` |
|
||||||
| beta | Swarm worker; `main`/beta.thermograph.org + hosts Forgejo | `75.119.132.91` | `agent` |
|
| vps1 | Swarm worker; Forgejo + Grafana/Loki + dev (`git.thermograph.org`, `dashboard.thermograph.org`) | `75.119.132.91` | `agent` |
|
||||||
| desktop | Swarm worker, LAN dev machine + Forgejo Actions runner | (local) | (already have access) |
|
| desktop | Swarm worker, AI-model hosting | (local) | (already have access) |
|
||||||
|
|
||||||
```
|
```
|
||||||
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # prod
|
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@169.58.46.181 # vps2 (prod + beta)
|
||||||
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # beta
|
ssh -i ~/.ssh/thermograph_agent_ed25519 agent@75.119.132.91 # vps1 (dev + Forgejo + Grafana)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**On vps2, always say which environment you mean.** A shell on vps2 can act on
|
||||||
|
either `/opt/thermograph` (prod) or `/opt/thermograph-beta` (beta) — there is
|
||||||
|
no "the app" on that box any more. `deploy.sh` refuses to run if
|
||||||
|
`THERMOGRAPH_ENV` disagrees with the checkout it was invoked from (see
|
||||||
|
`deploy/env-topology.sh`), which catches the one genuinely dangerous mistake
|
||||||
|
here: running prod's checkout with `THERMOGRAPH_ENV=beta`, or the reverse.
|
||||||
|
|
||||||
The private half of the agent's dedicated keypair lives at
|
The private half of the agent's dedicated keypair lives at
|
||||||
`~/.ssh/thermograph_agent_ed25519` on the operator's own machine (never in
|
`~/.ssh/thermograph_agent_ed25519` on the operator's own machine (never in
|
||||||
this repo, never in a CI secret) — same convention `DEPLOY.md` already uses
|
this repo, never in a CI secret) — same convention `DEPLOY.md` already uses
|
||||||
for the separate CI deploy key (`~/.ssh/thermograph_ci`). If you're setting
|
for the separate CI deploy keys. If you're setting this up fresh elsewhere,
|
||||||
this up fresh elsewhere, generate a new pair the same way
|
generate a new pair the same way
|
||||||
(`ssh-keygen -t ed25519 -a 100 -C "claude-agent@thermograph-infra" -f
|
(`ssh-keygen -t ed25519 -a 100 -C "claude-agent@thermograph-infra" -f
|
||||||
~/.ssh/thermograph_agent_ed25519 -N ""`), embed the new public half in
|
~/.ssh/thermograph_agent_ed25519 -N ""`), embed the new public half in
|
||||||
`AGENT_PUBKEY` at the top of `provision-agent-access.sh`, and re-run it on
|
`AGENT_PUBKEY` at the top of `provision-agent-access.sh`, and re-run it on
|
||||||
|
|
@ -67,31 +101,21 @@ directory that gets cleaned up, not anywhere-in-repo. Losing it just means
|
||||||
regenerating and re-running the provisioning script; it does not lock either
|
regenerating and re-running the provisioning script; it does not lock either
|
||||||
box, since your own login is untouched.
|
box, since your own login is untouched.
|
||||||
|
|
||||||
**Live as of 2026-07-21** (was: "no Docker / Terraform not applied / Swarm
|
|
||||||
inactive" — all now done): Docker is installed on all three nodes; the
|
|
||||||
WireGuard mesh and Docker Swarm are up with all three nodes `Ready` (prod
|
|
||||||
manager, beta + desktop workers); Terraform has been applied to both VPS
|
|
||||||
boxes (prod stood up fresh on `release`, beta rebuilt on `main`); Forgejo is
|
|
||||||
serving at `git.thermograph.org`; and the desktop runs the Forgejo Actions
|
|
||||||
runner. See the verification checklist at the end.
|
|
||||||
|
|
||||||
## Track 2 — Swarm
|
## Track 2 — Swarm
|
||||||
|
|
||||||
See `deploy/swarm/README.md` for the exact order of operations across all
|
See `deploy/swarm/README.md` for the exact order of operations across all
|
||||||
**three** nodes (WireGuard mesh first, then swarm init/join, then lock the
|
**three** nodes (WireGuard mesh first, then swarm init/join, then lock the
|
||||||
Swarm ports down to the tunnel interface, then label beta for Forgejo
|
Swarm ports down to the tunnel interface, then label vps1 for Forgejo
|
||||||
placement). This cluster's only job is hosting Forgejo — it does not
|
placement). This cluster's Swarm-scheduled workload is Forgejo; prod and
|
||||||
orchestrate the Terraform-managed app deploys.
|
beta's app stacks are separate `docker stack deploy`s that happen to live on
|
||||||
|
the same manager node (vps2) because their volumes are local to it today.
|
||||||
|
|
||||||
## Track 3 — Forgejo, replacing GitHub
|
## Track 3 — Forgejo, replacing GitHub
|
||||||
|
|
||||||
### 3a. Stand up Forgejo
|
### 3a. Stand up Forgejo
|
||||||
See `deploy/forgejo/README.md` — deploys the stack (pinned to beta), then
|
See `deploy/forgejo/README.md` — deploys the stack (pinned to **vps1**), then
|
||||||
walks through registering the Actions runner **on the desktop** as a plain
|
walks through registering the Actions runner as a plain systemd service
|
||||||
systemd service (`register-lan-runner.sh`), not as a Swarm-scheduled
|
(`register-lan-runner.sh`), not as a Swarm-scheduled container.
|
||||||
container. Only one Swarm secret to mint now (`forgejo_db_password`) — the
|
|
||||||
runner token is no longer a Swarm secret, since the runner isn't a Swarm
|
|
||||||
service anymore.
|
|
||||||
|
|
||||||
### 3b. Migrate the repo (mirror first, verify, then cut over)
|
### 3b. Migrate the repo (mirror first, verify, then cut over)
|
||||||
1. In Forgejo: **+ New Migration → GitHub**. Point it at
|
1. In Forgejo: **+ New Migration → GitHub**. Point it at
|
||||||
|
|
@ -103,13 +127,12 @@ service anymore.
|
||||||
3. **Don't retarget any secret or disable a GitHub workflow yet** — see 3d.
|
3. **Don't retarget any secret or disable a GitHub workflow yet** — see 3d.
|
||||||
|
|
||||||
### 3c. Workflows
|
### 3c. Workflows
|
||||||
`.forgejo/workflows/{build,pr-build,deploy-dev,deploy}.yml` mirror
|
`.forgejo/workflows/{build,pr-build,deploy}.yml` (a single `Deploy` workflow
|
||||||
|
now covers all three environments — see its own header comment) mirror
|
||||||
`.github/workflows/*.yml`, copied with the mechanical changes Forgejo needs:
|
`.github/workflows/*.yml`, copied with the mechanical changes Forgejo needs:
|
||||||
- `runs-on: ubuntu-latest` → `runs-on: docker` (one of the two labels the
|
- `runs-on: ubuntu-latest` → `runs-on: docker` (the runner's `docker` label —
|
||||||
desktop's runner registers under — see 3a; general CI/deploy jobs get a
|
containerized jobs get a fresh container via the docker-socket-automounted
|
||||||
fresh container, the LAN-deploy job in `deploy-dev.yml` runs bare/host-native
|
runner).
|
||||||
under the `thermograph-lan` label instead, since it needs real filesystem
|
|
||||||
access).
|
|
||||||
- `appleboy/ssh-action` referenced by full GitHub URL (not mirrored in
|
- `appleboy/ssh-action` referenced by full GitHub URL (not mirrored in
|
||||||
Forgejo's default action registry, unlike `actions/checkout` /
|
Forgejo's default action registry, unlike `actions/checkout` /
|
||||||
`actions/setup-python`, which resolve unchanged).
|
`actions/setup-python`, which resolve unchanged).
|
||||||
|
|
@ -119,44 +142,51 @@ service anymore.
|
||||||
`build` status check required (Settings → Branches). Forgejo does **not**
|
`build` status check required (Settings → Branches). Forgejo does **not**
|
||||||
auto-merge on green by itself — a ready, green, mergeable PR is merged
|
auto-merge on green by itself — a ready, green, mergeable PR is merged
|
||||||
**explicitly** (`POST .../pulls/{n}/merge`, `{"Do": "squash"}`), per
|
**explicitly** (`POST .../pulls/{n}/merge`, `{"Do": "squash"}`), per
|
||||||
`CLAUDE.md`. That merge is an ordinary push, so `deploy-dev.yml`'s push
|
`CLAUDE.md`. That merge is an ordinary push, so `deploy.yml`'s push
|
||||||
trigger fires naturally afterward.
|
trigger fires naturally afterward.
|
||||||
- **Branch/deploy mapping (settled):** `.forgejo/workflows/deploy.yml`
|
- **Branch/deploy mapping (settled):** `.forgejo/workflows/deploy.yml`
|
||||||
("Deploy to beta VPS") triggers on `main` and SSHes to **beta**
|
triggers on `dev`, `main` and `release`, and SSHes to the environment each
|
||||||
(`75.119.132.91`) running `deploy/deploy.sh`. **Prod is not deployed by a
|
branch maps to (`dev` → dev on **vps1**, `main` → beta on **vps2**,
|
||||||
push-triggered workflow** — it's deployed with `terraform apply`
|
`release` → prod on **vps2**), running `deploy/deploy-dev.sh` or
|
||||||
(`terraform/README.md`) on the `release` branch. So the promotion chain is
|
`deploy/deploy.sh` per `deploy/env-topology.sh`. Credentials are keyed by
|
||||||
`dev` → `main` (this workflow deploys beta) → `release` (Terraform deploys
|
**host** (`VPS1_SSH_*`, `VPS2_SSH_*`), not by environment, since vps2 alone
|
||||||
prod). There is deliberately no `release`-triggered workflow.
|
now answers to two of them — see the workflow's own header comment for why
|
||||||
|
that rename matters.
|
||||||
|
|
||||||
A second, independent set of Forgejo workflows
|
A second, independent set of Forgejo workflows
|
||||||
(`.forgejo/workflows/{ci,build-push,ops-cron}.yml`) was added by a parallel
|
(`.forgejo/workflows/{build-push,ops-cron}.yml`) was added by a parallel
|
||||||
effort implementing `thermograph-docs/runbooks/implementation-handoff.md` Track A chunk
|
effort implementing `thermograph-docs/runbooks/implementation-handoff.md` Track A chunk
|
||||||
7 — see that doc and its own reconciliation PR (which already fixed its
|
7 — see that doc and its own reconciliation PR. `build-push.yml` (image →
|
||||||
runner-label and ssh-action guesses to match the real infrastructure here).
|
Forgejo's built-in registry) and `ops-cron.yml` (backup + IndexNow) are novel
|
||||||
`ci.yml` duplicated this PR's `build.yml` exactly and was dropped in that
|
and still present; the latter's secrets are also keyed by host now (see its
|
||||||
reconciliation; `build-push.yml` (image → Forgejo's built-in registry) and
|
own header comment).
|
||||||
`ops-cron.yml` (backup + IndexNow) are novel and still present.
|
|
||||||
|
|
||||||
### 3d. Cut over — DONE (2026-07-21)
|
### 3d. Cut over — DONE (2026-07-21)
|
||||||
The GitHub → Forgejo cutover is complete; this records what was done:
|
The GitHub → Forgejo cutover is complete; this records what was done:
|
||||||
1. Deploy secrets (`SSH_HOST`, `SSH_USER`, `SSH_KEY`, `SSH_PORT`) live in
|
1. Deploy secrets live in Forgejo's repo Secrets, keyed by host
|
||||||
Forgejo's repo Secrets, pointing at beta with a dedicated CI deploy key.
|
(`VPS1_SSH_*`, `VPS2_SSH_*`) since the vps1/vps2 split, pointing at each
|
||||||
2. The LAN dev runner was re-pointed to Forgejo via
|
box with a dedicated CI deploy key.
|
||||||
`deploy/forgejo/register-lan-runner.sh` (systemd --user, on the desktop).
|
2. The Actions runner is registered against Forgejo via
|
||||||
|
`deploy/forgejo/register-lan-runner.sh`.
|
||||||
3. The repo was migrated into Forgejo; `dev`/`main`/`release` all promoted
|
3. The repo was migrated into Forgejo; `dev`/`main`/`release` all promoted
|
||||||
through Forgejo and deploys verified live.
|
through Forgejo and deploys verified live.
|
||||||
4. GitHub is retired as git host and CI (PR "Retire GitHub as the git host and
|
4. GitHub is retired as git host and CI (PR "Retire GitHub as the git host and
|
||||||
CI platform") — `origin` remains only as a read-only mirror of history.
|
CI platform") — `origin` remains only as a read-only mirror of history.
|
||||||
|
|
||||||
## Verification checklist — all met (2026-07-21)
|
## Verification checklist — all met (2026-07-21, topology as of 2026-07-25)
|
||||||
- [x] `ssh agent@<prod>` and `ssh agent@<beta>` both work; `sudo whoami` → `root`
|
- [x] `ssh agent@<vps1>` and `ssh agent@<vps2>` both work; `sudo whoami` → `root`
|
||||||
- [x] Password SSH auth confirmed dead on both boxes (tested with an actual
|
- [x] Password SSH auth confirmed dead on both boxes (tested with an actual
|
||||||
failed login attempt, not just config inspection)
|
failed login attempt, not just config inspection)
|
||||||
- [x] `docker node ls` (from prod) shows all three nodes `Ready`
|
- [x] `docker node ls` (from vps2, the manager) shows all three nodes `Ready`
|
||||||
- [x] Swarm ports (2377/7946/4789) unreachable from outside the WireGuard tunnel
|
- [x] Swarm ports (2377/7946/4789) unreachable from outside the WireGuard tunnel
|
||||||
- [x] `https://git.thermograph.org` serves Forgejo over TLS; `/v2/` registry
|
- [x] `https://git.thermograph.org` serves Forgejo over TLS; `/v2/` registry
|
||||||
API is 403 from off-mesh (firewalled to the WireGuard CIDR)
|
API is 403 from off-mesh (firewalled to the WireGuard CIDR)
|
||||||
- [x] A test PR flow verified: required `build` check runs → explicit squash
|
- [x] A test PR flow verified: required `build` check runs → explicit squash
|
||||||
merge → LAN dev deploy lands
|
merge → deploy lands on the branch's mapped environment
|
||||||
- [x] GitHub retired as git host + CI; kept only as a read-only history mirror
|
- [x] GitHub retired as git host + CI; kept only as a read-only history mirror
|
||||||
|
|
||||||
|
TODO(cutover): re-verify the checklist above against the vps1/vps2 rename
|
||||||
|
specifically (e.g. confirm the Forgejo Swarm label now reads `vps1`'s node
|
||||||
|
name, not a leftover `beta` node name) — this document was updated to match
|
||||||
|
the new topology brief, but a fresh live check wasn't run as part of this
|
||||||
|
pass.
|
||||||
|
|
|
||||||
|
|
@ -8,33 +8,67 @@ orchestrator.
|
||||||
|
|
||||||
## The machines
|
## The machines
|
||||||
|
|
||||||
Per `ACCESS.md`: **dev** (operator's box — LAN dev server and CI runner),
|
Two VPS boxes plus the operator's desktop, on one WireGuard mesh, named by
|
||||||
**prod** (`169.58.46.181`, thermograph.org, `agent` user, passwordless sudo),
|
ROLE rather than by environment:
|
||||||
**beta** (`75.119.132.91`, beta.thermograph.org + Forgejo + Grafana/Loki,
|
|
||||||
`agent` user). All four are on a WireGuard mesh. Hosts' `/opt/thermograph` is a
|
- **vps1** — `75.119.132.91`, mesh `10.10.0.2`. "Operational programs": Forgejo
|
||||||
checkout of **this monorepo**, not an infra-only repo.
|
(git + CI + registry, `git.thermograph.org`), Grafana/Loki/Alloy
|
||||||
|
(`dashboard.thermograph.org`), the `emigriffith.dev` portfolio site, and the
|
||||||
|
**dev** environment (`/opt/thermograph-dev`, tracks `dev`), with its own
|
||||||
|
Postgres container. Dev is mesh-only here — published on `10.10.0.2:8137`,
|
||||||
|
no public DNS, no Caddy site, no TLS.
|
||||||
|
- **vps2** — `169.58.46.181`, mesh `10.10.0.1`. "The deployed environment" —
|
||||||
|
everything an external user can reach: **prod** (`/opt/thermograph`, tracks
|
||||||
|
`main`, Swarm stack `thermograph`) AND **beta** (`/opt/thermograph-beta`,
|
||||||
|
tracks `main`, Swarm stack `thermograph-beta`), Centralis, Postfix, and the
|
||||||
|
backups. One shared TimescaleDB instance serves both, on separate
|
||||||
|
databases/roles (`thermograph` / `thermograph_beta`).
|
||||||
|
- **desktop** — mesh `10.10.0.3`. Hosts no Thermograph environment at all: AI
|
||||||
|
model hosting plus flex Swarm-worker capacity. `make dev-up` still works
|
||||||
|
there as a laptop-local rehearsal, but that is not an environment.
|
||||||
|
|
||||||
|
`agent` user, passwordless sudo, on vps1 and vps2. A host's `/opt/thermograph*`
|
||||||
|
is a checkout of **this monorepo**, not an infra-only repo — note vps2 carries
|
||||||
|
**two** such checkouts side by side (`/opt/thermograph` and
|
||||||
|
`/opt/thermograph-beta`), which is the one thing most deploy logic here has to
|
||||||
|
get right that a single-environment host never had to.
|
||||||
|
|
||||||
## Deploy paths
|
## Deploy paths
|
||||||
|
|
||||||
- **`deploy/deploy.sh`** — the single entry point for beta and prod. Resets the
|
- **`deploy/env-topology.sh`** — sourced by every path below. `THERMOGRAPH_ENV`
|
||||||
host checkout to `BRANCH` (default `main`), renders secrets, then routes:
|
(`dev`/`beta`/`prod`) is the input; host, checkout, branch, deploy mode,
|
||||||
if `/etc/thermograph/deploy-mode` says `stack` it execs
|
stack/compose name, env file, LB ports, DB role/database and service-name
|
||||||
`deploy/stack/deploy-stack.sh`, otherwise it rolls compose services. Per-service
|
prefix are all derived from it. This exists because vps2 alone runs two
|
||||||
tags persist in untracked `deploy/.image-tags.env` (compose) and
|
environments — a host-wide marker can no longer answer "which environment is
|
||||||
|
this", so the caller (the deploy workflow) says so explicitly.
|
||||||
|
- **`deploy/deploy.sh`** — the single entry point for dev, beta and prod.
|
||||||
|
Resets the host checkout to `BRANCH`, renders secrets, then routes: beta and
|
||||||
|
prod (`TG_DEPLOY_MODE=stack`) exec `deploy/stack/deploy-stack.sh`; dev
|
||||||
|
(`compose`) rolls compose services. The old host-wide
|
||||||
|
`/etc/thermograph/deploy-mode` marker is honoured only when
|
||||||
|
`env-topology.sh` can't resolve an environment at all (a checkout that
|
||||||
|
predates the split, or a by-hand run with nothing set). Per-service tags
|
||||||
|
persist in untracked `deploy/.image-tags.env` (compose) and
|
||||||
`deploy/.stack-image-tags.env` (stack) so the two never mix.
|
`deploy/.stack-image-tags.env` (stack) so the two never mix.
|
||||||
- **`deploy/stack/deploy-stack.sh`** — the Swarm path, live on prod. `backend`
|
- **`deploy/stack/deploy-stack.sh`** — the Swarm path, live on vps2 for both
|
||||||
rolls web **and** worker; `frontend` rolls frontend; `all` runs a full
|
prod and beta, picking the stack file/ports/DB role/service prefix out of
|
||||||
`docker stack deploy`. Start-first, health-gated, auto-rollback.
|
`env-topology.sh`. `backend` rolls web **and** worker; `frontend` rolls
|
||||||
`STACK_TEST=1` rehearses the whole thing under stack name `thermograph-test`
|
frontend; `all` runs a full `docker stack deploy`. Start-first, health-gated,
|
||||||
|
auto-rollback. `STACK_TEST=1` rehearses under stack name `thermograph-test`
|
||||||
on throwaway volumes and ports `18137`/`18080`.
|
on throwaway volumes and ports `18137`/`18080`.
|
||||||
- **`deploy/deploy-dev.sh`** — thin LAN-dev wrapper (dev compose overlay,
|
- **`deploy/deploy-dev.sh`** — thin wrapper around `deploy.sh` for dev on vps1
|
||||||
`~/thermograph-dev`). Its CI trigger is inert; see root `CLAUDE.md`.
|
(dev compose overlay, `/opt/thermograph-dev`, `THERMOGRAPH_SECRETS_SKIP_COMMON=1`).
|
||||||
|
Deployed like beta/prod now — a push to `dev` triggers the same `Deploy`
|
||||||
|
workflow over SSH, no LAN-specific runner involved.
|
||||||
- **`Makefile`** — compose orchestration only: `up`, `down`, `db-up`, `dev-up`,
|
- **`Makefile`** — compose orchestration only: `up`, `down`, `db-up`, `dev-up`,
|
||||||
`om-up`, `om-backfill`.
|
`om-up`, `om-backfill`.
|
||||||
|
|
||||||
Volumes are the reason the compose project name is pinned: compose creates
|
Volumes are the reason the compose project name is pinned: compose creates
|
||||||
`thermograph_pgdata`/`_appdata`/`_applogs`, and the Swarm stack declares those
|
`thermograph_pgdata`/`_appdata`/`_applogs`, and the Swarm stack declares those
|
||||||
same names as `external: true` at the same mount paths.
|
same names as `external: true` at the same mount paths. Beta's stack has no
|
||||||
|
`db` of its own — it reaches prod's `db` service over prod's `thermograph_internal`
|
||||||
|
overlay (declared `external: true` in beta's stack file) and keeps a separate
|
||||||
|
`internal` overlay for its own beta-to-beta traffic.
|
||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
|
|
||||||
|
|
@ -42,13 +76,29 @@ same names as `external: true` at the same mount paths.
|
||||||
an apply would attempt full re-provisioning of live hosts. Terraform here is
|
an apply would attempt full re-provisioning of live hosts. Terraform here is
|
||||||
executable documentation until state is bootstrapped.
|
executable documentation until state is bootstrapped.
|
||||||
- **Secrets: SOPS vault only** (`deploy/secrets/*.yaml`, `sops edit` → commit →
|
- **Secrets: SOPS vault only** (`deploy/secrets/*.yaml`, `sops edit` → commit →
|
||||||
deploy). Never hand-edit `/etc/thermograph.env` — it is a rendered artifact.
|
deploy). Never hand-edit `/etc/thermograph.env` / `/etc/thermograph-beta.env`
|
||||||
`secrets-guard` CI rejects plaintext. `deploy/secrets/seed-from-live.sh` reads
|
— they are rendered artifacts. `secrets-guard` CI rejects plaintext.
|
||||||
production secrets and is **not** for an agent to run.
|
`deploy/secrets/seed-from-live.sh` reads production secrets and is **not**
|
||||||
|
for an agent to run.
|
||||||
|
- **vps1 must never hold prod credentials.** It's the box that runs Forgejo,
|
||||||
|
its CI runner, and dev's unreviewed branch — the opposite of an isolation
|
||||||
|
boundary. This is why dev renders `dev.yaml` **alone** and never layers
|
||||||
|
`common.yaml` (the fleet's shared production credential set). See
|
||||||
|
`deploy/secrets/README.md`.
|
||||||
|
- **A foothold on vps2 is a foothold on both beta and prod's host** — they are
|
||||||
|
co-resident by design now. What still separates them is the database
|
||||||
|
(separate roles/databases, `CONNECT` revoked from `PUBLIC`) and the
|
||||||
|
filesystem (separate checkouts, separate rendered env files). Don't describe
|
||||||
|
beta and prod as isolated at the host or SSH-credential level; they aren't
|
||||||
|
anymore.
|
||||||
- **The ops cron (`.forgejo/workflows/ops-cron.yml`, at the repo root) is THE
|
- **The ops cron (`.forgejo/workflows/ops-cron.yml`, at the repo root) is THE
|
||||||
prod backup.** It uses `PROD_SSH_*`, not `SSH_*` — an earlier revision reused
|
backup for both application databases and for Forgejo.** Secrets are keyed
|
||||||
`SSH_*` and silently dumped beta while prod had no backup at all. If you touch
|
by host, not environment: `VPS2_SSH_*` for the `backup`/`indexnow` jobs
|
||||||
it, verify a dump actually lands in `agent@prod:~/thermograph-backups/`.
|
(prod and beta's databases, both on vps2) and `VPS1_SSH_*` for
|
||||||
|
`forgejo-backup` (Forgejo now lives on vps1, not co-located with beta). If
|
||||||
|
you touch it, verify a dump actually lands for **both** the prod and beta
|
||||||
|
databases, not just one — an earlier revision backed up only prod and
|
||||||
|
silently left beta uncovered.
|
||||||
- Shell here runs as root over SSH against live hosts with no test suite in
|
- Shell here runs as root over SSH against live hosts with no test suite in
|
||||||
front of it. `shell-lint` CI (pinned shellcheck) is the only guard — keep the
|
front of it. `shell-lint` CI (pinned shellcheck) is the only guard — keep the
|
||||||
tree at zero findings.
|
tree at zero findings.
|
||||||
|
|
|
||||||
|
|
@ -1,33 +1,45 @@
|
||||||
# Dev CI/CD → LAN server on this machine
|
# Dev CI/CD → the dev environment on vps1
|
||||||
|
|
||||||
Parallel to the prod pipeline (`main` → VPS, see `DEPLOY.md`), the **`dev`**
|
Parallel to beta/prod's pipeline (`main`→beta, `release`→prod, see `DEPLOY.md`),
|
||||||
branch continuously deploys to a **LAN server running on this computer**. Git
|
the **`dev`** branch continuously deploys to the **dev environment on vps1**
|
||||||
hosting and CI are self-hosted **Forgejo** (`git.thermograph.org`, reachable
|
(`75.119.132.91`, mesh `10.10.0.2`) — `/opt/thermograph-dev`, a normal fleet
|
||||||
at `http://10.10.0.2:3080` over the WireGuard mesh) — GitHub is retired.
|
checkout reached over SSH exactly like beta and prod, **not** a stack on the
|
||||||
|
operator's desktop or LAN any more. Git hosting and CI are self-hosted
|
||||||
|
**Forgejo** (`git.thermograph.org`, also on vps1, reachable at
|
||||||
|
`http://10.10.0.2:3080` over the WireGuard mesh) — GitHub is retired.
|
||||||
|
|
||||||
```
|
```
|
||||||
open PR ──▶ CI (build + boot/health, on the LAN Forgejo runner)
|
open PR ──▶ CI (build + boot/health, on the `docker`-labeled Forgejo runner)
|
||||||
│ green
|
│ green
|
||||||
▼
|
▼
|
||||||
merge into dev (explicit — Forgejo does not auto-merge on its own;
|
merge into dev (explicit — Forgejo does not auto-merge on its own;
|
||||||
see "Landing a PR" below)
|
see "Landing a PR" below)
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
LAN runner (thermograph-lan label) runs deploy/deploy-dev.sh
|
the Deploy workflow SSHes into vps1 (VPS1_SSH_*, THERMOGRAPH_ENV=dev)
|
||||||
|
and runs deploy/deploy-dev.sh
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
~/thermograph-dev updated ─▶ docker compose stack (backend + frontend +
|
/opt/thermograph-dev updated ─▶ docker compose stack (backend + frontend +
|
||||||
Postgres 18) brought up, backend on 0.0.0.0:8137
|
Postgres) brought up, backend published
|
||||||
|
MESH-ONLY on 10.10.0.2:8137
|
||||||
```
|
```
|
||||||
|
|
||||||
Reachable at `http://<lan-ip>:8137/` from any device on your Wi-Fi.
|
Reachable at `http://10.10.0.2:8137/` from anything already on the WireGuard
|
||||||
|
mesh, and from nowhere else — no public DNS record, no Caddy site, no TLS.
|
||||||
|
vps1 is a public VPS running whatever branch is currently in flight (including
|
||||||
|
unreviewed code), so unlike the old LAN box it must **never** publish on
|
||||||
|
`0.0.0.0`.
|
||||||
|
|
||||||
The dev server runs the **same containerized stack as prod** (`docker-compose.yml`),
|
The dev stack runs the **same containerized stack as prod** (`docker-compose.yml`),
|
||||||
overlaid with `docker-compose.dev.yml`: **uncapped** (no CPU limits, unlike prod's
|
overlaid with `docker-compose.dev.yml`: **uncapped** (no CPU limits, unlike
|
||||||
backend=4 / frontend=2 / db=2) and backend published on `0.0.0.0:8137` for the
|
prod's Swarm-managed replicas) and backend published on the WireGuard address
|
||||||
LAN (prod binds loopback behind Caddy; frontend has no published port here
|
(`10.10.0.2:8137` — set via `DEV_BIND_ADDR` in `deploy/env-topology.sh`; prod
|
||||||
either way, no Caddy to reach it directly, so it's only reached through
|
and beta bind loopback behind their own Caddy LBs instead). Frontend has no
|
||||||
backend's own reverse-proxy fallback). Bring it up by hand with `make dev-up`.
|
published port either way — reached only through backend's own
|
||||||
|
reverse-proxy fallback. Bring dev up by hand with `make dev-up`, which is a
|
||||||
|
**laptop-local convenience** (uncapped, loopback-published) — not the dev
|
||||||
|
environment itself, which only ever runs on vps1.
|
||||||
|
|
||||||
## Why it's built this way
|
## Why it's built this way
|
||||||
|
|
||||||
|
|
@ -39,26 +51,40 @@ via the API (`POST .../pulls/{n}/merge`, `{"Do": "squash"}`). Nothing merges
|
||||||
a ready PR for you automatically just because checks pass, unlike GitHub's
|
a ready PR for you automatically just because checks pass, unlike GitHub's
|
||||||
old in-workflow `ci-cd.yml` (retired along with the rest of `.github/`).
|
old in-workflow `ci-cd.yml` (retired along with the rest of `.github/`).
|
||||||
|
|
||||||
|
**Dev moved off the desktop and onto vps1 so the desktop could retire from the
|
||||||
|
Thermograph estate entirely.** The desktop now hosts AI models and offers flex
|
||||||
|
Swarm-worker capacity — it is no longer "the LAN dev server" and no longer the
|
||||||
|
primary CI runner story. What dev keeps that beta and prod do not:
|
||||||
|
|
||||||
|
- It renders `dev.yaml` **alone**, never layering `common.yaml` (the fleet's
|
||||||
|
shared production credentials) — vps1 also runs Forgejo and its CI, and dev
|
||||||
|
runs whatever branch is in flight. See the long note in
|
||||||
|
`deploy/render-secrets.sh` and `deploy/secrets/README.md`.
|
||||||
|
- It is **mesh-only** — see above. A public VPS running unreviewed branches
|
||||||
|
must never be reachable from the open internet the way the old LAN box
|
||||||
|
(behind a home router, reachable only on the Wi-Fi) safely could be.
|
||||||
|
|
||||||
## Moving parts
|
## Moving parts
|
||||||
|
|
||||||
| File | Purpose |
|
| File | Purpose |
|
||||||
|------|---------|
|
|------|---------|
|
||||||
| `.forgejo/workflows/pr-build.yml` | required status check for PRs into `dev` (calls `build.yml`) |
|
| `.forgejo/workflows/pr-build.yml` | required status check for PRs into `dev` (calls `build.yml`) |
|
||||||
| `.forgejo/workflows/deploy-dev.yml` | build + deploy on pushes to `dev` (direct, or a PR merge) |
|
| `.forgejo/workflows/deploy.yml` | the single Deploy workflow — `dev` branch pushes SSH into vps1 and run `deploy-dev.sh` |
|
||||||
| `.forgejo/workflows/build.yml` | shared build gate: deps, backend tests, JS syntax check, boot/health check |
|
| `.forgejo/workflows/build.yml` | shared build gate: deps, backend tests, JS syntax check, boot/health check |
|
||||||
| `deploy/deploy-dev.sh` | pull `dev` into `~/thermograph-dev`, `docker compose up` the uncapped LAN stack, health check |
|
| `deploy/env-topology.sh` | source of truth: dev = vps1, `/opt/thermograph-dev`, branch `dev`, compose mode, mesh-only bind address |
|
||||||
|
| `deploy/deploy-dev.sh` | thin wrapper around `deploy.sh`: dev's compose overlay + secrets policy, then delegates to the shared pull/roll/health-check logic |
|
||||||
| `deploy/secrets/dev.yaml` | dev's SOPS vault — its own secrets and config, rendered to `/etc/thermograph.env` at deploy time (see "Config and secrets") |
|
| `deploy/secrets/dev.yaml` | dev's SOPS vault — its own secrets and config, rendered to `/etc/thermograph.env` at deploy time (see "Config and secrets") |
|
||||||
| `docker-compose.dev.yml` | dev overlay: no CPU limits, backend published on `0.0.0.0:8137` (frontend unpublished) |
|
| `docker-compose.dev.yml` | dev overlay: no CPU limits, backend published on the mesh address (frontend unpublished) |
|
||||||
| `deploy/provision-dev-lan.sh` | one-time sudo-free bootstrap of the checkout + runner |
|
| `deploy/provision-dev.sh` | one-time bootstrap of the checkout + vault marker on vps1 |
|
||||||
| `deploy/forgejo/register-lan-runner.sh` | registers the Forgejo Actions runner on this machine |
|
| `deploy/forgejo/register-lan-runner.sh` | registers the Forgejo Actions runner (name predates the vps1/vps2 split — see the script's own header) |
|
||||||
|
|
||||||
`deploy-dev.yml`'s `deploy` job runs on the `thermograph-lan` label — **not**
|
`deploy.yml`'s dev leg SSHes in with `VPS1_SSH_*` and runs
|
||||||
`[self-hosted, thermograph-lan]` (the array form is what the original GitHub
|
`/opt/thermograph-dev/infra/deploy/deploy-dev.sh` directly over SSH — the same
|
||||||
version used; GitHub implicitly tags every self-hosted runner `self-hosted`,
|
shape as the beta and prod legs, just a different host and script. There is no
|
||||||
but Forgejo's runner has only the labels it was explicitly registered with,
|
longer a `thermograph-lan`-labeled runner job doing host-native
|
||||||
so the array form is permanently unschedulable there — a real bug found and
|
`systemctl`/`docker-compose` calls from inside a CI job; the deploy happens the
|
||||||
fixed once, worth knowing about if a "deploy" job ever silently stops
|
same way for all three environments now (a `docker`-labeled runner opens an SSH
|
||||||
appearing in the Actions history again).
|
session to the target host and runs that environment's deploy script there).
|
||||||
|
|
||||||
`deploy-dev.sh`'s git auth (`GH_TOKEN`/`GITHUB_TOKEN`, Forgejo's per-job
|
`deploy-dev.sh`'s git auth (`GH_TOKEN`/`GITHUB_TOKEN`, Forgejo's per-job
|
||||||
token exposed under that name for compatibility) is scoped to `REPO_URL`'s
|
token exposed under that name for compatibility) is scoped to `REPO_URL`'s
|
||||||
|
|
@ -68,15 +94,25 @@ hardcoded host would make git silently skip the header (no error) and fall
|
||||||
through to whatever ambient credentials happen to be available, not fail
|
through to whatever ambient credentials happen to be available, not fail
|
||||||
loudly.
|
loudly.
|
||||||
|
|
||||||
## The self-hosted runner
|
## The Forgejo Actions runner
|
||||||
|
|
||||||
A Forgejo Actions **runner** (`forgejo-runner`) runs on this machine, labels
|
TODO(cutover): `deploy/forgejo/register-lan-runner.sh`'s own header comment
|
||||||
|
still describes "the desktop" as the canonical placement for this runner, and
|
||||||
|
its `docker`-labeled jobs are what actually execute `deploy.yml`'s SSH-based
|
||||||
|
dev/beta/prod deploy steps today. Whether the runner has actually moved to
|
||||||
|
vps1 (alongside Forgejo) or still runs on the desktop under the new topology
|
||||||
|
is not settled by anything read for this pass — that script is out of this
|
||||||
|
document's edit scope. Confirm where it actually lives before relying on the
|
||||||
|
description below.
|
||||||
|
|
||||||
|
A Forgejo Actions **runner** (`forgejo-runner`) serves this repo, labels
|
||||||
`docker` (containerized jobs, node:20-bookworm, with the host's docker.sock
|
`docker` (containerized jobs, node:20-bookworm, with the host's docker.sock
|
||||||
automounted in — `container.docker_host: automount` in
|
automounted in — `container.docker_host: automount` in
|
||||||
`~/forgejo-runner/config.yaml` — so `docker build`/`push` work without
|
`~/forgejo-runner/config.yaml` — so `docker build`/`push` work without
|
||||||
privileged Docker-in-Docker) and `thermograph-lan` (host-native jobs, for
|
privileged Docker-in-Docker) and `thermograph-lan` (a legacy label from when a
|
||||||
`deploy-dev.sh`'s systemctl/docker-compose calls). Installed under
|
deploy job ran host-native on the LAN box; nothing in the current `deploy.yml`
|
||||||
`~/forgejo-runner`, runs as the `forgejo-runner` systemd `--user` service.
|
schedules against it, since every environment now deploys over SSH from the
|
||||||
|
`docker`-labeled job instead).
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# status / logs
|
# status / logs
|
||||||
|
|
@ -87,62 +123,66 @@ journalctl --user -u forgejo-runner -f
|
||||||
bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
|
bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
|
||||||
```
|
```
|
||||||
|
|
||||||
`git.thermograph.org`'s public DNS resolves to beta's **public** IP, which
|
Registry pushes (`build-push.yml`) run against the runner host's own
|
||||||
the registry's mesh-only ACL (see `deploy/forgejo/README.md`) rejects for
|
automounted Docker daemon, so it's that **host's** own resolver that needs to
|
||||||
`/v2/` paths — `docker build-push.yml` pushes run against the *host's*
|
route `git.thermograph.org` over the mesh, not anything settable from a job
|
||||||
automounted daemon, so it's this **host's** own resolver that needs to route
|
container. If registry pushes ever start failing with an HTTP/TLS mismatch or
|
||||||
git.thermograph.org over the mesh, not anything settable from a job
|
a 403 on `/v2/`, check `getent hosts git.thermograph.org` on the runner host —
|
||||||
container. If registry pushes ever start failing with an HTTP/TLS mismatch
|
it needs to resolve to vps1's WireGuard IP (`10.10.0.2`), typically via a
|
||||||
or a 403 on `/v2/`, check `getent hosts git.thermograph.org` here — it needs
|
`/etc/hosts` line, not the public one (a non-issue if the runner is co-located
|
||||||
to resolve to beta's WireGuard IP (`10.10.0.2`), typically via a `/etc/hosts`
|
with Forgejo on vps1 itself, since `git.thermograph.org` then resolves to
|
||||||
line, not the public one.
|
itself either way).
|
||||||
|
|
||||||
## The LAN dev service (Docker Compose)
|
## The dev environment (Docker Compose, on vps1)
|
||||||
|
|
||||||
Runs as a Docker Compose stack, not a bare systemd unit — `docker ps --filter
|
Runs as a Docker Compose stack, not a bare systemd unit — `docker ps --filter
|
||||||
name=thermograph-dev` shows `thermograph-dev-backend-1`, `thermograph-dev-frontend-1`
|
name=thermograph-dev` on vps1 shows `thermograph-dev-backend-1`,
|
||||||
(repo-split Stage 4 split the single "app" container in two — frontend has no
|
`thermograph-dev-frontend-1` (frontend has no published port, reached only
|
||||||
published port, reached only through backend's own reverse-proxy fallback) and
|
through backend's own reverse-proxy fallback) and `thermograph-dev-db-1`
|
||||||
`thermograph-dev-db-1` (TimescaleDB). A legacy pre-container
|
(TimescaleDB, dev's **own** container — not the shared instance on vps2).
|
||||||
`thermograph-dev.service` systemd --user unit may still exist from before this
|
|
||||||
cutover; it's stale and `deploy-dev.sh` stops/disables it on every run — don't
|
|
||||||
trust `systemctl --user status thermograph-dev` as a liveness check, use `docker
|
|
||||||
ps` instead.
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker ps --filter name=thermograph-dev # is it up?
|
docker ps --filter name=thermograph-dev # is it up?
|
||||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml logs -f backend frontend # app logs (from ~/thermograph-dev)
|
docker compose -f docker-compose.yml -f docker-compose.dev.yml logs -f backend frontend # app logs (from /opt/thermograph-dev/infra)
|
||||||
```
|
```
|
||||||
|
|
||||||
Bootstrap (or rebuild) it by hand:
|
Bootstrap (or rebuild) it:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bash deploy/provision-dev-lan.sh
|
sudo bash deploy/provision-dev.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
The dedicated checkout at `~/thermograph-dev` is separate from your working tree,
|
replacing the retired `deploy/provision-dev-lan.sh`, which bootstrapped dev on
|
||||||
so a deploy never touches uncommitted edits in `~/Code/Thermograph`.
|
the operator's desktop as a sudo-free `systemd --user` stack. Dev is
|
||||||
|
provisioned like any other fleet environment now — a checkout at
|
||||||
|
`/opt/thermograph-dev`, root-owned, deployed over SSH by CI like beta and
|
||||||
|
prod. The dedicated checkout is separate from your own working tree, so a
|
||||||
|
deploy never touches uncommitted edits there.
|
||||||
|
|
||||||
## Config and secrets
|
## Config and secrets
|
||||||
|
|
||||||
Dev's configuration lives in the same SOPS vault as prod's and beta's —
|
Dev's configuration lives in the same SOPS vault as prod's and beta's —
|
||||||
`deploy/secrets/dev.yaml`, encrypted to the same age recipient, rendered to
|
`deploy/secrets/dev.yaml`, encrypted to the same age recipient, rendered to
|
||||||
`/etc/thermograph.env` by the same `render-secrets.sh` at deploy time. Changing a
|
`/etc/thermograph.env` by the same `render-secrets.sh` at deploy time. Changing a
|
||||||
dev value is `sops deploy/secrets/dev.yaml` → commit → deploy, exactly as for the
|
dev value is `sops deploy/secrets/dev.yaml` → commit → deploy, exactly as for
|
||||||
VPS boxes. **`deploy/secrets/README.md` is the reference**; what follows is only
|
beta and prod. **`deploy/secrets/README.md` is the reference**; what follows
|
||||||
what is specific to this machine.
|
is only what is specific to this environment.
|
||||||
|
|
||||||
Dev renders `dev.yaml` **alone**. It does *not* layer `common.yaml`, which is the
|
Dev renders `dev.yaml` **alone**. It does *not* layer `common.yaml`, which is the
|
||||||
fleet's shared production credential set — registry token, both S3 keypairs (one
|
fleet's shared production credential set — registry token, both S3 keypairs (one
|
||||||
read-write pair, on the bucket that also holds prod's backups), the VAPID private
|
read-write pair, on the bucket that also holds prod's backups), the VAPID private
|
||||||
key that signs push to real subscribers, the IndexNow key, the metrics token. This
|
key that signs push to real subscribers, the IndexNow key, the metrics token. This
|
||||||
box is the CI runner and runs unreviewed `dev`-branch code; a plaintext render of
|
box also runs Forgejo and its CI, and executes unreviewed `dev`-branch code; a
|
||||||
that set into `/etc/thermograph.env` would put all of it in every dev container's
|
plaintext render of that set into `/etc/thermograph.env` would put all of it in
|
||||||
environment. `deploy-dev.sh` exports `THERMOGRAPH_SECRETS_SKIP_COMMON=1` to prevent
|
every dev container's environment. `deploy-dev.sh` exports
|
||||||
it, and **aborts the deploy** if the renderer in the checkout cannot honour that.
|
`THERMOGRAPH_SECRETS_SKIP_COMMON=1` to prevent it, and **aborts the deploy** if
|
||||||
|
the renderer in the checkout cannot honour that. This is a property of the
|
||||||
|
`dev` **environment** (encoded in `deploy/env-topology.sh`'s `TG_SKIP_COMMON`),
|
||||||
|
not just of `deploy-dev.sh` — so it holds regardless of which entry point is
|
||||||
|
used to deploy it.
|
||||||
|
|
||||||
So dev holds its own `THERMOGRAPH_AUTH_SECRET` (its own, not prod's — the `daemon`
|
So dev holds its own `THERMOGRAPH_AUTH_SECRET` (its own, not prod's — the `daemon`
|
||||||
service refuses to start without one), its own `POSTGRES_PASSWORD`, a LAN
|
service refuses to start without one), its own `POSTGRES_PASSWORD`, a mesh-only
|
||||||
`THERMOGRAPH_BASE_URL`, `THERMOGRAPH_COOKIE_SECURE=0` (plain HTTP — the shared
|
`THERMOGRAPH_BASE_URL`, `THERMOGRAPH_COOKIE_SECURE=0` (plain HTTP — the shared
|
||||||
value is `1` and would silently break login here), and the non-secret config it
|
value is `1` and would silently break login here), and the non-secret config it
|
||||||
would otherwise have inherited. It holds **no** S3 keys, no VAPID keypair, no
|
would otherwise have inherited. It holds **no** S3 keys, no VAPID keypair, no
|
||||||
|
|
@ -150,20 +190,26 @@ IndexNow key, no Discord or mail credentials: the lake falls through to the
|
||||||
Open-Meteo archive without bucket creds, and the app generates and persists its own
|
Open-Meteo archive without bucket creds, and the app generates and persists its own
|
||||||
VAPID and IndexNow keys into the `appdata` volume.
|
VAPID and IndexNow keys into the `appdata` volume.
|
||||||
|
|
||||||
Two dev-only mechanics, both because this box has **no passwordless sudo** and the
|
Two dev-only mechanics carried over from the old LAN setup, both worth keeping
|
||||||
runner is a `systemd --user` service with no tty:
|
even though dev is no longer sudo-free (vps1's `agent` user has passwordless
|
||||||
|
sudo like any other fleet host) — the render path just doesn't strictly need
|
||||||
|
them any more:
|
||||||
|
|
||||||
- the age key is read from `~/.config/sops/age/keys.txt`, not
|
- the age key can be read from `~/.config/sops/age/keys.txt` as a fallback
|
||||||
`/etc/thermograph/age.key` — a root-owned `0400` key would send the renderer down
|
when `/etc/thermograph/age.key` isn't present or readable — see
|
||||||
its `sudo cat` path, where it would hang on a prompt no CI job can answer;
|
`deploy-dev.sh`'s own comment on this;
|
||||||
- `/etc/thermograph.env` is **pre-created** owned by the deploying user, so the
|
- `/etc/thermograph.env` being pre-created and owned by the deploying user
|
||||||
renderer takes its in-place write path and no deploy needs `sudo` at all.
|
lets the renderer take its in-place write path without needing `sudo` at
|
||||||
|
all, which still matters under the Forgejo runner's `systemd --user`
|
||||||
|
context (no tty to answer a `sudo` prompt).
|
||||||
|
|
||||||
Setting this up on a fresh dev machine is four typed commands — see **"Setting up a
|
Setting this up on a fresh dev host is a few typed commands — see **"Setting up a
|
||||||
new dev machine"** in `deploy/secrets/README.md`. Until the
|
new dev machine"** in `deploy/secrets/README.md` (written for the old desktop
|
||||||
`/etc/thermograph/secrets-env` marker exists the box renders nothing and falls back
|
setup; the mechanics are the same on vps1, just root-owned rather than
|
||||||
to `deploy-dev.sh`'s built-in `POSTGRES_PASSWORD` default, which is the safe state
|
sudo-free). Until the `/etc/thermograph/secrets-env` marker exists the box
|
||||||
and the one to return to (delete the marker) if the vault ever gets in the way.
|
renders nothing and falls back to `deploy-dev.sh`'s built-in
|
||||||
|
`POSTGRES_PASSWORD` default, which is the safe state and the one to return to
|
||||||
|
(delete the marker) if the vault ever gets in the way.
|
||||||
|
|
||||||
`REGISTRY_TOKEN` is deliberately not in the vault: dev pulls with the persistent
|
`REGISTRY_TOKEN` is deliberately not in the vault: dev pulls with the persistent
|
||||||
`docker login` credential already in this host's docker config, like the prod/beta
|
`docker login` credential already in this host's docker config, like the prod/beta
|
||||||
|
|
@ -196,25 +242,28 @@ with `{"Do": "squash", "delete_branch_after_merge": true}`.
|
||||||
## Day-to-day
|
## Day-to-day
|
||||||
|
|
||||||
- **Ship:** open a PR into `dev`, confirm CI is green, merge it explicitly —
|
- **Ship:** open a PR into `dev`, confirm CI is green, merge it explicitly —
|
||||||
the merge triggers the deploy here.
|
the merge triggers a deploy to vps1.
|
||||||
- **Manual redeploy:** trigger `deploy-dev.yml` via `workflow_dispatch`
|
- **Manual redeploy:** trigger the `Deploy` workflow via `workflow_dispatch`
|
||||||
(`POST /api/v1/repos/{owner}/{repo}/actions/workflows/deploy-dev.yml/dispatches`,
|
(`POST /api/v1/repos/{owner}/{repo}/actions/workflows/deploy.yml/dispatches`,
|
||||||
`{"ref": "dev"}`), or locally `APP_DIR=~/thermograph-dev bash deploy/deploy-dev.sh`.
|
`{"ref": "dev"}`), or directly on vps1:
|
||||||
- **Promote to prod:** merge/fast-forward `dev` into `main` and push — that
|
`APP_DIR=/opt/thermograph-dev bash deploy/deploy-dev.sh`.
|
||||||
triggers the VPS pipeline in `DEPLOY.md`.
|
- **Promote to beta:** merge/fast-forward `dev` into `main` and push — that
|
||||||
|
triggers the beta leg of the same `Deploy` workflow, per `DEPLOY.md`.
|
||||||
|
|
||||||
## Monitoring
|
## Monitoring
|
||||||
|
|
||||||
Logs and dashboards moved to a **separate project** — a central Grafana + Loki
|
Logs and dashboards live in a **separate project** — a central Grafana + Loki
|
||||||
stack fed by a Grafana Alloy agent on every node (including this LAN dev box),
|
stack, hosted on **vps1 itself**, fed by a Grafana Alloy agent on every node
|
||||||
over the WireGuard mesh (`thermograph-observability` on Forgejo; UI at
|
(prod and beta on vps2, dev and the desktop's own agent, all over the
|
||||||
`https://dashboard.thermograph.org`). The dev node's Alloy agent ships its
|
WireGuard mesh; `thermograph-observability` on Forgejo; UI at
|
||||||
container/app logs under `host="dev"`, so LAN dev shows up alongside prod and
|
`https://dashboard.thermograph.org`). Dev's Alloy agent ships its
|
||||||
beta in the same dashboards. This replaced the old `scripts/dashboard.py`.
|
container/app logs under its own `host`/service labels, so dev shows up
|
||||||
|
alongside prod and beta in the same dashboards. This replaced the old
|
||||||
|
`scripts/dashboard.py`.
|
||||||
|
|
||||||
For a quick terminal check without Grafana, the app still serves raw counters at
|
For a quick terminal check without Grafana, the app still serves raw counters at
|
||||||
the gated `GET /api/v2/metrics` route. Dev sets no `THERMOGRAPH_METRICS_TOKEN`, so
|
the gated `GET /api/v2/metrics` route. Dev sets no `THERMOGRAPH_METRICS_TOKEN`, so
|
||||||
the route is direct-loopback-only — `curl localhost:8137/api/v2/metrics` from this
|
the route is direct-loopback-only — `curl localhost:8137/api/v2/metrics` from vps1
|
||||||
machine, which is where you already are. (A token exists on the VPS boxes to allow
|
itself. (A token exists on prod/beta to allow reads through an SSH tunnel; dev has
|
||||||
reads through an SSH tunnel; dev has no reason to carry a copy, and the estate's
|
no reason to carry a copy, and the estate's copy must not land here — see
|
||||||
copy must not land here — see `deploy/secrets/README.md`.)
|
`deploy/secrets/README.md`.)
|
||||||
|
|
|
||||||
188
infra/DEPLOY.md
188
infra/DEPLOY.md
|
|
@ -1,62 +1,71 @@
|
||||||
# Deploying Thermograph to a prod VPS
|
# Deploying Thermograph to prod (and beta) on vps2
|
||||||
|
|
||||||
> **Prod and beta are now provisioned by Terraform** (`terraform/README.md`) and
|
> **Prod and beta both run as Docker Swarm stacks on vps2** (`169.58.46.181`,
|
||||||
> run as a **`docker compose` stack** (backend + frontend + Postgres/TimescaleDB;
|
> mesh `10.10.0.1`) — two separate stacks, two separate checkouts
|
||||||
> repo-split Stage 4 split the single "app" service in two), not the
|
> (`/opt/thermograph` and `/opt/thermograph-beta`), sharing one host because a
|
||||||
> manual venv + systemd model this document originally described. Current shape:
|
> beta green light is meant to be evidence about prod (same orchestrator, same
|
||||||
> prod = new box `169.58.46.181`, branch `release`, deployed with `terraform
|
> Postgres build, same Caddy, same mesh position). `deploy/env-topology.sh` is
|
||||||
> apply`; beta = old box `75.119.132.91`, branch `main`, deployed when `main` is
|
> the single source of truth for which environment lives where; every path
|
||||||
> pushed (`.forgejo/workflows/deploy.yml` SSHes in and runs `deploy/deploy.sh`,
|
> below sources it rather than hardcoding a path or a port. The **manual
|
||||||
> which `docker compose pull`s the image `build-push.yml` already pushed for
|
> walkthrough below (`provision.sh`, `thermograph.service`, the venv) is
|
||||||
> that commit and `up`s it -- repo-split Stage 6's registry-pull cutover;
|
> legacy** — kept as a from-scratch, no-orchestrator reference; prod and beta's
|
||||||
> `deploy.sh` no longer builds in place). The **manual walkthrough below
|
> actual process model is a Swarm stack (`deploy/stack/`), and dev's is plain
|
||||||
> (`provision.sh`, `thermograph.service`, the venv) is legacy** — kept as a
|
> compose on vps1 (see `DEPLOY-DEV.md`), so the systemd/venv specifics below no
|
||||||
> from-scratch, no-Terraform reference; the process model is compose, so the
|
> longer match how any live environment actually runs.
|
||||||
> systemd/venv specifics no longer match how prod/beta actually run.
|
|
||||||
|
|
||||||
Pipeline (beta, on `main`): **push to `main` on Forgejo → `build-push.yml` builds
|
Pipeline, both environments: **push to `main` (beta) or `release` (prod) on
|
||||||
+ pushes the image, tagged by SHA → Forgejo Actions SSHes to beta →
|
Forgejo → `build-push.yml` builds + pushes the image, tagged by SHA → the
|
||||||
`deploy/deploy.sh` (`git reset` + `docker login` + `docker compose pull && up`,
|
single `Deploy` workflow (`.forgejo/workflows/deploy.yml`) SSHes into **vps2**
|
||||||
schema migration runs in the app entrypoint) → Caddy fronts it with Let's Encrypt
|
with `THERMOGRAPH_ENV=beta` or `THERMOGRAPH_ENV=prod` → `deploy/deploy.sh`
|
||||||
TLS.** Prod (on `release`) is deployed with `terraform apply`, not a push
|
resolves the right checkout/branch/stack from `env-topology.sh`, `git reset`s
|
||||||
trigger — its `remote-exec` provisioner does the same pull-instead-of-build.
|
it, renders that environment's secrets, and execs `deploy/stack/deploy-stack.sh`
|
||||||
(GitHub is retired/archived — Forgejo, self-hosted at `git.thermograph.org` /
|
(schema migration runs as a one-shot task before the roll) → each environment's
|
||||||
`10.10.0.2:3080` over the WireGuard mesh, is now the sole git host and CI.)
|
own loopback Caddy LB fronts it with TLS.** Credentials are keyed by **host**
|
||||||
|
(`VPS2_SSH_*`), not by environment — vps2 answers to both prod and beta, so the
|
||||||
|
environment is passed as data (`THERMOGRAPH_ENV`), not baked into which secret
|
||||||
|
is used. (GitHub is retired/archived — Forgejo, self-hosted at
|
||||||
|
`git.thermograph.org` on **vps1** over the WireGuard mesh, is now the sole git
|
||||||
|
host and CI.)
|
||||||
|
|
||||||
Files that make this work:
|
Files that make this work:
|
||||||
|
|
||||||
| File | Where it lives in prod | Purpose |
|
| File | Where it lives on vps2 | Purpose |
|
||||||
|------|------------------------|---------|
|
|------|------------------------|---------|
|
||||||
| `.forgejo/workflows/deploy.yml` | Forgejo | CI job that SSHes in and runs the deploy script (beta, on `main`) |
|
| `.forgejo/workflows/deploy.yml` | Forgejo | CI job that SSHes in and runs the deploy script, for all three environments |
|
||||||
| `.forgejo/workflows/build-push.yml` | Forgejo | builds + pushes the SHA-tagged image `deploy.sh`/Terraform pull |
|
| `.forgejo/workflows/build-push.yml` | Forgejo | builds + pushes the SHA-tagged image `deploy.sh` pulls |
|
||||||
| `deploy/deploy.sh` | `/opt/thermograph/deploy/deploy.sh` | `git reset` + `docker login` + `docker compose pull && up` + health check + warm |
|
| `deploy/env-topology.sh` | `/opt/thermograph{,-beta}/infra/deploy/env-topology.sh` | resolves `THERMOGRAPH_ENV` to a checkout, branch, stack name, ports, DB role |
|
||||||
| `docker-compose.yml` | `/opt/thermograph/docker-compose.yml` | app + Postgres/TimescaleDB services (the process model) |
|
| `deploy/deploy.sh` | `/opt/thermograph{,-beta}/infra/deploy/deploy.sh` | `git reset` + render secrets + routes to the Swarm-stack deploy |
|
||||||
| `deploy/thermograph.env.example` | `/etc/thermograph.env` | Postgres password, VAPID/auth secrets, `WORKERS`, sizing, base path |
|
| `deploy/stack/thermograph-stack.yml` / `thermograph-beta-stack.yml` | same | prod's / beta's Swarm stack (db, web, worker, lake, daemon, frontend; beta has no `db` of its own) |
|
||||||
| `deploy/Caddyfile` | `/etc/caddy/Caddyfile` | reverse proxy + automatic HTTPS |
|
| `deploy/thermograph.env.example` | `/etc/thermograph.env` (prod) / `/etc/thermograph-beta.env` (beta) | Postgres password, VAPID/auth secrets, `WORKERS`, sizing, base path |
|
||||||
| `terraform/` | run from your machine | provisions the box + brings the stack up (replaces `provision.sh`) |
|
| `deploy/stack/lb/Caddyfile` / `Caddyfile.beta` | each stack's loopback LB container | reverse proxy from the host's real Caddy into the Swarm overlay |
|
||||||
| `deploy/provision.sh`, `deploy/thermograph.service` | *(legacy)* | pre-compose venv+systemd bootstrap — superseded by Terraform |
|
| `terraform/` | run from your machine | provisions the box (host-level only; does not know about the two-environment split — see `ACCESS.md`) |
|
||||||
|
| `deploy/provision.sh`, `deploy/thermograph.service` | *(legacy)* | pre-compose venv+systemd bootstrap — superseded by the Swarm stack |
|
||||||
|
|
||||||
The app serves on **loopback only**; Caddy is the only thing exposed to the
|
Each app serves on **loopback only** (prod `127.0.0.1:8137`/`:8080`, beta
|
||||||
internet (ports 80/443). The parquet cache in `data/cache/` lives inside the
|
`127.0.0.1:8237`/`:8180`); the host's real Caddy is the only thing exposed to
|
||||||
`/opt/thermograph` checkout and is gitignored, so `git pull` never touches it.
|
the internet (ports 80/443), reverse-proxying into whichever loopback LB
|
||||||
|
matches the domain requested. Each environment's own checkout keeps its own
|
||||||
|
gitignored cache under `data/cache/`, so a `git pull` in one never touches the
|
||||||
|
other's.
|
||||||
|
|
||||||
**Any deploy host needs a `/etc/hosts` entry for the registry.**
|
**Any deploy host needs a `/etc/hosts` entry for the registry.**
|
||||||
`git.thermograph.org`'s public DNS resolves to beta's public IP, and beta's
|
`git.thermograph.org`'s public DNS resolves to **vps1's** public IP, and
|
||||||
own Caddy rejects `/v2/*` (the registry API) from anything outside the
|
vps1's own Caddy rejects `/v2/*` (the registry API) from anything outside the
|
||||||
WireGuard mesh (10.10.0.0/24) — confirmed live: `docker login`/`pull` from
|
WireGuard mesh (10.10.0.0/24) — confirmed live: `docker login`/`pull` from
|
||||||
prod failed with a 403 until adding `10.10.0.2 git.thermograph.org` to
|
vps2 failed with a 403 until adding `10.10.0.2 git.thermograph.org` to
|
||||||
`/etc/hosts` (10.10.0.2 is beta's mesh IP; every deploy host is already on
|
`/etc/hosts` (10.10.0.2 is vps1's mesh IP; every deploy host is already on
|
||||||
the mesh, this is purely a DNS-routing gap, not a connectivity one). Beta
|
the mesh, this is purely a DNS-routing gap, not a connectivity one). vps1
|
||||||
itself doesn't need this — it's the box Forgejo runs on, so `git.thermograph.org`
|
itself doesn't need this — it's the box Forgejo runs on, so `git.thermograph.org`
|
||||||
just resolves to itself either way. `build-push.yml`'s own header comment
|
just resolves to itself either way. `build-push.yml`'s own header comment
|
||||||
documents the same requirement for the CI runner host.
|
documents the same requirement for the CI runner host.
|
||||||
|
|
||||||
**Current production layout** (`deploy/Caddyfile`): the app owns
|
**Current production layout**: prod's Caddy config owns **`thermograph.org`**
|
||||||
**`thermograph.org`** at its root (`THERMOGRAPH_BASE=/`, so uvicorn serves `/`,
|
at its root (`THERMOGRAPH_BASE=/`, so uvicorn serves `/`, `/calendar`,
|
||||||
`/calendar`, `/api/v2/…` with no prefix), and **`emigriffith.dev`** serves a
|
`/api/v2/…` with no prefix) and beta's owns **`beta.thermograph.org`** the same
|
||||||
static portfolio at its root with `emigriffith.dev/thermograph*` permanently
|
way. **`emigriffith.dev`** is a separate static portfolio site, served from
|
||||||
redirecting to `thermograph.org` (prefix stripped). Both domains' `A` records
|
**vps1**, not vps2 — it shares no host, checkout or Caddy config with either
|
||||||
point at the same VPS; Caddy provisions a separate cert for each.
|
app environment. `emigriffith.dev/thermograph*` permanently redirects to
|
||||||
|
`thermograph.org` (prefix stripped).
|
||||||
|
|
||||||
> **Note:** `deploy.sh` (the CI deploy) only pulls code, reinstalls deps, and
|
> **Note:** `deploy.sh` (the CI deploy) only pulls code, reinstalls deps, and
|
||||||
> restarts the app service — it does **not** touch Caddy or `/etc/thermograph.env`.
|
> restarts the app service — it does **not** touch Caddy or `/etc/thermograph.env`.
|
||||||
|
|
@ -202,24 +211,32 @@ If the repo is **public**, skip this — `git pull` needs no auth.
|
||||||
## Accounts & notifications
|
## Accounts & notifications
|
||||||
|
|
||||||
Accounts, subscriptions, and notifications are **authoritative and not
|
Accounts, subscriptions, and notifications are **authoritative and not
|
||||||
regenerable** — back them up. On **prod/beta they live in Postgres/TimescaleDB**
|
regenerable** — back them up. **Prod, beta and dev all keep them in
|
||||||
(the compose `db` service, on the `pgdata` volume) alongside the climate record;
|
Postgres/TimescaleDB** — prod and beta on the one shared instance on vps2 (as
|
||||||
on **LAN dev** they fall back to SQLite (`data/accounts.sqlite`) when
|
the `thermograph` and `thermograph_beta` databases, separate roles, `CONNECT`
|
||||||
`THERMOGRAPH_DATABASE_URL` isn't a Postgres URL.
|
revoked from `PUBLIC` — see `deploy/db/provision-env-db.sh`), dev in its own
|
||||||
|
`db` container on vps1. SQLite (`data/accounts.sqlite`) is only a fallback for
|
||||||
|
running the app **outside** any of these stacks (a bare venv, the test suite)
|
||||||
|
when `THERMOGRAPH_DATABASE_URL` isn't a Postgres URL — it's not what any live
|
||||||
|
environment actually uses.
|
||||||
|
|
||||||
- **Back up the Postgres volume** (e.g. `pg_dump` the `thermograph` database) as
|
- **Back up the Postgres data** (e.g. `pg_dump` the relevant database) as part
|
||||||
part of the VPS backup routine — losing it wipes all accounts and alerts. (On
|
of each environment's backup routine — losing it wipes all accounts and
|
||||||
a SQLite LAN dev box, back up `data/accounts.sqlite` and its `-wal`/`-shm`
|
alerts for that environment. (Only a bare-venv/offline run backs up
|
||||||
sidecars instead.)
|
`data/accounts.sqlite` and its `-wal`/`-shm` sidecars instead.)
|
||||||
- **Multiple workers, one notifier (leader election).** Prod runs several uvicorn
|
- **Multiple workers, one notifier (leader election).** Each environment runs
|
||||||
workers (`WORKERS`, currently 8 on prod / 4 on beta). The subscription
|
several uvicorn workers (`WORKERS`, sized per environment — see
|
||||||
evaluator and the recurring scheduler must run in exactly one, so the workers
|
`deploy/secrets/{prod,beta,dev}.yaml` and each stack file's replica/worker
|
||||||
elect a single leader — a host-local `flock` (`THERMOGRAPH_SINGLETON_LOCK`,
|
counts). The subscription evaluator and the recurring scheduler must run in
|
||||||
set by the compose app service to `/app/data/notifier.lock`), or a cluster-wide
|
exactly one, so the workers elect a single leader — a host-local `flock`
|
||||||
Postgres advisory lock (`THERMOGRAPH_SINGLETON_PG=1`) if the notifier must be
|
(`THERMOGRAPH_SINGLETON_LOCK`, set by the compose/stack app service to
|
||||||
single across *hosts*. See `backend/core/singleton.py`. `THERMOGRAPH_ROLE`
|
`/app/data/notifier.lock`), or a cluster-wide Postgres advisory lock
|
||||||
(`web`/`worker`/`all`, default `all`) further restricts which processes may own
|
(`THERMOGRAPH_SINGLETON_PG=1`) if the notifier must be single across
|
||||||
it, so a stateless web tier can scale without scaling notifiers.
|
*replicas on multiple hosts* — this is what prod's Swarm stack uses, since
|
||||||
|
`web` scales to N replicas under the autoscaler. See
|
||||||
|
`backend/core/singleton.py`. `THERMOGRAPH_ROLE` (`web`/`worker`/`all`,
|
||||||
|
default `all`) further restricts which processes may own it, so a stateless
|
||||||
|
web tier can scale without scaling notifiers.
|
||||||
- **Env vars** (all optional, sensible defaults):
|
- **Env vars** (all optional, sensible defaults):
|
||||||
- `THERMOGRAPH_COOKIE_SECURE` — set to `1` when serving over HTTPS (the VPS/TLS
|
- `THERMOGRAPH_COOKIE_SECURE` — set to `1` when serving over HTTPS (the VPS/TLS
|
||||||
deploy) so the session cookie is `Secure`. Leave unset for plain-HTTP LAN, where
|
deploy) so the session cookie is `Secure`. Leave unset for plain-HTTP LAN, where
|
||||||
|
|
@ -296,9 +313,11 @@ Why route through a local MTA rather than a provider's API:
|
||||||
`ReadWritePaths=…`) needs no change.
|
`ReadWritePaths=…`) needs no change.
|
||||||
|
|
||||||
**Default is safe:** `THERMOGRAPH_MAIL_BACKEND` defaults to `console` — it logs
|
**Default is safe:** `THERMOGRAPH_MAIL_BACKEND` defaults to `console` — it logs
|
||||||
the message and sends nothing — so LAN dev and the test suite exercise the whole
|
the message and sends nothing — so dev and the test suite exercise the whole
|
||||||
signup path with no mail server and no risk of mailing a real person. Production
|
signup path with no mail server and no risk of mailing a real person. Prod
|
||||||
opts in with `=smtp`.
|
opts in with `=smtp`; beta stays on `console` too (see
|
||||||
|
`deploy/stack/thermograph-beta-stack.yml`'s header for why beta must never
|
||||||
|
hold live mail or Discord credentials).
|
||||||
|
|
||||||
### Postfix unit topology — and the check that lies
|
### Postfix unit topology — and the check that lies
|
||||||
|
|
||||||
|
|
@ -431,8 +450,8 @@ Check placement with <https://www.mail-tester.com>, which scores all four at onc
|
||||||
confirmation or password-reset link is mailed. It currently defaults to a
|
confirmation or password-reset link is mailed. It currently defaults to a
|
||||||
per-boot random value, so every outstanding link would break on restart.
|
per-boot random value, so every outstanding link would break on restart.
|
||||||
- **Digest signups** land in the `pending_digest` table in the accounts DB
|
- **Digest signups** land in the `pending_digest` table in the accounts DB
|
||||||
(Postgres on prod/beta; authoritative, back it up). The form ships ahead of
|
(Postgres on every environment; authoritative, back it up). The form ships
|
||||||
delivery on purpose:
|
ahead of delivery on purpose:
|
||||||
addresses are collected now and confirmed once SMTP is live.
|
addresses are collected now and confirmed once SMTP is live.
|
||||||
- **Logs:** `journalctl -u postfix -f`; queue with `mailq`.
|
- **Logs:** `journalctl -u postfix -f`; queue with `mailq`.
|
||||||
|
|
||||||
|
|
@ -464,26 +483,33 @@ regenerated with `python gen_cities.py [n_global] [n_english] [n_extended]`.
|
||||||
proved unreliable (wrong matches), so they're edited by hand; a city not in the list
|
proved unreliable (wrong matches), so they're edited by hand; a city not in the list
|
||||||
simply shows its flavor blurb. Add entries freely — a test checks the slugs are valid.
|
simply shows its flavor blurb. Add entries freely — a test checks the slugs are valid.
|
||||||
|
|
||||||
- **Archive warming runs automatically on every deploy.** `deploy.sh` (prod) and
|
- **Archive warming runs automatically on every deploy of prod and dev — not
|
||||||
`deploy-dev.sh` (dev) launch `warm_cities.py` detached in the background after the
|
beta.** `deploy/stack/deploy-stack.sh` (prod) and `deploy.sh`'s compose path
|
||||||
health check, so the ~750 city pages serve from cache and a crawl can't burst the
|
(dev, via `deploy-dev.sh`) launch `warm_cities.py` detached in the background
|
||||||
archive API quota. It's idempotent (skips already-cached cells), so only the first
|
after the health check, so the ~750 city pages serve from cache and a crawl
|
||||||
deploy does the full ~25-min warm; later deploys just top up new cities. A page
|
can't burst the archive API quota. Beta deliberately skips this
|
||||||
also self-heals (fetches its archive once) if hit before warming finishes. To run
|
(`TG_POST_DEPLOY=0` in `env-topology.sh`) — it would spend the shared
|
||||||
it by hand: `cd backend && python warm_cities.py --pace 2`. Logs: prod
|
upstream archive quota filling a cache only a rehearsal environment reads.
|
||||||
`logs/warm-cities.log`; dev `journalctl --user -u thermograph-warm-cities`.
|
Warming is idempotent (skips already-cached cells), so only the first deploy
|
||||||
|
does the full ~25-min warm; later deploys just top up new cities. A page
|
||||||
|
also self-heals (fetches its archive once) if hit before warming finishes. To
|
||||||
|
run it by hand: `cd backend && python warm_cities.py --pace 2`. Logs land at
|
||||||
|
`logs/warm-cities.log` inside each environment's own backend container (dev
|
||||||
|
is a normal container stack on vps1 now, not a bare systemd unit; there's no
|
||||||
|
`journalctl --user` path any more).
|
||||||
- **Submit the sitemap** in Google Search Console: `https://thermograph.org/sitemap.xml`.
|
- **Submit the sitemap** in Google Search Console: `https://thermograph.org/sitemap.xml`.
|
||||||
- `jinja2` is a new dependency (already in `requirements.txt`).
|
- `jinja2` is a new dependency (already in `requirements.txt`).
|
||||||
|
|
||||||
## Monitoring
|
## Monitoring
|
||||||
|
|
||||||
Fleet-wide logs and dashboards live in a **separate project** — a central
|
Fleet-wide logs and dashboards live in a **separate project** — a central
|
||||||
Grafana + Loki stack fed by a Grafana Alloy agent on every node, over the
|
Grafana + Loki stack, hosted on **vps1** alongside Forgejo, fed by a Grafana
|
||||||
WireGuard mesh (`thermograph-observability` on Forgejo; UI at
|
Alloy agent on every node over the WireGuard mesh (`thermograph-observability`
|
||||||
**`https://dashboard.thermograph.org`**, Google SSO). It ingests every
|
on Forgejo; UI at **`https://dashboard.thermograph.org`**, Google SSO). It
|
||||||
container's stdout, Caddy's access logs, and the app's structured JSON logs
|
ingests every container's stdout, Caddy's access logs, and the app's
|
||||||
(`logs/{errors,access,audit}/*.jsonl`). This replaced the old SSH-tailed
|
structured JSON logs (`logs/{errors,access,audit}/*.jsonl`) — from prod and
|
||||||
`scripts/dashboard.py`.
|
beta on vps2 as well as dev on vps1, each distinguishable by its `host`/service
|
||||||
|
labels. This replaced the old SSH-tailed `scripts/dashboard.py`.
|
||||||
|
|
||||||
The app still exposes raw counters at the gated `GET /api/v2/metrics` route
|
The app still exposes raw counters at the gated `GET /api/v2/metrics` route
|
||||||
(loopback-only; refuses any request carrying a proxy `X-Forwarded-*` header, so
|
(loopback-only; refuses any request carrying a proxy `X-Forwarded-*` header, so
|
||||||
|
|
|
||||||
|
|
@ -30,8 +30,14 @@ db-up:
|
||||||
db-down:
|
db-down:
|
||||||
docker compose stop db
|
docker compose stop db
|
||||||
|
|
||||||
# The LAN dev stack: uncapped (no CPU limits) and published on 0.0.0.0:8137 so
|
# The dev overlay: uncapped (no CPU limits). Default publish is 0.0.0.0:8137,
|
||||||
# phones on the Wi-Fi can reach it (the base file is prod: capped + loopback).
|
# right for a laptop-local rehearsal (e.g. phones on the same Wi-Fi) -- this is
|
||||||
|
# also the *fleet* dev environment's own overlay, but that environment now runs
|
||||||
|
# on vps1 (a public VPS, not a LAN box) and deploys via deploy/deploy-dev.sh,
|
||||||
|
# which overrides the bind address to the WireGuard mesh IP only
|
||||||
|
# (DEV_BIND_ADDR, see deploy/env-topology.sh) -- never invoke `make dev-up`
|
||||||
|
# directly on a fleet host, or it publishes on every interface. (The base file
|
||||||
|
# is prod/beta's shape: capped + loopback, fronted by their own Caddy LB.)
|
||||||
DEV_COMPOSE = docker compose -f docker-compose.yml -f docker-compose.dev.yml
|
DEV_COMPOSE = docker compose -f docker-compose.yml -f docker-compose.dev.yml
|
||||||
dev-up:
|
dev-up:
|
||||||
POSTGRES_PASSWORD=$${POSTGRES_PASSWORD:-thermograph-dev} $(DEV_COMPOSE) up -d --build
|
POSTGRES_PASSWORD=$${POSTGRES_PASSWORD:-thermograph-dev} $(DEV_COMPOSE) up -d --build
|
||||||
|
|
|
||||||
|
|
@ -3,8 +3,9 @@
|
||||||
Infrastructure for [Thermograph](https://thermograph.org): Terraform host
|
Infrastructure for [Thermograph](https://thermograph.org): Terraform host
|
||||||
provisioning, the SOPS+age secrets vault, WireGuard/Swarm networking, Forgejo,
|
provisioning, the SOPS+age secrets vault, WireGuard/Swarm networking, Forgejo,
|
||||||
Caddy, mail, and the deploy scripts that run the already-built app images on each
|
Caddy, mail, and the deploy scripts that run the already-built app images on each
|
||||||
host. This is a domain of the `emi/thermograph` monorepo — hosts' `/opt/thermograph`
|
host. This is a domain of the `emi/thermograph` monorepo — a host's checkout
|
||||||
is a checkout of the whole monorepo, and `infra/` never builds app source; it only
|
(`/opt/thermograph`, `/opt/thermograph-beta`, or `/opt/thermograph-dev`) is a
|
||||||
|
checkout of the whole monorepo, and `infra/` never builds app source; it only
|
||||||
runs published images.
|
runs published images.
|
||||||
|
|
||||||
- **`terraform/`** — provisions/configures hosts (SSH-driven by default; an
|
- **`terraform/`** — provisions/configures hosts (SSH-driven by default; an
|
||||||
|
|
@ -13,37 +14,61 @@ runs published images.
|
||||||
executable documentation, not a routine operation.
|
executable documentation, not a routine operation.
|
||||||
- **`deploy/secrets/`** — the git-native SOPS+age secrets vault (every app secret,
|
- **`deploy/secrets/`** — the git-native SOPS+age secrets vault (every app secret,
|
||||||
encrypted at rest, rendered at deploy time). See `deploy/secrets/README.md`.
|
encrypted at rest, rendered at deploy time). See `deploy/secrets/README.md`.
|
||||||
- **`deploy/swarm/`, `deploy/forgejo/`** — the WireGuard/Swarm cluster hosting
|
- **`deploy/swarm/`, `deploy/forgejo/`** — the WireGuard/Swarm mesh spanning
|
||||||
Forgejo (git + CI + registry). See `ACCESS.md`.
|
vps1, vps2 and the desktop, and Forgejo (git + CI + registry), pinned to
|
||||||
- **`deploy/deploy.sh`** — the single deploy entry point for beta and prod.
|
vps1. See `ACCESS.md`.
|
||||||
Takes `SERVICE=backend|frontend|all` plus `BACKEND_IMAGE_TAG`/`FRONTEND_IMAGE_TAG`,
|
- **`deploy/env-topology.sh`** — the single source of truth for where each
|
||||||
resets the host checkout, renders secrets, and routes to the right orchestrator.
|
environment (`dev`/`beta`/`prod`) lives: host, checkout path, branch, deploy
|
||||||
- **`deploy/stack/`** — the **Swarm** path, live on **prod**:
|
mode, stack/compose name, env file, LB ports, DB role/database, service-name
|
||||||
`thermograph-stack.yml` (db, web, worker, lake, daemon, frontend, autoscaler,
|
prefix. Every deploy path sources it and derives its behavior from it rather
|
||||||
autoscaler-lake), `deploy-stack.sh`, `autoscale.sh`, and the LB. Rolling updates
|
than guessing from the host it happens to be running on — necessary since
|
||||||
are start-first, health-gated, with auto-rollback. `STACK_TEST=1` rehearses the
|
vps2 alone now runs two environments.
|
||||||
whole stack on throwaway volumes and ports.
|
- **`deploy/deploy.sh`** — the single deploy entry point for dev, beta and prod.
|
||||||
- **`docker-compose*.yml`** — the **compose** path, live on **beta** and LAN dev
|
Takes `SERVICE=backend|frontend|all` plus `BACKEND_IMAGE_TAG`/`FRONTEND_IMAGE_TAG`
|
||||||
(db, backend, lake, daemon, frontend). `docker-compose.dev.yml` is the LAN
|
(and, on vps2, `THERMOGRAPH_ENV=beta|prod` to say which of the two checkouts
|
||||||
overlay; `docker-compose.openmeteo.yml` is the self-hosted Open-Meteo overlay.
|
it's acting on), resets the host checkout, renders secrets, and routes to the
|
||||||
|
right orchestrator per `env-topology.sh`.
|
||||||
|
- **`deploy/deploy-dev.sh`** — a thin dev-specific wrapper around `deploy.sh`
|
||||||
|
(dev compose overlay, dev's secrets policy). See `DEPLOY-DEV.md`.
|
||||||
|
- **`deploy/stack/`** — the **Swarm** path, live on **vps2** for both prod and
|
||||||
|
beta: `thermograph-stack.yml` (prod: db, web, worker, lake, daemon, frontend,
|
||||||
|
autoscaler, autoscaler-lake) and `thermograph-beta-stack.yml` (beta: the same
|
||||||
|
service shape minus `db` and the autoscalers, every service name prefixed
|
||||||
|
`beta-`). `deploy-stack.sh`, `autoscale.sh` and `lb/` are shared by both.
|
||||||
|
Rolling updates are start-first, health-gated, with auto-rollback.
|
||||||
|
`STACK_TEST=1` rehearses the whole stack on throwaway volumes and ports.
|
||||||
|
- **`docker-compose*.yml`** — the **compose** path, live only on **dev** (vps1)
|
||||||
|
(db, backend, lake, daemon, frontend). `docker-compose.dev.yml` is dev's
|
||||||
|
mesh-only overlay; `docker-compose.openmeteo.yml` is the self-hosted
|
||||||
|
Open-Meteo overlay (prod only). `make dev-up` also runs this path locally as
|
||||||
|
a laptop convenience — that is not an "environment", just a local rehearsal.
|
||||||
|
|
||||||
Which path a host takes is decided by `/etc/thermograph/deploy-mode`: the string
|
Which path an environment takes is decided by `deploy/env-topology.sh`
|
||||||
`stack` makes `deploy.sh` exec `deploy/stack/deploy-stack.sh`; anything else is
|
(`TG_DEPLOY_MODE`, keyed by `dev`/`beta`/`prod`): dev is `compose`, beta and prod
|
||||||
compose. The workflows never need to know which mode a host runs.
|
are both `stack`. The old host-wide marker `/etc/thermograph/deploy-mode` still
|
||||||
|
exists as a fallback for a by-hand run with no explicit environment, but it
|
||||||
|
cannot describe vps2, which runs two environments in two different checkouts —
|
||||||
|
so it is no longer the thing that decides where files go. The workflows never
|
||||||
|
need to know which mode an environment runs; they only pass `THERMOGRAPH_ENV`.
|
||||||
|
|
||||||
## Branches & how changes reach each environment
|
## Branches & how changes reach each environment
|
||||||
|
|
||||||
- **`main`** — what **prod and beta** run. `infra-sync.yml` fires on a push to
|
- **`dev`** — deploys to **dev on vps1** (`/opt/thermograph-dev`).
|
||||||
`main` touching `infra/**`, fast-forwards each host's `/opt/thermograph` checkout
|
- **`main`** — deploys to **beta on vps2** (`/opt/thermograph-beta`).
|
||||||
and re-renders `/etc/thermograph.env` from the vault. It deliberately does
|
- **`release`** — deploys to **prod on vps2** (`/opt/thermograph`).
|
||||||
**not** roll any service: image tags are the app domains' axis, not infra's. A
|
|
||||||
compose or stack change that must recreate containers takes effect on the next
|
|
||||||
app deploy, or a by-hand `SERVICE=all … deploy/deploy.sh`.
|
|
||||||
- **`dev`** — what LAN dev would run via `deploy/deploy-dev.sh`. The CI trigger
|
|
||||||
for this is currently inert (the LAN box still holds a split-era checkout); use
|
|
||||||
`make dev-up` locally.
|
|
||||||
- **`release`** — consumed by app deploys only. Both hosts track infra via `main`;
|
|
||||||
prod's *app images* are staged by `release`, but its checkout follows `main`.
|
|
||||||
|
|
||||||
Note the asymmetry with the app domains: app code IS environment-staged
|
App code IS environment-staged this way (`dev`→`main`→`release` maps to
|
||||||
(`dev`→`main`→`release` maps to LAN→beta→prod via image tags); infra is not.
|
vps1/dev → vps2/beta → vps2/prod via image tags, one `Deploy` workflow keyed by
|
||||||
|
branch). **Infra itself is not environment-staged** the same way: `infra-sync.yml`
|
||||||
|
fires on a push touching `infra/**` — on `dev` it fast-forwards vps1's dev
|
||||||
|
checkout, on `main` it fast-forwards **both** of vps2's checkouts (beta and
|
||||||
|
prod) — re-rendering each environment's own env file from the vault. It
|
||||||
|
deliberately does **not** roll any service: image tags are the app domains'
|
||||||
|
axis, not infra's. A compose or stack change that must recreate containers
|
||||||
|
takes effect on the next app deploy, or a by-hand `SERVICE=all …
|
||||||
|
deploy/deploy.sh` (or `deploy-dev.sh`) run.
|
||||||
|
|
||||||
|
Note the asymmetry this leaves: dev's infra checkout tracks `dev`, the same
|
||||||
|
branch its app images are staged by. Beta's and prod's infra checkouts both
|
||||||
|
track `main` — prod's *app images* are staged by `release`, but prod's *infra
|
||||||
|
checkout* follows `main`, same as beta's.
|
||||||
|
|
|
||||||
|
|
@ -1,120 +0,0 @@
|
||||||
# /etc/caddy/Caddyfile on the VPS.
|
|
||||||
# Each domain's A/AAAA record must already point at this VPS — Caddy provisions a
|
|
||||||
# Let's Encrypt cert on first request and auto-renews. Nothing else to do for TLS
|
|
||||||
# (just make sure ports 80 and 443 are open).
|
|
||||||
#
|
|
||||||
# Layout:
|
|
||||||
# thermograph.org/* -> the Thermograph app (path-split across
|
|
||||||
# backend/frontend -- see below)
|
|
||||||
# emigriffith.dev/ -> static portfolio site (served straight from disk)
|
|
||||||
# emigriffith.dev/thermograph* -> permanent redirect to thermograph.org (the app moved)
|
|
||||||
#
|
|
||||||
# Thermograph now owns thermograph.org's root, so both services run with
|
|
||||||
# THERMOGRAPH_BASE=/ (see /etc/thermograph.env) — pages, assets and API all sit
|
|
||||||
# at "/" with no sub-path prefix. Repo-split Stage 4: backend and frontend are
|
|
||||||
# two containers now (docker-compose.yml), each on its own loopback port --
|
|
||||||
# Caddy path-splits directly to whichever owns a given path, so the browser
|
|
||||||
# still sees one apparent origin. Repo-split Stage 7a flipped which side owns
|
|
||||||
# the enumerated list: frontend now owns everything (content pages, the
|
|
||||||
# interactive tool's SPA shells, every static asset, the dynamic IndexNow key
|
|
||||||
# file) except the short, stable set below, which mirrors backend/web/app.py's
|
|
||||||
# own routing exactly (a single catch-all proxy to frontend for everything
|
|
||||||
# else) -- unlike frontend's paths, backend's don't grow every time a new
|
|
||||||
# static asset filename is added. A gap in this list still just degrades to
|
|
||||||
# "one extra hop" through backend's own proxy fallback, never a 404.
|
|
||||||
|
|
||||||
thermograph.org {
|
|
||||||
encode zstd gzip
|
|
||||||
|
|
||||||
@backend_paths path /api/* /digest /discord/interactions
|
|
||||||
|
|
||||||
# Active health check on the same cheap /healthz route each container's own
|
|
||||||
# HEALTHCHECK uses (Dockerfile) — so a deploy that's still restarting/booting
|
|
||||||
# never gets proxied into (a reload alone has no gate, hop-1 runbook hazard #10).
|
|
||||||
# 15s (was 5s): plenty responsive for a process that only restarts on a deploy,
|
|
||||||
# and a quarter of the polling load.
|
|
||||||
handle @backend_paths {
|
|
||||||
reverse_proxy 127.0.0.1:8137 {
|
|
||||||
health_uri /healthz
|
|
||||||
health_interval 15s
|
|
||||||
health_timeout 3s
|
|
||||||
health_status 2xx
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
handle {
|
|
||||||
reverse_proxy 127.0.0.1:8080 {
|
|
||||||
health_uri /healthz
|
|
||||||
health_interval 15s
|
|
||||||
health_timeout 3s
|
|
||||||
health_status 2xx
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
# Access-log hygiene: the default JSON encoder serializes full request headers,
|
|
||||||
# the TLS block, and response headers on every line (measured ~1,133B/line) --
|
|
||||||
# strip those with the `filter` format encoder. Also strip the query string from
|
|
||||||
# the logged URI: Caddy's default logger records request.uri *including* the
|
|
||||||
# query string, so every `?q=<search text>` a visitor typed sat in Loki next to
|
|
||||||
# their client IP for the full 30-day retention -- a real privacy leak, not just
|
|
||||||
# noise. Bot/crawler skipping stays out of here: `log_skip` needs Caddy >= 2.7
|
|
||||||
# and an upgrade is out of scope, so that's handled downstream in Alloy's
|
|
||||||
# loki.process "caddy" stage instead (see observability/alloy/config.alloy).
|
|
||||||
log {
|
|
||||||
output file /var/log/caddy/thermograph.log {
|
|
||||||
roll_size 20MiB
|
|
||||||
roll_keep 5
|
|
||||||
}
|
|
||||||
format filter {
|
|
||||||
wrap json
|
|
||||||
fields {
|
|
||||||
request>headers delete
|
|
||||||
request>tls delete
|
|
||||||
resp_headers delete
|
|
||||||
request>uri regexp \?.* ""
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
emigriffith.dev {
|
|
||||||
encode zstd gzip
|
|
||||||
|
|
||||||
# Thermograph moved to its own domain. Send the old sub-path there with a
|
|
||||||
# permanent redirect, stripping the /thermograph prefix so deep links map
|
|
||||||
# straight across (…/thermograph/calendar -> thermograph.org/calendar). The
|
|
||||||
# bare /thermograph (no trailing slash) goes to the new root.
|
|
||||||
handle_path /thermograph/* {
|
|
||||||
redir https://thermograph.org{uri} permanent
|
|
||||||
}
|
|
||||||
handle /thermograph {
|
|
||||||
redir https://thermograph.org/ permanent
|
|
||||||
}
|
|
||||||
|
|
||||||
# Portfolio at the root. Point `root` at the built static site (for the Astro
|
|
||||||
# portfolio that's its `dist/` output). file_server serves index.html for
|
|
||||||
# directories and returns a real 404 for missing paths.
|
|
||||||
handle {
|
|
||||||
root * /var/www/emigriffith
|
|
||||||
file_server
|
|
||||||
}
|
|
||||||
|
|
||||||
log {
|
|
||||||
output file /var/log/caddy/emigriffith.log
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
# Old bookmarks to the raw IP (the pre-domain URL) would otherwise get bounced to
|
|
||||||
# HTTPS-on-the-IP, which has no cert and fails. Redirect them to the portfolio domain.
|
|
||||||
http://75.119.132.91 {
|
|
||||||
redir https://emigriffith.dev{uri} permanent
|
|
||||||
}
|
|
||||||
|
|
||||||
# Optional: redirect www -> apex for either domain. Add the www CNAME/A record
|
|
||||||
# first, then uncomment the matching block.
|
|
||||||
# www.emigriffith.dev {
|
|
||||||
# redir https://emigriffith.dev{uri} permanent
|
|
||||||
# }
|
|
||||||
# www.thermograph.org {
|
|
||||||
# redir https://thermograph.org{uri} permanent
|
|
||||||
# }
|
|
||||||
108
infra/deploy/Caddyfile.vps1
Normal file
108
infra/deploy/Caddyfile.vps1
Normal file
|
|
@ -0,0 +1,108 @@
|
||||||
|
# /etc/caddy/Caddyfile on **vps1** (75.119.132.91) — the operational-programs box.
|
||||||
|
#
|
||||||
|
# vps1 serves:
|
||||||
|
# emigriffith.dev -> static portfolio site (from disk)
|
||||||
|
# git.thermograph.org -> Forgejo (see deploy/forgejo/caddy-git.conf)
|
||||||
|
# dashboard.thermograph.org -> Grafana (see observability/caddy-grafana.conf)
|
||||||
|
#
|
||||||
|
# and hosts the **dev** environment at dev.thermograph.org (below).
|
||||||
|
#
|
||||||
|
# beta.thermograph.org is NOT here either — beta moved to vps2, next to prod.
|
||||||
|
# That is the one edit in this file most likely to be made by mistake: a "beta"
|
||||||
|
# reference elsewhere in the repo that still points at 75.119.132.91 means this
|
||||||
|
# box, which no longer runs beta at all.
|
||||||
|
#
|
||||||
|
# Each domain's A/AAAA record must already point here — Caddy provisions a
|
||||||
|
# Let's Encrypt cert on first request and auto-renews.
|
||||||
|
|
||||||
|
emigriffith.dev {
|
||||||
|
encode zstd gzip
|
||||||
|
|
||||||
|
# Thermograph moved to its own domain. Send the old sub-path there with a
|
||||||
|
# permanent redirect, stripping the /thermograph prefix so deep links map
|
||||||
|
# straight across (…/thermograph/calendar -> thermograph.org/calendar). The
|
||||||
|
# bare /thermograph (no trailing slash) goes to the new root.
|
||||||
|
handle_path /thermograph/* {
|
||||||
|
redir https://thermograph.org{uri} permanent
|
||||||
|
}
|
||||||
|
handle /thermograph {
|
||||||
|
redir https://thermograph.org/ permanent
|
||||||
|
}
|
||||||
|
|
||||||
|
# Portfolio at the root. Point `root` at the built static site (for the Astro
|
||||||
|
# portfolio that's its `dist/` output). file_server serves index.html for
|
||||||
|
# directories and returns a real 404 for missing paths.
|
||||||
|
handle {
|
||||||
|
root * /var/www/emigriffith
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
|
||||||
|
log {
|
||||||
|
output file /var/log/caddy/emigriffith.log
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Old bookmarks to the raw IP (the pre-domain URL) would otherwise get bounced to
|
||||||
|
# HTTPS-on-the-IP, which has no cert and fails. Redirect them to the portfolio domain.
|
||||||
|
http://75.119.132.91 {
|
||||||
|
redir https://emigriffith.dev{uri} permanent
|
||||||
|
}
|
||||||
|
|
||||||
|
# Dev. Publicly resolvable, deliberately NOT publicly usable.
|
||||||
|
#
|
||||||
|
# Dev runs whatever branch is in flight — including unreviewed ones — on the
|
||||||
|
# same box as Forgejo and its CI runner. Three things make that acceptable, and
|
||||||
|
# all three are load-bearing:
|
||||||
|
#
|
||||||
|
# 1. The stack binds LOOPBACK (deploy/env-topology.sh sets DEV_BIND_ADDR to
|
||||||
|
# 127.0.0.1), so this site block is the only route in. Not even the mesh
|
||||||
|
# reaches the app directly.
|
||||||
|
# 2. basic auth, so a stumbled-upon URL is not a running build.
|
||||||
|
# 3. X-Robots-Tag: noindex, so dev never competes with thermograph.org in a
|
||||||
|
# search index the way a second copy of the same content otherwise would.
|
||||||
|
#
|
||||||
|
# Only :8137 is proxied. The dev overlay does not publish the frontend at all
|
||||||
|
# (docker-compose.dev.yml `ports: !reset null`) — the backend reverse-proxies to
|
||||||
|
# it internally, so one upstream serves the whole site here, unlike vps2 where
|
||||||
|
# Caddy path-splits across two ports.
|
||||||
|
#
|
||||||
|
# The credential below is a bcrypt hash, not a password. Rotate with:
|
||||||
|
# caddy hash-password --plaintext '<new>'
|
||||||
|
dev.thermograph.org {
|
||||||
|
encode zstd gzip
|
||||||
|
|
||||||
|
header {
|
||||||
|
X-Robots-Tag "noindex, nofollow"
|
||||||
|
}
|
||||||
|
|
||||||
|
# /healthz is deliberately OUTSIDE the auth boundary. It returns liveness and
|
||||||
|
# nothing else — no data, no version, no configuration — and Centralis polls
|
||||||
|
# it to report dev in `fleet_status`. Behind basic auth that poll gets a 401
|
||||||
|
# and dev reads as permanently down, which is how a monitoring blind spot
|
||||||
|
# gets created in the name of security.
|
||||||
|
@protected not path /healthz
|
||||||
|
|
||||||
|
basic_auth @protected {
|
||||||
|
dev $2a$14$LU5sNyxbop3HOsjhXB6ZrOQiveqhbcYVE6.Wi0bydAv4QhNpj4HMC
|
||||||
|
}
|
||||||
|
|
||||||
|
reverse_proxy 127.0.0.1:8137 {
|
||||||
|
health_uri /healthz
|
||||||
|
health_interval 15s
|
||||||
|
health_timeout 3s
|
||||||
|
health_status 2xx
|
||||||
|
}
|
||||||
|
|
||||||
|
log {
|
||||||
|
output file /var/log/caddy/dev.log {
|
||||||
|
roll_size 20MiB
|
||||||
|
roll_keep 5
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Optional: redirect www -> apex. Add the www CNAME/A record first, then
|
||||||
|
# uncomment.
|
||||||
|
# www.emigriffith.dev {
|
||||||
|
# redir https://emigriffith.dev{uri} permanent
|
||||||
|
# }
|
||||||
145
infra/deploy/Caddyfile.vps2
Normal file
145
infra/deploy/Caddyfile.vps2
Normal file
|
|
@ -0,0 +1,145 @@
|
||||||
|
# /etc/caddy/Caddyfile on **vps2** (169.58.46.181) — the public-facing box.
|
||||||
|
#
|
||||||
|
# vps2 serves BOTH environments an external user can reach:
|
||||||
|
# thermograph.org -> prod (loopback LB on 127.0.0.1:8137 / :8080)
|
||||||
|
# beta.thermograph.org -> beta (loopback LB on 127.0.0.1:8237 / :8180)
|
||||||
|
#
|
||||||
|
# and Centralis at mcp.thermograph.org, which is provisioned separately.
|
||||||
|
#
|
||||||
|
# Each domain's A/AAAA record must already point here — Caddy provisions a
|
||||||
|
# Let's Encrypt cert on first request and auto-renews. Nothing else to do for
|
||||||
|
# TLS (just make sure ports 80 and 443 are open).
|
||||||
|
#
|
||||||
|
# This file was split out of a single deploy/Caddyfile that described both
|
||||||
|
# boxes' sites at once. That was workable while each box served an unrelated
|
||||||
|
# set; it stopped being workable when the two ports 8137/8080 started meaning
|
||||||
|
# "prod on vps2" here and nothing at all on the other box. See Caddyfile.vps1
|
||||||
|
# for git/dashboard/portfolio.
|
||||||
|
#
|
||||||
|
# Thermograph owns thermograph.org's root, so both services run with
|
||||||
|
# THERMOGRAPH_BASE=/ — pages, assets and API all sit at "/" with no sub-path
|
||||||
|
# prefix. Backend and frontend are separate containers, each on its own loopback
|
||||||
|
# port; Caddy path-splits directly to whichever owns a given path, so the
|
||||||
|
# browser still sees one apparent origin. The enumerated backend list below
|
||||||
|
# mirrors backend/web/app.py's own routing exactly (a single catch-all proxy to
|
||||||
|
# frontend for everything else) -- unlike frontend's paths, backend's don't grow
|
||||||
|
# every time a new static asset filename is added. A gap in this list still just
|
||||||
|
# degrades to "one extra hop" through backend's own proxy fallback, never a 404.
|
||||||
|
|
||||||
|
thermograph.org {
|
||||||
|
encode zstd gzip
|
||||||
|
|
||||||
|
@backend_paths path /api/* /digest /discord/interactions
|
||||||
|
|
||||||
|
# Active health check on the same cheap /healthz route each container's own
|
||||||
|
# HEALTHCHECK uses (Dockerfile) — so a deploy that's still restarting/booting
|
||||||
|
# never gets proxied into (a reload alone has no gate, hop-1 runbook hazard #10).
|
||||||
|
# 15s (was 5s): plenty responsive for a process that only restarts on a deploy,
|
||||||
|
# and a quarter of the polling load.
|
||||||
|
handle @backend_paths {
|
||||||
|
reverse_proxy 127.0.0.1:8137 {
|
||||||
|
health_uri /healthz
|
||||||
|
health_interval 15s
|
||||||
|
health_timeout 3s
|
||||||
|
health_status 2xx
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
handle {
|
||||||
|
reverse_proxy 127.0.0.1:8080 {
|
||||||
|
health_uri /healthz
|
||||||
|
health_interval 15s
|
||||||
|
health_timeout 3s
|
||||||
|
health_status 2xx
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Access-log hygiene: the default JSON encoder serializes full request headers,
|
||||||
|
# the TLS block, and response headers on every line (measured ~1,133B/line) --
|
||||||
|
# strip those with the `filter` format encoder. Also strip the query string from
|
||||||
|
# the logged URI: Caddy's default logger records request.uri *including* the
|
||||||
|
# query string, so every `?q=<search text>` a visitor typed sat in Loki next to
|
||||||
|
# their client IP for the full 30-day retention -- a real privacy leak, not just
|
||||||
|
# noise. Bot/crawler skipping stays out of here: `log_skip` needs Caddy >= 2.7
|
||||||
|
# and an upgrade is out of scope, so that's handled downstream in Alloy's
|
||||||
|
# loki.process "caddy" stage instead (see observability/alloy/config.alloy).
|
||||||
|
#
|
||||||
|
# The FILENAME is load-bearing now: Alloy attributes each access log to an
|
||||||
|
# environment by filename (thermograph.log -> host="prod", beta.log ->
|
||||||
|
# host="beta"). One Caddy fronts two environments here, so a single log file
|
||||||
|
# would file every beta request under prod. Renaming either file means
|
||||||
|
# updating loki.process "caddy" in observability/alloy/config.alloy.
|
||||||
|
log {
|
||||||
|
output file /var/log/caddy/thermograph.log {
|
||||||
|
roll_size 20MiB
|
||||||
|
roll_keep 5
|
||||||
|
}
|
||||||
|
format filter {
|
||||||
|
wrap json
|
||||||
|
fields {
|
||||||
|
request>headers delete
|
||||||
|
request>tls delete
|
||||||
|
resp_headers delete
|
||||||
|
request>uri regexp \?.* ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Beta — same shape as prod, pointed at beta's loopback LB. It moved here from
|
||||||
|
# its own box; the DNS A record for beta.thermograph.org must point at vps2.
|
||||||
|
#
|
||||||
|
# Deliberately NOT indexed: beta serves the same content as prod at a different
|
||||||
|
# hostname, which is a duplicate-content problem for search engines and an
|
||||||
|
# invitation for users to land on a rehearsal environment from a search result.
|
||||||
|
# The app itself never pings IndexNow from beta (see env-topology.sh's
|
||||||
|
# TG_POST_DEPLOY), and this header closes the other half.
|
||||||
|
beta.thermograph.org {
|
||||||
|
encode zstd gzip
|
||||||
|
|
||||||
|
header {
|
||||||
|
X-Robots-Tag "noindex, nofollow"
|
||||||
|
}
|
||||||
|
|
||||||
|
@backend_paths path /api/* /digest /discord/interactions
|
||||||
|
|
||||||
|
handle @backend_paths {
|
||||||
|
reverse_proxy 127.0.0.1:8237 {
|
||||||
|
health_uri /healthz
|
||||||
|
health_interval 15s
|
||||||
|
health_timeout 3s
|
||||||
|
health_status 2xx
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
handle {
|
||||||
|
reverse_proxy 127.0.0.1:8180 {
|
||||||
|
health_uri /healthz
|
||||||
|
health_interval 15s
|
||||||
|
health_timeout 3s
|
||||||
|
health_status 2xx
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
log {
|
||||||
|
output file /var/log/caddy/beta.log {
|
||||||
|
roll_size 20MiB
|
||||||
|
roll_keep 5
|
||||||
|
}
|
||||||
|
format filter {
|
||||||
|
wrap json
|
||||||
|
fields {
|
||||||
|
request>headers delete
|
||||||
|
request>tls delete
|
||||||
|
resp_headers delete
|
||||||
|
request>uri regexp \?.* ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Optional: redirect www -> apex. Add the www CNAME/A record first, then
|
||||||
|
# uncomment.
|
||||||
|
# www.thermograph.org {
|
||||||
|
# redir https://thermograph.org{uri} permanent
|
||||||
|
# }
|
||||||
452
infra/deploy/RUNBOOK-vps1-vps2-cutover.md
Normal file
452
infra/deploy/RUNBOOK-vps1-vps2-cutover.md
Normal file
|
|
@ -0,0 +1,452 @@
|
||||||
|
# Runbook — moving beta to vps2 and dev to vps1
|
||||||
|
|
||||||
|
**Status: not executed.** Everything this runbook describes is landed in the
|
||||||
|
repo and live nowhere. The estate still runs the old shape until someone works
|
||||||
|
through the steps below.
|
||||||
|
|
||||||
|
## What changes
|
||||||
|
|
||||||
|
| | before | after |
|
||||||
|
|---|---|---|
|
||||||
|
| `75.119.132.91` (**vps1**) | beta + Forgejo + Grafana/Loki | Forgejo + Grafana/Loki + **dev** |
|
||||||
|
| `169.58.46.181` (**vps2**) | prod + Centralis + Postfix + backups | **prod + beta** + Centralis + Postfix + backups |
|
||||||
|
| desktop | LAN dev server + CI runner | AI models + flex Swarm worker, **no environment** |
|
||||||
|
| databases | one Postgres per environment | one instance on vps2, `thermograph` + `thermograph_beta` |
|
||||||
|
|
||||||
|
Mesh addresses do **not** move: vps1 stays `10.10.0.2`, vps2 stays `10.10.0.1`.
|
||||||
|
Every "beta = 75.119.132.91" reference anywhere is wrong after this — that
|
||||||
|
address is vps1, and sending a beta-intended command there is the most likely
|
||||||
|
way to do damage during this cutover.
|
||||||
|
|
||||||
|
## Order, and why it is this order
|
||||||
|
|
||||||
|
Beta moves first and completely, while dev stays where it is. Then dev moves.
|
||||||
|
The two halves are independent, so a problem in one never forces a rollback of
|
||||||
|
the other — and beta is the half that carries a public hostname and a TLS
|
||||||
|
certificate, so it gets done while you have the most attention.
|
||||||
|
|
||||||
|
Prod is touched twice, and both touches are additive: a new database + role on
|
||||||
|
its Postgres instance, and a new site block in its Caddyfile. Prod's stack file,
|
||||||
|
its services, its volumes and its env file are not modified by this work at all.
|
||||||
|
|
||||||
|
**Expected downtime:** beta, roughly the length of a dump/restore plus a stack
|
||||||
|
bring-up (~10–20 min). Dev, as long as you like. **Prod: none** — unless step 3
|
||||||
|
or 8 is done wrong, which is what the verification lines are for.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. Before you start
|
||||||
|
|
||||||
|
- [ ] **Lower the DNS TTL** on `beta.thermograph.org` to 300s, at least an hour
|
||||||
|
before step 7. Do this first; it is the only step with a lead time.
|
||||||
|
- [ ] **Create the new Forgejo Actions secrets** (Settings → Secrets). The
|
||||||
|
workflows on this branch reference them and will fail without them:
|
||||||
|
|
||||||
|
VPS1_SSH_HOST=75.119.132.91 VPS1_SSH_USER=agent
|
||||||
|
VPS1_SSH_KEY=<the CI deploy key> VPS1_SSH_PORT=22
|
||||||
|
VPS2_SSH_HOST=169.58.46.181 VPS2_SSH_USER=agent
|
||||||
|
VPS2_SSH_KEY=<the CI deploy key> VPS2_SSH_PORT=22
|
||||||
|
|
||||||
|
Keep the old `SSH_*` / `PROD_SSH_*` secrets until step 11 — they are the
|
||||||
|
rollback path, and deleting them early strands you.
|
||||||
|
|
||||||
|
**Create these BEFORE merging.** `infra-sync.yml` fires on any push to
|
||||||
|
`dev` or `main` touching `infra/**`, and this change touches a great deal
|
||||||
|
of it. Without the new secrets those jobs SSH to an empty host and fail;
|
||||||
|
`sync-beta` additionally targets `/opt/thermograph-beta`, which does not
|
||||||
|
exist until step 1. Nothing is damaged either way — the jobs only fetch
|
||||||
|
and render — but you will get a red run and a misleading alert.
|
||||||
|
|
||||||
|
The app `Deploy` workflow does **not** fire on this merge: it is
|
||||||
|
path-filtered to `backend/**` and `frontend/**`, and this change touches
|
||||||
|
neither.
|
||||||
|
- [ ] **Take a fresh prod backup and confirm it landed**, rather than trusting
|
||||||
|
the schedule:
|
||||||
|
|
||||||
|
```
|
||||||
|
# from a machine with repo access
|
||||||
|
# Actions -> "Ops cron (backup + IndexNow)" -> Run workflow
|
||||||
|
ssh agent@169.58.46.181 'ls -lh ~/thermograph-backups | tail -3'
|
||||||
|
```
|
||||||
|
- [ ] **Confirm you can decrypt the vault** (`sops -d infra/deploy/secrets/beta.yaml`
|
||||||
|
from `infra/`). Steps 3 and 5 need the rendered beta password.
|
||||||
|
- [ ] Have the PR merged to `dev` and promoted to `main`. Hosts pull `main`, so
|
||||||
|
nothing below works from an unmerged branch.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Give beta a checkout on vps2
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh agent@169.58.46.181
|
||||||
|
sudo git clone --branch main http://10.10.0.2:3080/emi/thermograph.git /opt/thermograph-beta
|
||||||
|
sudo chown -R agent:agent /opt/thermograph-beta
|
||||||
|
```
|
||||||
|
|
||||||
|
Prod's checkout at `/opt/thermograph` is untouched. Two checkouts side by side
|
||||||
|
is the whole point: a `git reset --hard` for beta must never be able to move
|
||||||
|
prod's tree.
|
||||||
|
|
||||||
|
**Verify:** `git -C /opt/thermograph-beta log --oneline -1` matches
|
||||||
|
`git -C /opt/thermograph log --oneline -1`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Render beta's env file on vps2
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh agent@169.58.46.181
|
||||||
|
cd /opt/thermograph-beta/infra
|
||||||
|
. deploy/render-secrets.sh
|
||||||
|
render_thermograph_secrets /opt/thermograph-beta/infra beta /etc/thermograph-beta.env
|
||||||
|
```
|
||||||
|
|
||||||
|
The age key is already at `/etc/thermograph/age.key` on this box (prod uses it);
|
||||||
|
beta needs no second copy.
|
||||||
|
|
||||||
|
**Verify** — beta's file must name beta's role and database, and prod's must be
|
||||||
|
untouched:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo grep -c . /etc/thermograph-beta.env # non-zero
|
||||||
|
sudo grep -o 'thermograph_beta' /etc/thermograph-beta.env | head -1
|
||||||
|
sudo grep -o '@db:5432/thermograph$' /etc/thermograph.env # prod: still 'thermograph'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Create beta's role and database on the shared instance
|
||||||
|
|
||||||
|
This is the first step that touches prod's Postgres. It only ever adds; it drops
|
||||||
|
nothing.
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh agent@169.58.46.181
|
||||||
|
sudo bash /opt/thermograph-beta/infra/deploy/db/provision-env-db.sh beta
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verify the isolation is real** — this is the whole justification for one
|
||||||
|
instance serving two environments, so actually run it:
|
||||||
|
|
||||||
|
```
|
||||||
|
DBC=$(docker ps -q --filter label=com.docker.swarm.service.name=thermograph_db | head -1)
|
||||||
|
# beta's role must NOT be able to reach prod's database:
|
||||||
|
docker exec "$DBC" psql -U thermograph_beta -d thermograph -c 'select 1' # must FAIL
|
||||||
|
# beta's read-only role exists and really is read-only:
|
||||||
|
docker exec "$DBC" psql -U thermograph_beta_ro -d thermograph_beta -c 'select 1' # must SUCCEED
|
||||||
|
docker exec "$DBC" psql -U thermograph_beta_ro -d thermograph_beta \
|
||||||
|
-c 'create table _x(i int)' # must FAIL
|
||||||
|
# prod's own role is UNTOUCHED — still a superuser:
|
||||||
|
docker exec "$DBC" psql -U thermograph -d postgres -tAc \
|
||||||
|
"select rolsuper from pg_roles where rolname='thermograph'" # must be t
|
||||||
|
# and prod is still healthy:
|
||||||
|
curl -fsS -o /dev/null -w '%{http_code}\n' https://thermograph.org/api/health
|
||||||
|
```
|
||||||
|
|
||||||
|
The script **refuses to run for prod** — prod's `thermograph` role is the
|
||||||
|
instance's bootstrap superuser, not a guest, and provisioning it would demote
|
||||||
|
it. That guard is why the superuser check above should never fail; run it
|
||||||
|
anyway.
|
||||||
|
|
||||||
|
If the first command SUCCEEDS, stop. `CONNECT` was not revoked and beta would be
|
||||||
|
able to open a session against the production database; re-run the provisioning
|
||||||
|
script and re-check before going further.
|
||||||
|
|
||||||
|
This is not hypothetical — it happened on the live cutover. The first version of
|
||||||
|
the script revoked `PUBLIC` connect on the *guest's* database only, and a fresh
|
||||||
|
Postgres database carries `=Tc` (TEMP **and** CONNECT) for `PUBLIC`, so
|
||||||
|
`thermograph_beta` could still reach `thermograph`. The script now hardens the
|
||||||
|
owner's database too, granting the owner's read-only role an explicit `CONNECT`
|
||||||
|
first so the revoke cannot strip `ops/dbq.sh` of its access. The check above is
|
||||||
|
what caught it.
|
||||||
|
|
||||||
|
**Rollback:** `DROP DATABASE thermograph_beta; DROP ROLE thermograph_beta;` —
|
||||||
|
prod is unaffected either way.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Move beta's data (optional)
|
||||||
|
|
||||||
|
Beta's existing database still lives on vps1. Decide honestly whether you want
|
||||||
|
it: beta's value is a rehearsal of prod, and a fresh schema with a handful of
|
||||||
|
test accounts is often *better* than carrying old beta state across. If you do
|
||||||
|
want it:
|
||||||
|
|
||||||
|
```
|
||||||
|
# on vps1 — dump the old beta database
|
||||||
|
ssh agent@75.119.132.91 \
|
||||||
|
'docker exec thermograph-db-1 pg_dump -U thermograph -d thermograph --format=custom' \
|
||||||
|
> /tmp/beta-premove.dump
|
||||||
|
|
||||||
|
# on vps2 — restore into beta's database as beta's role
|
||||||
|
scp /tmp/beta-premove.dump agent@169.58.46.181:/tmp/
|
||||||
|
ssh agent@169.58.46.181 '
|
||||||
|
DBC=$(docker ps -q --filter label=com.docker.swarm.service.name=thermograph_db | head -1)
|
||||||
|
docker cp /tmp/beta-premove.dump "$DBC":/tmp/
|
||||||
|
docker exec "$DBC" pg_restore -U thermograph -d thermograph_beta --no-owner --role=thermograph_beta /tmp/beta-premove.dump
|
||||||
|
'
|
||||||
|
```
|
||||||
|
|
||||||
|
`--no-owner --role=thermograph_beta` matters: the dump's objects are owned by the
|
||||||
|
old `thermograph` role, and restoring them verbatim would leave beta's tables
|
||||||
|
owned by a role beta does not connect as.
|
||||||
|
|
||||||
|
**Do not** restore a *prod* dump into beta. Beta's daemon runs the notification
|
||||||
|
timers, and beta's subscriber table would then be full of real people. Beta's
|
||||||
|
vault sets `THERMOGRAPH_MAIL_BACKEND=console` so nothing would actually send,
|
||||||
|
but that is one `sops edit` away from not being true.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Bring beta's stack up on vps2
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh agent@169.58.46.181
|
||||||
|
cd /opt/thermograph-beta
|
||||||
|
THERMOGRAPH_ENV=beta SERVICE=all \
|
||||||
|
BACKEND_IMAGE_TAG=<current beta backend tag> \
|
||||||
|
FRONTEND_IMAGE_TAG=<current beta frontend tag> \
|
||||||
|
infra/deploy/deploy.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Get the current tags from the old beta host first
|
||||||
|
(`cat /opt/thermograph/infra/deploy/.image-tags.env` on vps1), so beta comes up
|
||||||
|
on exactly the code it was already running.
|
||||||
|
|
||||||
|
**Verify** — beta answers on its own loopback ports, and prod's are unmoved:
|
||||||
|
|
||||||
|
```
|
||||||
|
curl -fsS -o /dev/null -w 'beta %{http_code}\n' http://127.0.0.1:8237/healthz
|
||||||
|
curl -fsS -o /dev/null -w 'prod %{http_code}\n' http://127.0.0.1:8137/healthz
|
||||||
|
docker stack services thermograph-beta # 5 services, all 1/1
|
||||||
|
docker stack services thermograph # prod: unchanged, still 1/1
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verify the DNS-collision fix actually held** — this is the failure mode that
|
||||||
|
would be subtle and awful in production:
|
||||||
|
|
||||||
|
```
|
||||||
|
# prod's frontend must resolve prod's web, not beta's
|
||||||
|
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=thermograph_frontend | head -1) \
|
||||||
|
getent hosts web
|
||||||
|
docker exec $(docker ps -q -f label=com.docker.swarm.service.name=thermograph_frontend | head -1) \
|
||||||
|
getent hosts beta-web # should NOT resolve on prod's network
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rollback:** `docker stack rm thermograph-beta` and remove the LB container
|
||||||
|
(`docker rm -f thermograph-beta-lb`). Beta on vps1 is still running and still
|
||||||
|
serving; nothing has moved yet.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Add beta's site block to vps2's Caddy
|
||||||
|
|
||||||
|
`infra/deploy/Caddyfile.vps2` is the reference copy. Append its
|
||||||
|
`beta.thermograph.org` block to the live `/etc/caddy/Caddyfile` on vps2.
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh agent@169.58.46.181
|
||||||
|
sudo caddy validate --config /etc/caddy/Caddyfile # BEFORE reloading
|
||||||
|
# `validate` runs as root and CREATES any log file the config names, owned by
|
||||||
|
# root — which the caddy user then cannot open, and the reload fails. Fix the
|
||||||
|
# ownership before reloading, or the first reload rejects the whole config:
|
||||||
|
sudo chown caddy:caddy /var/log/caddy/beta.log
|
||||||
|
sudo systemctl reload caddy
|
||||||
|
```
|
||||||
|
|
||||||
|
Validate before reload, always: a malformed Caddyfile takes `thermograph.org`
|
||||||
|
down with it, and prod is on this box now.
|
||||||
|
|
||||||
|
The `chown` is not optional and it bit this cutover. `caddy validate` as root
|
||||||
|
left `/var/log/caddy/beta.log` as `root:root`, so the reload failed with
|
||||||
|
`permission denied` on the log writer. (The root-owned `centralis.log` sitting
|
||||||
|
in that directory is the same mistake, made earlier.) Note what did NOT happen:
|
||||||
|
Caddy rejects a bad config atomically and keeps serving the old one, so
|
||||||
|
`thermograph.org` stayed up throughout — verify it anyway.
|
||||||
|
|
||||||
|
Caddy cannot issue the certificate until DNS moves (step 7), so expect the beta
|
||||||
|
hostname to fail TLS until then. That is fine and expected.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Move DNS
|
||||||
|
|
||||||
|
Point `beta.thermograph.org`'s A record at **169.58.46.181**.
|
||||||
|
|
||||||
|
Then wait for propagation and the certificate:
|
||||||
|
|
||||||
|
```
|
||||||
|
dig +short beta.thermograph.org # 169.58.46.181
|
||||||
|
curl -fsS -o /dev/null -w '%{http_code}\n' https://beta.thermograph.org/api/health
|
||||||
|
ssh agent@169.58.46.181 'sudo journalctl -u caddy -n 30 --no-pager | grep -i certificate'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rollback:** point the A record back at 75.119.132.91. Beta on vps1 is still
|
||||||
|
up (you have not stopped it yet — that is step 9), so this is a clean revert
|
||||||
|
for as long as you leave it running.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Stand dev up on vps1
|
||||||
|
|
||||||
|
Independent of everything above; do it whenever.
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh agent@75.119.132.91
|
||||||
|
sudo bash /opt/thermograph-dev/infra/deploy/provision-dev.sh # clones if absent
|
||||||
|
```
|
||||||
|
|
||||||
|
The script writes `/etc/thermograph/secrets-env` = `dev` and removes any
|
||||||
|
`deploy-mode` marker.
|
||||||
|
|
||||||
|
**That marker flip is load-bearing, and it is why this step comes after step 9.**
|
||||||
|
dev's env file is `/etc/thermograph.env` — the very path beta uses while beta is
|
||||||
|
still on vps1. `infra-sync`'s `sync-dev` job refuses to render unless the marker
|
||||||
|
already says `dev`, so until you run this script it syncs the checkout and skips
|
||||||
|
the render with a loud message. Flip the marker before beta has vacated and the
|
||||||
|
next dev push overwrites beta's live env file with dev's credentials; beta's
|
||||||
|
running containers survive it and then come up wrong on their next deploy.
|
||||||
|
|
||||||
|
(The checkout itself is safe to create at any time — only the render is gated.)
|
||||||
|
|
||||||
|
Then deploy:
|
||||||
|
|
||||||
|
```
|
||||||
|
cd /opt/thermograph-dev
|
||||||
|
SERVICE=all BACKEND_IMAGE_TAG=<dev tag> FRONTEND_IMAGE_TAG=<dev tag> \
|
||||||
|
infra/deploy/deploy-dev.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verify Caddy is the only way in.** This is the security-relevant check of the
|
||||||
|
whole cutover — vps1 is a public box and dev runs unreviewed branches:
|
||||||
|
|
||||||
|
```
|
||||||
|
ss -ltnp | grep 8137 # MUST show 127.0.0.1:8137, never 0.0.0.0
|
||||||
|
curl --max-time 5 http://75.119.132.91:8137/healthz # MUST fail/refuse
|
||||||
|
curl -o /dev/null -w '%{http_code}\n' https://dev.thermograph.org/ # 401 without creds
|
||||||
|
curl -o /dev/null -w '%{http_code}\n' -u dev:<pw> https://dev.thermograph.org/healthz # 200
|
||||||
|
```
|
||||||
|
|
||||||
|
Also confirm dev did **not** get the fleet's shared production credentials —
|
||||||
|
`dev.yaml` is rendered alone, and vps1 is exactly the box that must not hold
|
||||||
|
them:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo grep -c 'THERMOGRAPH_S3_SECRET_KEY\|REGISTRY_TOKEN\|VAPID_PRIVATE' /etc/thermograph.env # expect 0
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Retire beta from vps1
|
||||||
|
|
||||||
|
Only after beta has been serving from vps2 through step 7 for long enough that
|
||||||
|
you believe it.
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh agent@75.119.132.91
|
||||||
|
cd /opt/thermograph/infra
|
||||||
|
docker compose down # stops the old beta app stack
|
||||||
|
# remove beta's site block from /etc/caddy/Caddyfile (see Caddyfile.vps1)
|
||||||
|
sudo caddy validate --config /etc/caddy/Caddyfile && sudo systemctl reload caddy
|
||||||
|
```
|
||||||
|
|
||||||
|
**Keep the old `thermograph_pgdata` volume on vps1** until you are certain beta
|
||||||
|
on vps2 is healthy and backed up. It is the only copy of pre-move beta data.
|
||||||
|
Delete it deliberately, later, not as part of this runbook.
|
||||||
|
|
||||||
|
Forgejo, Grafana, Loki and the portfolio site all keep running on this box
|
||||||
|
untouched — do not stop anything else here.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Repoint the Alloy agents
|
||||||
|
|
||||||
|
Both nodes need the new variables; the label model changed (see
|
||||||
|
`observability/alloy/config.alloy`).
|
||||||
|
|
||||||
|
```
|
||||||
|
# vps2 — two environments, so two log volumes and the beta overlay
|
||||||
|
ssh agent@169.58.46.181
|
||||||
|
cd /opt/thermograph/observability/alloy
|
||||||
|
ALLOY_NODE=vps2 ALLOY_ENV=prod \
|
||||||
|
APPLOGS_VOLUME=thermograph_applogs BETA_APPLOGS_VOLUME=thermograph-beta_applogs \
|
||||||
|
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
|
||||||
|
docker compose -f docker-compose.agent.yml -f docker-compose.agent.beta.yml up -d
|
||||||
|
|
||||||
|
# vps1
|
||||||
|
ssh agent@75.119.132.91
|
||||||
|
cd /opt/thermograph/observability/alloy
|
||||||
|
ALLOY_NODE=vps1 ALLOY_ENV=dev APPLOGS_VOLUME=thermograph-dev_applogs \
|
||||||
|
LOKI_URL=http://10.10.0.2:3100/loki/api/v1/push \
|
||||||
|
docker compose -f docker-compose.agent.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verify each environment is labelled as itself** — the point of the change is
|
||||||
|
that beta's logs on vps2 are not filed as prod:
|
||||||
|
|
||||||
|
```
|
||||||
|
# in Grafana Explore, or via the Loki API:
|
||||||
|
# {host="beta"} |= "" -> should show beta traffic, from node="vps2"
|
||||||
|
# {host="prod"} |= "" -> prod only
|
||||||
|
# {host="dev"} |= "" -> from node="vps1"
|
||||||
|
# count by (host) (count_over_time({job="docker"}[5m])) -> three values, not one
|
||||||
|
```
|
||||||
|
|
||||||
|
If `{host="beta"}` is empty while beta is clearly serving, the container-name
|
||||||
|
relabel did not match — check `docker ps --format '{{.Names}}'` on vps2 against
|
||||||
|
the `/thermograph-beta_.*` regex.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Clean up
|
||||||
|
|
||||||
|
- [ ] Delete the old `SSH_*` and `PROD_SSH_*` Forgejo secrets (only now — they
|
||||||
|
were the rollback path).
|
||||||
|
- [ ] Run the ops cron by hand and confirm **both** databases are dumped:
|
||||||
|
`ssh agent@169.58.46.181 'ls ~/thermograph-backups | tail -4'` should show
|
||||||
|
both a `thermograph-*.dump` and a `thermograph_beta-*.dump`.
|
||||||
|
- [ ] Confirm the Forgejo backup job still works — it now uses `VPS1_SSH_*`,
|
||||||
|
and it follows Forgejo (vps1), not beta.
|
||||||
|
- [ ] Restore the DNS TTL on `beta.thermograph.org`.
|
||||||
|
- [ ] Desktop: stop the old LAN dev stack and free the box for the AI models.
|
||||||
|
Leave it joined to the Swarm as a worker; nothing schedules onto it today
|
||||||
|
(every app service is pinned `node.role == manager`), which is what makes
|
||||||
|
it safe to also run models there.
|
||||||
|
- [ ] Update Centralis — see below.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Centralis (outside this repo)
|
||||||
|
|
||||||
|
Centralis' own text still describes the old estate, and it is not in this
|
||||||
|
repository. These need a separate change in the Centralis repo:
|
||||||
|
|
||||||
|
- The `onboarding` tool's estate paragraph ("Four machines on a WireGuard
|
||||||
|
mesh... beta (beta.thermograph.org, Forgejo, Grafana + Loki), the operator's
|
||||||
|
desktop (LAN dev, the main CI runner)").
|
||||||
|
- The MCP server's own protocol-level `instructions` string, which carries a
|
||||||
|
second, separately-worded copy of the same paragraph.
|
||||||
|
- The `thermograph-orientation` skill's estate table, and `thermograph-ops`'
|
||||||
|
release-flow diagram ("LAN dev box"), its "ssh -L works for beta and is
|
||||||
|
impossible for prod" claim (beta is a Swarm overlay now, so it behaves like
|
||||||
|
prod), and its prod-only Swarm service-name list (beta has
|
||||||
|
`beta-`-prefixed ones now).
|
||||||
|
- Tool descriptions that hardcode the old shape: `fleet_status` ("Swarm services
|
||||||
|
on prod, compose containers on beta/dev"), `estate_status` and `rollback_to`
|
||||||
|
("Centralis cannot SSH to the dev desktop"), `logs_query`'s per-environment
|
||||||
|
service lists (Forgejo/Grafana/Loki are listed under `beta`; they belong under
|
||||||
|
`dev`/vps1 now), and `secrets_render`'s assumption that a host's own
|
||||||
|
`secrets-env` marker identifies one environment — vps2 has two.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## If it goes wrong
|
||||||
|
|
||||||
|
| symptom | most likely cause | action |
|
||||||
|
|---|---|---|
|
||||||
|
| prod 502s after step 5 | beta's stack collided with prod's service DNS | `docker stack rm thermograph-beta`; prod recovers on its own. Check beta's services are `beta-`-prefixed |
|
||||||
|
| beta 502s, prod fine | beta's LB or stack not up | `docker ps \| grep beta-lb`; `docker stack ps thermograph-beta --no-trunc` |
|
||||||
|
| beta can't reach the database | role/database missing, or the wrong network | re-run `provision-env-db.sh beta`; confirm beta's tasks are on `thermograph_internal` |
|
||||||
|
| beta serves prod's data | beta's env file has prod's URL | check `/etc/thermograph-beta.env` names `thermograph_beta` twice; re-render (step 2). **Stop beta until fixed** |
|
||||||
|
| prod's nightly backup fails | `thermograph_beta` doesn't exist yet | either finish step 3 or revert the ops-cron change; the job fails loudly by design rather than skipping silently |
|
||||||
|
| dev reachable from the internet | `DEV_BIND_ADDR` not applied | `ss -ltnp \| grep 8137`; redeploy via `deploy-dev.sh`, which sets it from `env-topology.sh` |
|
||||||
|
|
@ -6,11 +6,21 @@ single VPS. Driven by `.forgejo/workflows/ops-cron.yml` (03:00 UTC daily +
|
||||||
|
|
||||||
## What is backed up
|
## What is backed up
|
||||||
|
|
||||||
| Prefix in bucket `era5-thermograph` | Source | Job | Retention off-box |
|
| Prefix in bucket `era5-thermograph` | Source | Job | Host / secrets | Retention off-box |
|
||||||
|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| `backups/db/prod/` | prod Postgres/TimescaleDB (`pg_dump --format=custom`) | `backup` (prod, `PROD_SSH_*`) | 30 days |
|
| `backups/db/prod/` | prod's `thermograph` database (`pg_dump --format=custom`) | `backup` | vps2, `VPS2_SSH_*` | 30 days |
|
||||||
| `backups/forgejo/db/` | Forgejo Postgres (`forgejo_db`) | `forgejo-backup` (beta, `SSH_*`) | 30 days |
|
| `backups/db/beta/` | beta's `thermograph_beta` database, same shared instance | `backup` | vps2, `VPS2_SSH_*` | 30 days |
|
||||||
| `backups/forgejo/data/` | Forgejo data volume (repos/LFS/config) | `forgejo-backup` | 30 days |
|
| `backups/forgejo/db/` | Forgejo Postgres (`forgejo_db`) | `forgejo-backup` | vps1, `VPS1_SSH_*` | 30 days |
|
||||||
|
| `backups/forgejo/data/` | Forgejo data volume (repos/LFS/config) | `forgejo-backup` | vps1, `VPS1_SSH_*` | 30 days |
|
||||||
|
|
||||||
|
Both application databases live on the ONE shared TimescaleDB instance on
|
||||||
|
vps2 now (separate roles/databases, not separate boxes) — the `backup` job
|
||||||
|
dumps each by name so beta isn't silently left uncovered. Forgejo moved to
|
||||||
|
**vps1** with the vps1/vps2 rename (it used to be co-located with beta on the
|
||||||
|
box now called vps2's predecessor); its backup targets vps1 accordingly. See
|
||||||
|
`ops-cron.yml`'s own header comment for the history: an earlier revision keyed
|
||||||
|
these secrets by environment name rather than host, which is what let "the
|
||||||
|
prod backup" silently dump the wrong box for a while.
|
||||||
|
|
||||||
Everything is streamed through **`age`** encryption before upload — encrypted to the
|
Everything is streamed through **`age`** encryption before upload — encrypted to the
|
||||||
vault recipient `age1xx4dz…`, so the private key each host already has at
|
vault recipient `age1xx4dz…`, so the private key each host already has at
|
||||||
|
|
@ -23,9 +33,10 @@ decrypts it. Nothing plaintext leaves the box.
|
||||||
(also mirrored into the SOPS vault `deploy/secrets/{prod,beta}.yaml` for host-side
|
(also mirrored into the SOPS vault `deploy/secrets/{prod,beta}.yaml` for host-side
|
||||||
use). Contabo needs **path-style** addressing (`force_path_style=true`,
|
use). Contabo needs **path-style** addressing (`force_path_style=true`,
|
||||||
`provider=Other`, `region=default`).
|
`provider=Other`, `region=default`).
|
||||||
- `rclone` + `age` must be installed on prod and beta (one-time: `apt-get install -y
|
- `rclone` + `age` must be installed on vps1 and vps2 (one-time: `apt-get install -y
|
||||||
rclone age`). The prod job apt-installs them if missing; the beta user has no sudo,
|
rclone age`). The prod/beta job on vps2 apt-installs them if missing; check
|
||||||
so they are pre-installed there.
|
whether the Forgejo-backup job's user on vps1 has the sudo to do the same, or
|
||||||
|
whether they need pre-installing there.
|
||||||
|
|
||||||
## Restore (DR runbook)
|
## Restore (DR runbook)
|
||||||
|
|
||||||
|
|
@ -33,17 +44,27 @@ Prereq on the restoring host: `rclone` + `age` + the age key at
|
||||||
`/etc/thermograph/age.key` (or the operator key), and rclone configured for the
|
`/etc/thermograph/age.key` (or the operator key), and rclone configured for the
|
||||||
`archive:` remote (env vars `RCLONE_CONFIG_ARCHIVE_*` or `/etc/rclone/rclone.conf`).
|
`archive:` remote (env vars `RCLONE_CONFIG_ARCHIVE_*` or `/etc/rclone/rclone.conf`).
|
||||||
|
|
||||||
### Prod database
|
### Prod / beta database
|
||||||
|
|
||||||
|
Both restore into the SAME shared instance on vps2 (`<db-container>` is that
|
||||||
|
one instance's container/task, resolved the same way `dbq.sh` does) — only
|
||||||
|
the bucket prefix and the target database/role differ:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# list available encrypted dumps
|
# list available encrypted dumps
|
||||||
rclone ls archive:era5-thermograph/backups/db/prod/
|
rclone ls archive:era5-thermograph/backups/db/prod/
|
||||||
|
rclone ls archive:era5-thermograph/backups/db/beta/
|
||||||
# download newest, decrypt, restore into a fresh db
|
# download newest, decrypt, restore into a fresh db
|
||||||
OBJ=archive:era5-thermograph/backups/db/prod/<name>.dump.age
|
OBJ=archive:era5-thermograph/backups/db/prod/<name>.dump.age
|
||||||
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
|
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
|
||||||
| docker exec -i <db-container> pg_restore -U thermograph -d thermograph --clean --if-exists
|
| docker exec -i <db-container> pg_restore -U thermograph -d thermograph --clean --if-exists
|
||||||
|
# beta: same instance, its own role/database
|
||||||
|
OBJ=archive:era5-thermograph/backups/db/beta/<name>.dump.age
|
||||||
|
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
|
||||||
|
| docker exec -i <db-container> pg_restore -U thermograph -d thermograph_beta --clean --if-exists
|
||||||
```
|
```
|
||||||
|
|
||||||
### Forgejo
|
### Forgejo (restore on vps1)
|
||||||
```sh
|
```sh
|
||||||
# database
|
# database
|
||||||
rclone cat archive:era5-thermograph/backups/forgejo/db/<name>.dump.age \
|
rclone cat archive:era5-thermograph/backups/forgejo/db/<name>.dump.age \
|
||||||
|
|
|
||||||
|
|
@ -1,14 +1,17 @@
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# Postgres memory / performance tuning, scaled to the container's DB_MEMORY budget so
|
# Postgres memory / performance tuning, scaled to the container's DB_MEMORY budget so
|
||||||
# the same init serves every host (beta 8g; prod 16g on the 48 GB box) with no
|
# the same init works for any host with no hardcoding. There is ONE shared TimescaleDB
|
||||||
# hardcoding. Runs once on a fresh data volume from /docker-entrypoint-initdb.d, after
|
# instance now (prod and beta are separate databases/roles on it, not separate
|
||||||
|
# containers), sized from prod's vault values (16g on the 48 GB box) -- beta no longer
|
||||||
|
# gets a second database or a second tuning pass. Dev keeps its own, separate container
|
||||||
|
# (8g). Runs once on a fresh data volume from /docker-entrypoint-initdb.d, after
|
||||||
# 10-timescaledb.sql enables timescaledb. Settings are written via ALTER SYSTEM
|
# 10-timescaledb.sql enables timescaledb. Settings are written via ALTER SYSTEM
|
||||||
# (persisted to postgresql.auto.conf, which the timescaledb image's own
|
# (persisted to postgresql.auto.conf, which the timescaledb image's own
|
||||||
# timescaledb-tune postgresql.conf defers to); the container's post-init restart brings
|
# timescaledb-tune postgresql.conf defers to); the container's post-init restart brings
|
||||||
# restart-only settings (shared_buffers, …) into effect. NB: never ALTER SYSTEM SET
|
# restart-only settings (shared_buffers, max_connections, …) into effect. NB: never
|
||||||
# shared_preload_libraries here — that would land in auto.conf and override the image's
|
# ALTER SYSTEM SET shared_preload_libraries here — that would land in auto.conf and
|
||||||
# `timescaledb` preload. Init scripts do NOT re-run on an existing volume — to
|
# override the image's `timescaledb` preload. Init scripts do NOT re-run on an existing
|
||||||
# re-tune later, set DB_MEMORY and run this by hand, then restart:
|
# volume — to re-tune later, set DB_MEMORY and run this by hand, then restart:
|
||||||
# docker compose exec -e DB_MEMORY=16g db bash /docker-entrypoint-initdb.d/20-tuning.sh
|
# docker compose exec -e DB_MEMORY=16g db bash /docker-entrypoint-initdb.d/20-tuning.sh
|
||||||
# docker compose restart db
|
# docker compose restart db
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
@ -30,13 +33,30 @@ if [ "$mb" -lt 1024 ]; then mb=8192; fi
|
||||||
# maintenance_work_mem 512 MB) and scale linearly on a bigger box.
|
# maintenance_work_mem 512 MB) and scale linearly on a bigger box.
|
||||||
shared_buffers=$((mb / 4)) # 25% — the shared page cache
|
shared_buffers=$((mb / 4)) # 25% — the shared page cache
|
||||||
effective_cache=$((mb * 3 / 4)) # 75% — planner's view of total cache (PG + OS)
|
effective_cache=$((mb * 3 / 4)) # 75% — planner's view of total cache (PG + OS)
|
||||||
work_mem=$((mb / 128)) # ~64 MB at 8 GB (per-operation; kept modest)
|
maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM
|
||||||
|
|
||||||
|
# work_mem is per *sort operation*, not per connection, so its real cost is
|
||||||
|
# work_mem x concurrent sorts x max_connections (200, below) — and the db container
|
||||||
|
# has a hard memory limit. Left uncapped the ratio gives 128 MB at prod's 16g
|
||||||
|
# budget, which is more than a 200-connection instance should carry: a burst of
|
||||||
|
# heavy sorts can walk into the container limit, and an OOM-killed Postgres is a
|
||||||
|
# far worse day than a refused connection. Cap at 64 MB (dev's 8g is unaffected).
|
||||||
|
work_mem=$((mb / 128))
|
||||||
|
if [ "$work_mem" -gt 64 ]; then work_mem=64; fi
|
||||||
if [ "$work_mem" -lt 16 ]; then work_mem=16; fi
|
if [ "$work_mem" -lt 16 ]; then work_mem=16; fi
|
||||||
maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM
|
|
||||||
|
# Connection ceiling. NOT derived from DB_MEMORY: it is bounded by how many pools
|
||||||
|
# the app opens, not by RAM. One instance serves prod and beta, and every backend
|
||||||
|
# process opens three pools (two async engines + the notifier's sync engine, see
|
||||||
|
# backend/accounts/db.py), so the image default of 100 was below the configured
|
||||||
|
# worst case — prod saturated it on 2026-07-26 and started refusing clients with
|
||||||
|
# "sorry, too many clients already". 200 covers today's replica counts including
|
||||||
|
# web's autoscale maximum. ~+1 GB of per-connection overhead at 200.
|
||||||
|
max_conn=200
|
||||||
|
|
||||||
echo "[tuning] DB_MEMORY=${budget} -> ${mb}MB: shared_buffers=${shared_buffers}MB" \
|
echo "[tuning] DB_MEMORY=${budget} -> ${mb}MB: shared_buffers=${shared_buffers}MB" \
|
||||||
"effective_cache_size=${effective_cache}MB work_mem=${work_mem}MB" \
|
"effective_cache_size=${effective_cache}MB work_mem=${work_mem}MB" \
|
||||||
"maintenance_work_mem=${maint_mem}MB"
|
"maintenance_work_mem=${maint_mem}MB max_connections=${max_conn}"
|
||||||
|
|
||||||
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<SQL
|
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<SQL
|
||||||
-- Caching: the shared page cache, and the planner's view of total cache (PG + OS).
|
-- Caching: the shared page cache, and the planner's view of total cache (PG + OS).
|
||||||
|
|
@ -47,6 +67,10 @@ ALTER SYSTEM SET effective_cache_size = '${effective_cache}MB';
|
||||||
ALTER SYSTEM SET work_mem = '${work_mem}MB';
|
ALTER SYSTEM SET work_mem = '${work_mem}MB';
|
||||||
ALTER SYSTEM SET maintenance_work_mem = '${maint_mem}MB';
|
ALTER SYSTEM SET maintenance_work_mem = '${maint_mem}MB';
|
||||||
|
|
||||||
|
-- Connections. Restart-only, like shared_buffers: changing this on a live instance
|
||||||
|
-- takes an ALTER SYSTEM plus a db restart, it does not reload.
|
||||||
|
ALTER SYSTEM SET max_connections = ${max_conn};
|
||||||
|
|
||||||
-- Write throughput: fewer, larger checkpoints.
|
-- Write throughput: fewer, larger checkpoints.
|
||||||
ALTER SYSTEM SET wal_buffers = '16MB';
|
ALTER SYSTEM SET wal_buffers = '16MB';
|
||||||
ALTER SYSTEM SET min_wal_size = '1GB';
|
ALTER SYSTEM SET min_wal_size = '1GB';
|
||||||
|
|
|
||||||
201
infra/deploy/db/provision-env-db.sh
Executable file
201
infra/deploy/db/provision-env-db.sh
Executable file
|
|
@ -0,0 +1,201 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Give an environment its own role + database on the SHARED Postgres instance.
|
||||||
|
#
|
||||||
|
# sudo bash infra/deploy/db/provision-env-db.sh beta # run on vps2
|
||||||
|
#
|
||||||
|
# Since beta moved onto vps2 there is ONE TimescaleDB instance serving two
|
||||||
|
# environments. That is a capacity decision — one Postgres to tune, back up and
|
||||||
|
# keep on one extension build — and it is emphatically NOT a decision to let the
|
||||||
|
# two environments see each other's data. This script is what makes the second
|
||||||
|
# half true:
|
||||||
|
#
|
||||||
|
# * a role per environment (thermograph_beta), with its own password taken
|
||||||
|
# from that environment's own vault render,
|
||||||
|
# * a database per environment (thermograph_beta) OWNED by that role,
|
||||||
|
# * CONNECT revoked from PUBLIC on it, so the split is enforced by Postgres
|
||||||
|
# rather than by everyone remembering to use the right URL,
|
||||||
|
# * and NO superuser, NO CREATEDB, NO CREATEROLE on that role.
|
||||||
|
#
|
||||||
|
# The prod role keeps its own database and cannot be reached with beta's
|
||||||
|
# credentials; beta's role cannot connect to prod's database at all.
|
||||||
|
#
|
||||||
|
# Idempotent by construction — safe to re-run after a password rotation (it
|
||||||
|
# re-applies the password) or on a fresh instance. It never drops anything.
|
||||||
|
#
|
||||||
|
# WHY THIS IS NOT WIRED INTO deploy.sh: creating roles and databases is a
|
||||||
|
# privileged, once-per-environment act, and a deploy that can mint database
|
||||||
|
# roles is a deploy that can mint them wrongly at 3am. Run it by hand from the
|
||||||
|
# runbook when an environment is first stood up, and after a password rotation.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ENV_NAME="${1:?usage: provision-env-db.sh <env> (beta|prod)}"
|
||||||
|
|
||||||
|
SELF_DIR=$(cd "$(dirname "$0")" && pwd)
|
||||||
|
# shellcheck source=infra/deploy/env-topology.sh
|
||||||
|
. "$SELF_DIR/../env-topology.sh"
|
||||||
|
thermograph_topology "$ENV_NAME"
|
||||||
|
|
||||||
|
if [ "$TG_DEPLOY_MODE" != stack ]; then
|
||||||
|
echo "!! $ENV_NAME does not use the shared instance (dev keeps its own db container)" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
# REFUSE to run for the environment that OWNS the instance.
|
||||||
|
#
|
||||||
|
# This script provisions a GUEST environment onto someone else's Postgres. Run
|
||||||
|
# for prod, it would target prod's `thermograph` role — which is the instance's
|
||||||
|
# bootstrap superuser, not a guest — and the `ALTER ROLE ... NOSUPERUSER` below
|
||||||
|
# would strip superuser from the role the whole instance is administered with.
|
||||||
|
# Prod's roles predate this script and are created by the container's own
|
||||||
|
# initdb; there is nothing here for prod to need.
|
||||||
|
if [ "$TG_DB_SERVICE" = "${TG_STACK_NAME}_db" ]; then
|
||||||
|
echo "!! '$ENV_NAME' OWNS this Postgres instance (${TG_DB_SERVICE} is its own stack's db)." >&2
|
||||||
|
echo "!! This script provisions a guest environment onto a shared instance." >&2
|
||||||
|
echo "!! Running it here would demote ${TG_DB_USER} from superuser. Refusing." >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
# The database SERVER is prod's, wherever we are provisioning FOR.
|
||||||
|
DB_CID=$(docker ps -q --filter "label=com.docker.swarm.service.name=${TG_DB_SERVICE}" | head -1)
|
||||||
|
if [ -z "$DB_CID" ]; then
|
||||||
|
echo "!! no running task for ${TG_DB_SERVICE} on this host" >&2
|
||||||
|
echo "!! run this on vps2, with prod's stack up." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# The new role's password is whatever that environment's vault says it is, so
|
||||||
|
# the database and the app can never disagree about it. Read from the rendered
|
||||||
|
# env file (a deploy of that environment writes it) rather than from sops here:
|
||||||
|
# this script should not need the age key.
|
||||||
|
if [ ! -r "$TG_ENV_FILE" ]; then
|
||||||
|
PW=$(sudo grep -m1 '^POSTGRES_PASSWORD=' "$TG_ENV_FILE" 2>/dev/null | cut -d= -f2- || true)
|
||||||
|
else
|
||||||
|
PW=$(grep -m1 '^POSTGRES_PASSWORD=' "$TG_ENV_FILE" | cut -d= -f2- || true)
|
||||||
|
fi
|
||||||
|
if [ -z "${PW:-}" ]; then
|
||||||
|
echo "!! no POSTGRES_PASSWORD in $TG_ENV_FILE" >&2
|
||||||
|
echo "!! deploy $ENV_NAME once first so the vault renders it, then re-run." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Provisioning role/database '${TG_DB_USER}'/'${TG_DB_NAME}' on ${TG_DB_SERVICE}"
|
||||||
|
|
||||||
|
# The bootstrap superuser of the instance is prod's POSTGRES_USER (`thermograph`),
|
||||||
|
# and psql runs inside the container over its local socket — the database is
|
||||||
|
# never exposed on a TCP port anyone outside the overlay can reach.
|
||||||
|
#
|
||||||
|
# The password reaches psql as an environment variable rather than a -v
|
||||||
|
# argument, so it is not in psql's argv inside the container. It IS briefly in
|
||||||
|
# the `docker exec` argv on the host; that is visible only to root on vps2, who
|
||||||
|
# can read the vault render anyway. It is never interpolated into SQL text:
|
||||||
|
# :'pw' is psql's quote-and-escape form, which is also what makes a password
|
||||||
|
# containing a quote safe.
|
||||||
|
#
|
||||||
|
# Note both statements below are generated and run via \gexec rather than a
|
||||||
|
# DO block. psql does NOT substitute :variables inside dollar-quoted strings,
|
||||||
|
# so a DO $$ ... :'role' ... $$ body would be sent to the server literally.
|
||||||
|
docker exec -i -e ENV_DB_PASSWORD="$PW" "$DB_CID" \
|
||||||
|
psql -v ON_ERROR_STOP=1 -U thermograph -d postgres \
|
||||||
|
-v role="$TG_DB_USER" -v dbname="$TG_DB_NAME" <<'SQL'
|
||||||
|
\set pw `echo "$ENV_DB_PASSWORD"`
|
||||||
|
|
||||||
|
-- Role: create if absent.
|
||||||
|
SELECT format('CREATE ROLE %I LOGIN', :'role')
|
||||||
|
WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'role')
|
||||||
|
\gexec
|
||||||
|
|
||||||
|
-- Always (re)apply the password and the negative privileges, so a vault
|
||||||
|
-- rotation is just "rotate, redeploy, re-run this" — and so a role that was
|
||||||
|
-- created by hand with more rights than it should have gets corrected.
|
||||||
|
ALTER ROLE :"role" WITH LOGIN PASSWORD :'pw' NOSUPERUSER NOCREATEDB NOCREATEROLE;
|
||||||
|
|
||||||
|
-- CREATE DATABASE cannot run inside a transaction block, hence \gexec here too.
|
||||||
|
SELECT format('CREATE DATABASE %I OWNER %I', :'dbname', :'role')
|
||||||
|
WHERE NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = :'dbname')
|
||||||
|
\gexec
|
||||||
|
SQL
|
||||||
|
|
||||||
|
# Extension + privileges have to run INSIDE the new database, hence a second
|
||||||
|
# connection. CREATE EXTENSION needs superuser, which is why it is done here
|
||||||
|
# rather than left to the app's own migration to attempt as the env role.
|
||||||
|
docker exec -i "$DB_CID" \
|
||||||
|
psql -v ON_ERROR_STOP=1 -U thermograph -d "$TG_DB_NAME" \
|
||||||
|
-v role="$TG_DB_USER" -v dbname="$TG_DB_NAME" -v rorole="${TG_DB_USER}_ro" <<'SQL'
|
||||||
|
CREATE EXTENSION IF NOT EXISTS timescaledb;
|
||||||
|
|
||||||
|
-- Nobody but this environment's roles connects to this database. Without this,
|
||||||
|
-- PUBLIC retains CONNECT and any role on the instance could read it.
|
||||||
|
REVOKE CONNECT ON DATABASE :"dbname" FROM PUBLIC;
|
||||||
|
GRANT CONNECT ON DATABASE :"dbname" TO :"role";
|
||||||
|
|
||||||
|
-- The role owns the database but not necessarily the public schema, which is
|
||||||
|
-- owned by the bootstrap superuser on a fresh database; Alembic needs to create
|
||||||
|
-- tables in it.
|
||||||
|
ALTER SCHEMA public OWNER TO :"role";
|
||||||
|
|
||||||
|
-- The read-only role that ad-hoc queries use (ops/dbq.sh). Every environment
|
||||||
|
-- gets one, named <role>_ro, so a human or an agent poking at data cannot
|
||||||
|
-- write. Without it, beta's queries would have had to connect as the OWNER —
|
||||||
|
-- read-write on its own database — losing a guarantee prod and dev already had
|
||||||
|
-- purely because beta was newer.
|
||||||
|
SELECT format('CREATE ROLE %I LOGIN', :'rorole')
|
||||||
|
WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'rorole')
|
||||||
|
\gexec
|
||||||
|
ALTER ROLE :"rorole" WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE;
|
||||||
|
|
||||||
|
GRANT CONNECT ON DATABASE :"dbname" TO :"rorole";
|
||||||
|
GRANT USAGE ON SCHEMA public TO :"rorole";
|
||||||
|
GRANT SELECT ON ALL TABLES IN SCHEMA public TO :"rorole";
|
||||||
|
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO :"rorole";
|
||||||
|
-- Future tables too: without this, every new migration would create a table the
|
||||||
|
-- read-only role cannot see, and the omission would only surface as a confusing
|
||||||
|
-- "permission denied" months later.
|
||||||
|
ALTER DEFAULT PRIVILEGES FOR ROLE :"role" IN SCHEMA public
|
||||||
|
GRANT SELECT ON TABLES TO :"rorole";
|
||||||
|
ALTER DEFAULT PRIVILEGES FOR ROLE :"role" IN SCHEMA public
|
||||||
|
GRANT SELECT ON SEQUENCES TO :"rorole";
|
||||||
|
SQL
|
||||||
|
|
||||||
|
# --- harden the OWNER environment's database too -------------------------------
|
||||||
|
#
|
||||||
|
# Revoking CONNECT from PUBLIC on the guest's own database is only half the job,
|
||||||
|
# and the missing half is the half that matters. A fresh Postgres database
|
||||||
|
# carries `=Tc` for PUBLIC — TEMP *and* CONNECT — so the moment a second role
|
||||||
|
# exists on the instance, that role can connect to every database that still has
|
||||||
|
# the default. Provisioning beta and stopping here left `thermograph_beta` able
|
||||||
|
# to open a session against prod's database; caught live by the runbook's own
|
||||||
|
# isolation check, which is why that check exists.
|
||||||
|
#
|
||||||
|
# Order matters: grant the owner's read-only role an EXPLICIT connect first, so
|
||||||
|
# the revoke below cannot strip the ad-hoc query path (ops/dbq.sh) of its access.
|
||||||
|
# The owner's app role is the database owner and a superuser, so it is unaffected
|
||||||
|
# either way.
|
||||||
|
OWNER_DB=$(
|
||||||
|
# shellcheck disable=SC1090 # sourced above; re-resolving prod in a subshell
|
||||||
|
thermograph_topology prod >/dev/null 2>&1
|
||||||
|
printf '%s' "$TG_DB_NAME"
|
||||||
|
)
|
||||||
|
OWNER_RO=$(
|
||||||
|
thermograph_topology prod >/dev/null 2>&1
|
||||||
|
printf '%s_ro' "$TG_DB_USER"
|
||||||
|
)
|
||||||
|
# Restore this run's environment — thermograph_topology overwrote the TG_* vars.
|
||||||
|
thermograph_topology "$ENV_NAME"
|
||||||
|
|
||||||
|
if [ "$OWNER_DB" != "$TG_DB_NAME" ]; then
|
||||||
|
echo "==> Hardening the owner database '${OWNER_DB}' against PUBLIC connects"
|
||||||
|
docker exec -i "$DB_CID" \
|
||||||
|
psql -v ON_ERROR_STOP=1 -U thermograph -d postgres \
|
||||||
|
-v ownerdb="$OWNER_DB" -v ownerro="$OWNER_RO" <<'SQL'
|
||||||
|
-- Only if that read-only role actually exists (it predates this script).
|
||||||
|
SELECT format('GRANT CONNECT ON DATABASE %I TO %I', :'ownerdb', :'ownerro')
|
||||||
|
WHERE EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'ownerro')
|
||||||
|
\gexec
|
||||||
|
|
||||||
|
REVOKE CONNECT ON DATABASE :"ownerdb" FROM PUBLIC;
|
||||||
|
SQL
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> OK: ${TG_DB_USER} owns ${TG_DB_NAME} (timescaledb enabled, PUBLIC revoked)"
|
||||||
|
echo " Verify isolation:"
|
||||||
|
echo " docker exec $DB_CID psql -U ${TG_DB_USER} -d thermograph -c 'select 1' # must FAIL"
|
||||||
|
|
@ -1,41 +1,54 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# LAN-dev deploy: roll the per-service registry-pull stack (see deploy.sh) onto
|
# Dev deploy: roll the per-service registry-pull stack (see deploy.sh) onto the
|
||||||
# the ~/thermograph-dev overlay instead of prod/beta's loopback-only stack.
|
# dev compose overlay instead of beta/prod's Swarm stacks.
|
||||||
#
|
#
|
||||||
# This script assumes $APP_DIR is a checkout of the monorepo (deploy assets under infra/),
|
# Dev lives on vps1 now, at /opt/thermograph-dev — not on the operator's desktop.
|
||||||
# the same way /opt/thermograph is on prod/beta. That is the live state: the LAN
|
# The desktop hosts no Thermograph environment any more; it runs the AI models
|
||||||
# box's ~/thermograph-dev was reprovisioned as an infra checkout during the
|
# and offers flex Swarm capacity. What moved is only WHERE and HOW dev is
|
||||||
# 2026-07-22 cutover (the app monorepo is archived), provision-dev-lan.sh's
|
# reached: it is a normal fleet host now (SSH deploy from CI, SOPS render at
|
||||||
# REPO_URL defaults to this repo, and the app repos' deploy-dev.yml workflows
|
# /etc/thermograph.env, mesh-only exposure on 10.10.0.2:8137), rather than a
|
||||||
# invoke this script on the thermograph-lan runner. This IS the live LAN-dev
|
# sudo-free systemd --user stack on someone's Wi-Fi.
|
||||||
# deploy path.
|
#
|
||||||
|
# $APP_DIR is a checkout of the monorepo (deploy assets under infra/), the same
|
||||||
|
# way /opt/thermograph is for prod. A laptop can still point APP_DIR at a
|
||||||
|
# checkout under $HOME and get the old local behaviour.
|
||||||
#
|
#
|
||||||
# Design: a thin wrapper around deploy.sh, not a duplicate. deploy.sh already owns
|
# Design: a thin wrapper around deploy.sh, not a duplicate. deploy.sh already owns
|
||||||
# the entire per-service registry-pull mechanism -- secrets sourcing, docker login,
|
# the entire per-service registry-pull mechanism -- secrets sourcing, docker login,
|
||||||
# the retry-pull loop, --no-deps single-service rolls vs. --remove-orphans `all`,
|
# the retry-pull loop, --no-deps single-service rolls vs. --remove-orphans `all`,
|
||||||
# .image-tags.env persistence, and the 8137/8080 health checks. None of that is
|
# .image-tags.env persistence, and the 8137/8080 health checks. None of that is
|
||||||
# dev-specific; the only things LAN dev actually changes are WHERE it deploys
|
# dev-specific; the only things dev actually changes are WHERE it deploys (a
|
||||||
# (a separate checkout + branch) and WHICH compose files are in play (the base
|
# separate checkout + branch), WHICH compose files are in play (the base file
|
||||||
# file plus docker-compose.dev.yml's uncapped/LAN-published overrides). Both are
|
# plus docker-compose.dev.yml's uncapped/mesh-published overrides), and its
|
||||||
# expressible as environment (APP_DIR/BRANCH that deploy.sh already reads, and
|
# secrets policy (below). All are expressible as environment (APP_DIR/BRANCH
|
||||||
# docker compose's own COMPOSE_FILE variable), so re-exec'ing deploy.sh with that
|
# that deploy.sh already reads, and docker compose's own COMPOSE_FILE
|
||||||
# environment set covers it with no forked copy of the pull/roll/health logic to
|
# variable), so re-exec'ing deploy.sh with that environment set covers it with
|
||||||
# drift out of sync. If LAN dev ever needs deploy logic that genuinely diverges
|
# no forked copy of the pull/roll/health logic to drift out of sync. If dev ever
|
||||||
# from prod/beta (not just "different files/directory"), promote this to a
|
# needs deploy logic that genuinely diverges from beta/prod (not just "different
|
||||||
# standalone script at that point rather than growing special cases into deploy.sh.
|
# files/directory"), promote this to a standalone script at that point rather
|
||||||
|
# than growing special cases into deploy.sh.
|
||||||
#
|
#
|
||||||
# Usage, mirrors deploy.sh directly:
|
# Usage, mirrors deploy.sh directly:
|
||||||
# # roll just the backend onto a dev-tagged image:
|
# # roll just the backend onto a dev-tagged image:
|
||||||
# ssh dev-box 'SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> deploy/deploy-dev.sh'
|
# ssh vps1 'SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph-dev/infra/deploy/deploy-dev.sh'
|
||||||
# # bring the whole dev stack up (both tags required, same as deploy.sh):
|
# # bring the whole dev stack up (both tags required, same as deploy.sh):
|
||||||
# ssh dev-box 'SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> deploy/deploy-dev.sh'
|
# ssh vps1 'SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> /opt/thermograph-dev/infra/deploy/deploy-dev.sh'
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
# Dev context: a separate checkout + branch from prod/beta's /opt/thermograph
|
# Dev context: its own checkout and its own branch, on its own host (vps1).
|
||||||
# (main), so a dev deploy never touches or is touched by the prod/beta one.
|
# Both come from deploy/env-topology.sh so there is one place that says where
|
||||||
APP_DIR="${APP_DIR:-$HOME/thermograph-dev}"
|
# dev lives. $APP_DIR is still honoured when set explicitly — that is how a
|
||||||
BRANCH="${BRANCH:-dev}"
|
# laptop points this at a checkout under $HOME.
|
||||||
|
_dev_self="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
# shellcheck source=infra/deploy/env-topology.sh
|
||||||
|
. "$_dev_self/env-topology.sh"
|
||||||
|
thermograph_topology dev
|
||||||
|
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
|
||||||
|
BRANCH="${BRANCH:-$TG_BRANCH}"
|
||||||
export APP_DIR BRANCH
|
export APP_DIR BRANCH
|
||||||
|
# deploy.sh resolves the environment itself; say so explicitly rather than
|
||||||
|
# letting it fall back to a host marker that, on vps1, names dev anyway.
|
||||||
|
export THERMOGRAPH_ENV=dev
|
||||||
|
|
||||||
# Point every `docker compose` invocation inside deploy.sh at the LAN-dev overlay
|
# Point every `docker compose` invocation inside deploy.sh at the LAN-dev overlay
|
||||||
# (uncapped CPU, backend published on 0.0.0.0:8137, frontend unpublished -- see
|
# (uncapped CPU, backend published on 0.0.0.0:8137, frontend unpublished -- see
|
||||||
|
|
@ -72,23 +85,33 @@ export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-thermograph-dev}"
|
||||||
# genuinely needs, and nothing else.
|
# genuinely needs, and nothing else.
|
||||||
export THERMOGRAPH_SECRETS_SKIP_COMMON=1
|
export THERMOGRAPH_SECRETS_SKIP_COMMON=1
|
||||||
|
|
||||||
# (1b) Docker on this box is the snap package, confined to $HOME (plus a short
|
# (1b) Snap-packaged Docker is confined to $HOME (plus a short allowlist) and
|
||||||
# allowlist) -- it silently can't see /etc/thermograph.env at all, so
|
# silently can't see /etc/thermograph.env at all, so env_file: /etc/thermograph.env
|
||||||
# env_file: /etc/thermograph.env resolves to nothing for every container no
|
# resolves to nothing for every container no matter how correctly it's rendered.
|
||||||
# matter how correctly it's rendered. Mirror the render to a path under
|
# Where that is the case, mirror the render to a path under $APP_DIR;
|
||||||
# $APP_DIR too; docker-compose.dev.yml points env_file at this copy instead of
|
# docker-compose.dev.yml carries a matching optional env_file entry.
|
||||||
# /etc/thermograph.env. Untracked (see infra/.gitignore); re-rendered on every
|
#
|
||||||
# deploy, never committed.
|
# CONDITIONAL, not unconditional: this writes a plaintext copy of the render
|
||||||
export THERMOGRAPH_SECRETS_ENV_FILE_MIRROR="$APP_DIR/infra/deploy/dev-secrets.env"
|
# into the checkout, and on vps1 (ordinary apt Docker, which reads /etc fine)
|
||||||
|
# that copy would be pure liability. Detect the confinement rather than add a
|
||||||
|
# knob someone has to remember — the docker binary resolving under /snap is
|
||||||
|
# exactly the condition that breaks the /etc path.
|
||||||
|
_docker_bin="$(command -v docker || true)"
|
||||||
|
if [ -n "$_docker_bin" ] && case "$(readlink -f "$_docker_bin")" in /snap/*) true ;; *) false ;; esac; then
|
||||||
|
export THERMOGRAPH_SECRETS_ENV_FILE_MIRROR="$APP_DIR/infra/deploy/dev-secrets.env"
|
||||||
|
fi
|
||||||
|
|
||||||
# (2) The age key is read from wherever it is READABLE, not necessarily
|
# (2) The age key is read from wherever it is READABLE, not necessarily
|
||||||
# /etc/thermograph/age.key. render-secrets.sh falls back to `sudo cat` for a
|
# /etc/thermograph/age.key. render-secrets.sh falls back to `sudo cat` for a
|
||||||
# root-owned 0400 key, and this box has no passwordless sudo -- under the Forgejo
|
# root-owned 0400 key, which needs passwordless sudo.
|
||||||
# runner (a systemd --user service, no tty) that sudo cannot prompt, so the
|
#
|
||||||
# fleet-standard placement would make every dev deploy fail at the render. Prefer
|
# On vps1 that is the normal case: the `agent` user has passwordless sudo and
|
||||||
# the standard path when it is readable; otherwise the operator's own keyring, which
|
# the key sits at the fleet-standard path, so this block does nothing. The
|
||||||
# is where dev's key already lives (this box is the machine `sops` is run on), so
|
# fallback exists for the laptop case — a checkout where the key lives only in
|
||||||
# provisioning dev needs no second copy of the estate's recovery root.
|
# the operator's own keyring (that machine is where `sops` is run), and where a
|
||||||
|
# non-interactive shell could not prompt for sudo even if the key were in /etc.
|
||||||
|
# Preferring the standard path when readable keeps vps1 on the fleet convention
|
||||||
|
# and means provisioning dev needs no second copy of the estate's recovery root.
|
||||||
if [ -z "${THERMOGRAPH_AGE_KEY:-}" ] && [ ! -r /etc/thermograph/age.key ] \
|
if [ -z "${THERMOGRAPH_AGE_KEY:-}" ] && [ ! -r /etc/thermograph/age.key ] \
|
||||||
&& [ -r "$HOME/.config/sops/age/keys.txt" ]; then
|
&& [ -r "$HOME/.config/sops/age/keys.txt" ]; then
|
||||||
export THERMOGRAPH_AGE_KEY="$HOME/.config/sops/age/keys.txt"
|
export THERMOGRAPH_AGE_KEY="$HOME/.config/sops/age/keys.txt"
|
||||||
|
|
|
||||||
|
|
@ -26,11 +26,59 @@
|
||||||
# two axes are independent and rarely change together.
|
# two axes are independent and rarely change together.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
APP_DIR="${APP_DIR:-/opt/thermograph}"
|
# --- which environment is this? ------------------------------------------------
|
||||||
|
# vps2 runs beta AND prod, so "the host" no longer answers this — the caller does,
|
||||||
|
# via THERMOGRAPH_ENV (the deploy workflow always passes it). A by-hand run on a
|
||||||
|
# single-environment box still falls back to the host marker, and a box with
|
||||||
|
# neither still defaults to prod's historical paths, so nothing about an existing
|
||||||
|
# single-env host changes.
|
||||||
|
#
|
||||||
|
# Resolved from the SCRIPT'S OWN LOCATION, not a hardcoded /opt/thermograph:
|
||||||
|
# invoking /opt/thermograph-beta/infra/deploy/deploy.sh must act on the beta
|
||||||
|
# checkout even if something in the environment says otherwise. That is also the
|
||||||
|
# check that catches the one genuinely dangerous mistake on a two-environment
|
||||||
|
# host — running prod's checkout with THERMOGRAPH_ENV=beta, or the reverse.
|
||||||
|
SELF_DIR=$(cd "$(dirname "$0")" && pwd)
|
||||||
|
SELF_APP_DIR=$(cd "$SELF_DIR/../.." && pwd)
|
||||||
|
|
||||||
|
# Guarded exactly like render-secrets.sh below: the deploy that INTRODUCES this
|
||||||
|
# file runs with a checkout that predates it (it arrives with the git reset
|
||||||
|
# further down, after which deploy.sh re-execs). Missing => keep the pre-split
|
||||||
|
# behaviour rather than fail.
|
||||||
|
if [ -f "$SELF_DIR/env-topology.sh" ]; then
|
||||||
|
# shellcheck source=infra/deploy/env-topology.sh
|
||||||
|
. "$SELF_DIR/env-topology.sh"
|
||||||
|
ENV_NAME=$(thermograph_env_name)
|
||||||
|
# Nothing to go on anywhere: this is a pre-split host whose paths are prod's.
|
||||||
|
[ -n "$ENV_NAME" ] || ENV_NAME=prod
|
||||||
|
thermograph_topology "$ENV_NAME"
|
||||||
|
if [ "$SELF_APP_DIR" != "$TG_APP_DIR" ] && [ -z "${APP_DIR:-}" ]; then
|
||||||
|
echo "!! environment/checkout mismatch: THERMOGRAPH_ENV=$ENV_NAME expects" >&2
|
||||||
|
echo "!! $TG_APP_DIR but this script lives in $SELF_APP_DIR." >&2
|
||||||
|
echo "!! On vps2 that means beta and prod have been crossed. Refusing to deploy." >&2
|
||||||
|
echo "!! If this checkout really is $ENV_NAME (a rehearsal copy, a relocated" >&2
|
||||||
|
echo "!! checkout), say so explicitly: APP_DIR=$SELF_APP_DIR ..." >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
|
||||||
|
BRANCH="${BRANCH:-$TG_BRANCH}"
|
||||||
|
ENV_FILE="$TG_ENV_FILE"
|
||||||
|
DEPLOY_MODE="$TG_DEPLOY_MODE"
|
||||||
|
[ "$TG_SKIP_COMMON" = 1 ] && export THERMOGRAPH_SECRETS_SKIP_COMMON=1
|
||||||
|
# The second pass (after the re-exec) must resolve the SAME environment, and
|
||||||
|
# the stack path needs it too.
|
||||||
|
export THERMOGRAPH_ENV="$ENV_NAME"
|
||||||
|
else
|
||||||
|
ENV_NAME=""
|
||||||
|
APP_DIR="${APP_DIR:-/opt/thermograph}"
|
||||||
|
BRANCH="${BRANCH:-main}"
|
||||||
|
ENV_FILE=/etc/thermograph.env
|
||||||
|
DEPLOY_MODE=""
|
||||||
|
fi
|
||||||
|
|
||||||
# Monorepo layout: git operations act on the checkout root ($APP_DIR); all
|
# Monorepo layout: git operations act on the checkout root ($APP_DIR); all
|
||||||
# compose files and deploy assets live under infra/.
|
# compose files and deploy assets live under infra/.
|
||||||
INFRA_DIR="$APP_DIR/infra"
|
INFRA_DIR="$APP_DIR/infra"
|
||||||
BRANCH="${BRANCH:-main}"
|
|
||||||
# Which service this deploy rolls: backend | frontend | all. Defaults to `all`
|
# Which service this deploy rolls: backend | frontend | all. Defaults to `all`
|
||||||
# (a full-stack bring-up) so a by-hand run with both tags still works; the
|
# (a full-stack bring-up) so a by-hand run with both tags still works; the
|
||||||
# per-repo deploy.yml workflows always pass an explicit single service.
|
# per-repo deploy.yml workflows always pass an explicit single service.
|
||||||
|
|
@ -69,16 +117,20 @@ fi
|
||||||
# after which deploy.sh re-execs), so a missing helper simply falls back to the
|
# after which deploy.sh re-execs), so a missing helper simply falls back to the
|
||||||
# existing /etc/thermograph.env. Then source it so a by-hand run interpolates the
|
# existing /etc/thermograph.env. Then source it so a by-hand run interpolates the
|
||||||
# same as the systemd unit does. See deploy/render-secrets.sh + deploy/secrets/.
|
# same as the systemd unit does. See deploy/render-secrets.sh + deploy/secrets/.
|
||||||
|
#
|
||||||
|
# $ENV_FILE, not a hardcoded /etc/thermograph.env: on vps2 prod renders
|
||||||
|
# prod.yaml there while beta renders beta.yaml to /etc/thermograph-beta.env.
|
||||||
|
# One file per environment, never shared.
|
||||||
if [ -f "$INFRA_DIR/deploy/render-secrets.sh" ]; then
|
if [ -f "$INFRA_DIR/deploy/render-secrets.sh" ]; then
|
||||||
# shellcheck source=infra/deploy/render-secrets.sh
|
# shellcheck source=infra/deploy/render-secrets.sh
|
||||||
. "$INFRA_DIR/deploy/render-secrets.sh"
|
. "$INFRA_DIR/deploy/render-secrets.sh"
|
||||||
render_thermograph_secrets "$INFRA_DIR"
|
render_thermograph_secrets "$INFRA_DIR" "$ENV_NAME" "$ENV_FILE"
|
||||||
fi
|
fi
|
||||||
# /etc/thermograph.env is rendered at deploy time from the SOPS vault — it
|
# $ENV_FILE is rendered at deploy time from the SOPS vault — it cannot exist at
|
||||||
# cannot exist at lint time, so don't ask shellcheck to follow it.
|
# lint time, so don't ask shellcheck to follow it.
|
||||||
set -a
|
set -a
|
||||||
# shellcheck source=/dev/null
|
# shellcheck source=/dev/null
|
||||||
. /etc/thermograph.env 2>/dev/null || true
|
. "$ENV_FILE" 2>/dev/null || true
|
||||||
set +a
|
set +a
|
||||||
|
|
||||||
# Pre-warm the ~750 city-page archives so /climate pages serve from cache and a
|
# Pre-warm the ~750 city-page archives so /climate pages serve from cache and a
|
||||||
|
|
@ -122,12 +174,18 @@ if [ -z "${DEPLOY_SH_REEXECED:-}" ]; then
|
||||||
exec "$0" "$@"
|
exec "$0" "$@"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Stack-mode routing: a host whose /etc/thermograph/deploy-mode says "stack"
|
# Stack-mode routing: prod and beta are both Swarm stacks, dev is compose. The
|
||||||
# (prod, after the Swarm cutover) deploys via the Swarm stack path instead of
|
# mode is a property of the ENVIRONMENT (env-topology.sh), not of the host --
|
||||||
# compose. Checked AFTER the reset+re-exec so the stack script is always the
|
# vps2 runs two stacks, and a host-wide /etc/thermograph/deploy-mode marker
|
||||||
# freshly-pulled one, and the SERVICE/tag contract passes through unchanged --
|
# cannot describe a host that runs more than one environment. The marker is
|
||||||
# the app repos' workflows never need to know which mode a host runs.
|
# still honoured when the topology file is absent (a checkout that predates it).
|
||||||
if [ "$(cat /etc/thermograph/deploy-mode 2>/dev/null || true)" = "stack" ]; then
|
# Checked AFTER the reset+re-exec so the stack script is always the freshly
|
||||||
|
# pulled one, and the SERVICE/tag contract passes through unchanged -- the
|
||||||
|
# deploy workflow never needs to know which mode an environment runs.
|
||||||
|
if [ -z "$DEPLOY_MODE" ]; then
|
||||||
|
DEPLOY_MODE=$(cat /etc/thermograph/deploy-mode 2>/dev/null || true)
|
||||||
|
fi
|
||||||
|
if [ "$DEPLOY_MODE" = "stack" ]; then
|
||||||
exec bash "$INFRA_DIR/deploy/stack/deploy-stack.sh"
|
exec bash "$INFRA_DIR/deploy/stack/deploy-stack.sh"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|
@ -137,6 +195,20 @@ fi
|
||||||
# rename the project and recreate the stack under a second name.
|
# rename the project and recreate the stack under a second name.
|
||||||
cd "$INFRA_DIR"
|
cd "$INFRA_DIR"
|
||||||
|
|
||||||
|
# Compose-mode environments (dev) carry their project name, file list and bind
|
||||||
|
# address in the topology table. Set only when not already in the environment,
|
||||||
|
# so deploy-dev.sh's own exports and a by-hand override both still win.
|
||||||
|
if [ -n "${TG_COMPOSE_PROJECT:-}" ]; then
|
||||||
|
export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-$TG_COMPOSE_PROJECT}"
|
||||||
|
export COMPOSE_FILE="${COMPOSE_FILE:-$TG_COMPOSE_FILE}"
|
||||||
|
fi
|
||||||
|
# Which address the dev overlay publishes on. Defaults to loopback in the
|
||||||
|
# compose file; dev on vps1 sets the mesh address here, because a public VPS
|
||||||
|
# must never publish an unreviewed branch's stack on 0.0.0.0.
|
||||||
|
if [ -n "${TG_BIND_ADDR:-}" ]; then
|
||||||
|
export DEV_BIND_ADDR="${DEV_BIND_ADDR:-$TG_BIND_ADDR}"
|
||||||
|
fi
|
||||||
|
|
||||||
# Registry-pull cutover: pull the image each app repo's build-push.yml already
|
# Registry-pull cutover: pull the image each app repo's build-push.yml already
|
||||||
# built and pushed, instead of building in place. This checkout is
|
# built and pushed, instead of building in place. This checkout is
|
||||||
# thermograph-infra, not an app repo, so there's no "current commit" to derive
|
# thermograph-infra, not an app repo, so there's no "current commit" to derive
|
||||||
|
|
@ -345,7 +417,21 @@ docker images --format '{{.Repository}}:{{.Tag}}' \
|
||||||
|
|
||||||
# Post-deploy warm/IndexNow only make sense once the backend is (re)deployed --
|
# Post-deploy warm/IndexNow only make sense once the backend is (re)deployed --
|
||||||
# they exec inside the backend container. Skip them on a frontend-only roll.
|
# they exec inside the backend container. Skip them on a frontend-only roll.
|
||||||
if [ "$SERVICE" = backend ] || [ "$SERVICE" = all ]; then
|
#
|
||||||
|
# TG_POST_DEPLOY gates them per ENVIRONMENT (env-topology.sh): prod does both,
|
||||||
|
# beta and dev do neither. Both are wrong outside prod. The warm spends the
|
||||||
|
# shared upstream archive quota to fill a cache nobody serves from, and the
|
||||||
|
# IndexNow ping announces URLs to Bing/DuckDuckGo/Yandex — from dev that means
|
||||||
|
# submitting a mesh address no crawler can reach.
|
||||||
|
#
|
||||||
|
# This gate existed in deploy/stack/deploy-stack.sh (beta, prod) but was missed
|
||||||
|
# here, on the compose path, which is the ONE environment where the default is
|
||||||
|
# wrong: dev. Standing dev up on vps1 duly warmed 1000 cities and tried to
|
||||||
|
# submit 14,007 URLs to IndexNow before this was fixed.
|
||||||
|
#
|
||||||
|
# Defaults to 1 when unset so a checkout predating env-topology.sh keeps prod's
|
||||||
|
# historical behaviour.
|
||||||
|
if { [ "$SERVICE" = backend ] || [ "$SERVICE" = all ]; } && [ "${TG_POST_DEPLOY:-1}" = 1 ]; then
|
||||||
warm_city_archives
|
warm_city_archives
|
||||||
ping_indexnow
|
ping_indexnow
|
||||||
fi
|
fi
|
||||||
|
|
|
||||||
264
infra/deploy/env-topology.sh
Normal file
264
infra/deploy/env-topology.sh
Normal file
|
|
@ -0,0 +1,264 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# WHERE EACH ENVIRONMENT LIVES — the single source of truth, sourced by every
|
||||||
|
# deploy path. Not executable on its own; `. env-topology.sh` then call
|
||||||
|
# `thermograph_topology <env>`.
|
||||||
|
#
|
||||||
|
# Why this file exists at all. Until the vps1/vps2 split, "which environment is
|
||||||
|
# this?" was answerable from the machine you were standing on: one host ran one
|
||||||
|
# environment, so a host-wide marker (/etc/thermograph/secrets-env) and a
|
||||||
|
# host-wide deploy mode (/etc/thermograph/deploy-mode) were enough, and every
|
||||||
|
# path could hardcode /opt/thermograph, /etc/thermograph.env, ports 8137/8080.
|
||||||
|
# vps2 now runs BOTH beta and prod. Every one of those assumptions becomes a
|
||||||
|
# collision: two checkouts, two rendered env files, two stacks, two loopback
|
||||||
|
# port pairs, two image-tag files, two deploy locks — on one box. So the
|
||||||
|
# environment is now an INPUT (THERMOGRAPH_ENV, passed by the deploy caller),
|
||||||
|
# and everything else is derived here rather than re-guessed per script.
|
||||||
|
#
|
||||||
|
# The host markers survive as the fallback for a by-hand run with no explicit
|
||||||
|
# env (`deploy.sh` on prod still means prod), but they are no longer the thing
|
||||||
|
# that decides where files go.
|
||||||
|
#
|
||||||
|
# THE ESTATE
|
||||||
|
#
|
||||||
|
# vps1 75.119.132.91 10.10.0.2 "operational programs"
|
||||||
|
# Forgejo (git + CI + registry), Grafana/Loki/Alloy, emigriffith.dev,
|
||||||
|
# and the DEV environment with its own Postgres. Nothing here is
|
||||||
|
# reachable by an external user except git/dashboard/the portfolio site;
|
||||||
|
# dev itself is mesh-only, deliberately (see DEV_BIND_ADDR below).
|
||||||
|
#
|
||||||
|
# vps2 169.58.46.181 10.10.0.1 "the deployed environment"
|
||||||
|
# Everything an external user can touch: prod (thermograph.org) AND beta
|
||||||
|
# (beta.thermograph.org), Centralis, Postfix, the backups, and the ONE
|
||||||
|
# TimescaleDB instance that serves both environments on separate
|
||||||
|
# databases with separate roles.
|
||||||
|
#
|
||||||
|
# desktop 10.10.0.3 operator's box
|
||||||
|
# AI model hosting (voice-to-text, the LLM behind upcoming features) and
|
||||||
|
# flex capacity as a Swarm worker. It hosts NO Thermograph environment —
|
||||||
|
# that is the whole point of the vps1/vps2 split. A laptop-style
|
||||||
|
# `make dev-up` still works locally; it is just not the dev server.
|
||||||
|
#
|
||||||
|
# WHY BETA SITS NEXT TO PROD RATHER THAN NEXT TO DEV
|
||||||
|
#
|
||||||
|
# Beta is a pre-production rehearsal, so what it needs to rehearse is prod's
|
||||||
|
# environment: the same orchestrator (Swarm, not compose), the same Postgres
|
||||||
|
# major and extension build, the same Caddy in front, the same mail path, the
|
||||||
|
# same mesh position. Co-locating beta with dev bought resemblance to the thing
|
||||||
|
# it is NOT trying to predict. Co-locating it with prod means a beta green light
|
||||||
|
# is evidence about prod. The cost — a bad beta deploy shares a machine with
|
||||||
|
# prod — is bounded by the per-environment isolation this file defines: separate
|
||||||
|
# stacks, separate volumes, separate DB roles, separate CPU limits.
|
||||||
|
|
||||||
|
# thermograph_topology <env>
|
||||||
|
#
|
||||||
|
# Exports the TG_* variables describing that environment. Every value is a
|
||||||
|
# derived fact about the estate, not a preference: change one here and the
|
||||||
|
# deploy scripts, the stack files and the runbooks all follow.
|
||||||
|
thermograph_topology() {
|
||||||
|
local env_name="${1:?thermograph_topology needs an environment: dev|beta|prod}"
|
||||||
|
|
||||||
|
# Reset first: this function is sourced into long-lived shells (deploy.sh
|
||||||
|
# sources it before and after its self re-exec), and a stale TG_STACK_NAME
|
||||||
|
# from a previous call would silently target the wrong stack.
|
||||||
|
TG_ENV=""; TG_HOST=""; TG_APP_DIR=""; TG_BRANCH=""; TG_DEPLOY_MODE=""
|
||||||
|
TG_STACK_NAME=""; TG_STACK_FILE=""; TG_LB_CONFIG=""
|
||||||
|
TG_COMPOSE_PROJECT=""; TG_COMPOSE_FILE=""
|
||||||
|
TG_ENV_FILE=""; TG_STACK_ENV_FILE=""; TG_LB_NAME=""
|
||||||
|
TG_LB_HTTP_PORT=""; TG_LB_FE_PORT=""; TG_DB_NAME=""; TG_DB_USER=""
|
||||||
|
TG_TAGS_FILE=""; TG_LOCK_FILE=""; TG_BIND_ADDR=""; TG_SKIP_COMMON=0
|
||||||
|
TG_SVC_PREFIX=""; TG_DATA_NETWORK=""; TG_DB_SERVICE=""; TG_POST_DEPLOY=0
|
||||||
|
TG_SSH_HOST=""; TG_SSH_TARGET=""
|
||||||
|
# Which store render-secrets.sh reads: sops | openbao. Defaults to sops for every
|
||||||
|
# environment and is overridden per environment below, so a cutover is a one-line
|
||||||
|
# reviewed change that follows the estate's own dev -> main -> release promotion.
|
||||||
|
#
|
||||||
|
# This lives here, and NOT in a host marker file like /etc/thermograph/deploy-mode,
|
||||||
|
# for the reason that marker was demoted to a fallback in the first place: vps2 runs
|
||||||
|
# prod and beta side by side, and one host-wide file cannot name two backends.
|
||||||
|
TG_SECRETS_BACKEND=sops
|
||||||
|
# Which AppRole credential file render-secrets-openbao.sh authenticates with. Same
|
||||||
|
# reasoning as TG_SECRETS_BACKEND above, and the same trap: vps2 renders TWO
|
||||||
|
# environments, so a single host-wide default cannot serve both. prod and dev each
|
||||||
|
# get the conventional path (dev is alone on vps1, so there is no collision there);
|
||||||
|
# beta is the one that must differ, because it shares a filesystem with prod.
|
||||||
|
#
|
||||||
|
# Without this, beta's render falls back to prod's credentials and tg-host-prod.hcl
|
||||||
|
# correctly denies thermograph/data/env/beta — a 403 at deploy time on the box that
|
||||||
|
# runs prod.
|
||||||
|
TG_BAO_APPROLE=/etc/thermograph/openbao-approle
|
||||||
|
|
||||||
|
case "$env_name" in
|
||||||
|
prod)
|
||||||
|
TG_ENV=prod
|
||||||
|
TG_HOST=vps2
|
||||||
|
# Unchanged from before the split — prod keeps every path it already has,
|
||||||
|
# so nothing about the live prod host moves during the cutover.
|
||||||
|
TG_SSH_HOST=169.58.46.181
|
||||||
|
TG_SSH_TARGET=agent@169.58.46.181
|
||||||
|
TG_APP_DIR=/opt/thermograph
|
||||||
|
TG_BRANCH=main
|
||||||
|
TG_DEPLOY_MODE=stack
|
||||||
|
TG_STACK_NAME=thermograph
|
||||||
|
TG_STACK_FILE=deploy/stack/thermograph-stack.yml
|
||||||
|
TG_ENV_FILE=/etc/thermograph.env
|
||||||
|
TG_STACK_ENV_FILE=/etc/thermograph/stack.env
|
||||||
|
TG_LB_NAME=thermograph-lb
|
||||||
|
TG_LB_CONFIG=deploy/stack/lb/Caddyfile
|
||||||
|
TG_LB_HTTP_PORT=8137
|
||||||
|
TG_LB_FE_PORT=8080
|
||||||
|
TG_DB_NAME=thermograph
|
||||||
|
TG_DB_USER=thermograph
|
||||||
|
# Prod's services keep their bare names (web, worker, frontend, ...): the
|
||||||
|
# cutover must not touch prod's stack file, its env or its LB config.
|
||||||
|
TG_SVC_PREFIX=""
|
||||||
|
# Where the database lives. Prod's own overlay, which is `attachable` and
|
||||||
|
# which beta joins as an external network.
|
||||||
|
TG_DATA_NETWORK=thermograph_internal
|
||||||
|
TG_DB_SERVICE=thermograph_db
|
||||||
|
# Post-deploy city warm + IndexNow ping. Prod only — see the beta note.
|
||||||
|
TG_POST_DEPLOY=1
|
||||||
|
;;
|
||||||
|
beta)
|
||||||
|
TG_ENV=beta
|
||||||
|
TG_HOST=vps2
|
||||||
|
# Same box, same public IP as prod — beta and prod are two stacks on one
|
||||||
|
# Swarm manager now, not two separate hosts. A beta SSH target is
|
||||||
|
# therefore prod's blast radius too (see .claude/hooks/prod-guard.sh).
|
||||||
|
TG_SSH_HOST=169.58.46.181
|
||||||
|
TG_SSH_TARGET=agent@169.58.46.181
|
||||||
|
# The one environment that cannot use the default AppRole path: it shares a
|
||||||
|
# filesystem with prod, so it needs its own credential file to authenticate as
|
||||||
|
# tg-beta rather than tg-prod. bootstrap-policies.sh installs it here.
|
||||||
|
TG_BAO_APPROLE=/etc/thermograph/openbao-approle-beta
|
||||||
|
# A SECOND checkout on the same box. Separate from prod's so the two can
|
||||||
|
# sit on different commits of this repo, and so `git reset --hard` in one
|
||||||
|
# deploy can never yank the tree out from under the other's running
|
||||||
|
# deploy. Everything below is likewise a distinct name/port/path from
|
||||||
|
# prod's, on purpose: co-location is only safe if nothing is shared by
|
||||||
|
# accident. The one deliberate exception is the database SERVER (see
|
||||||
|
# TG_DB_* — separate role and database on a shared instance).
|
||||||
|
TG_APP_DIR=/opt/thermograph-beta
|
||||||
|
TG_BRANCH=main
|
||||||
|
TG_DEPLOY_MODE=stack
|
||||||
|
TG_STACK_NAME=thermograph-beta
|
||||||
|
TG_STACK_FILE=deploy/stack/thermograph-beta-stack.yml
|
||||||
|
TG_ENV_FILE=/etc/thermograph-beta.env
|
||||||
|
TG_STACK_ENV_FILE=/etc/thermograph/beta-stack.env
|
||||||
|
TG_LB_NAME=thermograph-beta-lb
|
||||||
|
TG_LB_CONFIG=deploy/stack/lb/Caddyfile.beta
|
||||||
|
# NOT 8137/8080: prod's loopback LB already owns those on this host.
|
||||||
|
TG_LB_HTTP_PORT=8237
|
||||||
|
TG_LB_FE_PORT=8180
|
||||||
|
TG_DB_NAME=thermograph_beta
|
||||||
|
# Its own role, not prod's `thermograph` superuser-of-its-own-database.
|
||||||
|
# One instance is a capacity decision; it is not a decision to let a beta
|
||||||
|
# deploy running an unmerged branch read or write the production database.
|
||||||
|
TG_DB_USER=thermograph_beta
|
||||||
|
# Beta's Swarm services are named beta-web, beta-worker, ... Swarm
|
||||||
|
# registers a service's SHORT name as a DNS alias on every network it
|
||||||
|
# joins, so two stacks that both call a service `web` on one shared
|
||||||
|
# network make `web` ambiguous — prod's frontend could resolve a beta
|
||||||
|
# task. Prefixing beta's names removes the collision without editing a
|
||||||
|
# single line of prod's stack.
|
||||||
|
TG_SVC_PREFIX="beta-"
|
||||||
|
# Beta has no db service of its own: it joins prod's overlay (declared
|
||||||
|
# `external` in its stack file) purely to reach `db`, and keeps its own
|
||||||
|
# `internal` overlay for beta-to-beta traffic.
|
||||||
|
TG_DATA_NETWORK=thermograph_internal
|
||||||
|
TG_DB_SERVICE=thermograph_db
|
||||||
|
# No warm, no IndexNow on beta. IndexNow announces URLs to Bing/DDG/
|
||||||
|
# Yandex — from beta that means asking search engines to index
|
||||||
|
# beta.thermograph.org, which is the opposite of what beta is for. The
|
||||||
|
# city warm is worse than useless here: it spends the shared upstream
|
||||||
|
# archive quota to fill a cache that only a rehearsal environment reads.
|
||||||
|
# (Both ran on beta under compose; that was a side effect of beta and
|
||||||
|
# prod sharing one script, not a decision.)
|
||||||
|
TG_POST_DEPLOY=0
|
||||||
|
;;
|
||||||
|
dev)
|
||||||
|
TG_ENV=dev
|
||||||
|
TG_HOST=vps1
|
||||||
|
# vps1 — Forgejo/Grafana/dev, mesh-only for dev itself (see TG_BIND_ADDR).
|
||||||
|
TG_SSH_HOST=75.119.132.91
|
||||||
|
TG_SSH_TARGET=agent@75.119.132.91
|
||||||
|
TG_APP_DIR=/opt/thermograph-dev
|
||||||
|
# The only environment that tracks `dev`; beta and prod both run this
|
||||||
|
# repo's `main` (infra is not environment-staged — app code is, via image
|
||||||
|
# tags). See the branch model in infra/README.md.
|
||||||
|
TG_BRANCH=dev
|
||||||
|
# Compose, not Swarm: dev's value is a fast, legible, single-box stack
|
||||||
|
# you can `docker compose logs` at. It is the one environment not
|
||||||
|
# pretending to be prod.
|
||||||
|
TG_DEPLOY_MODE=compose
|
||||||
|
TG_COMPOSE_PROJECT=thermograph-dev
|
||||||
|
TG_COMPOSE_FILE="docker-compose.yml:docker-compose.dev.yml"
|
||||||
|
TG_ENV_FILE=/etc/thermograph.env
|
||||||
|
# Dev keeps its OWN Postgres container (a `db` service in its compose
|
||||||
|
# project). It is not on the shared instance and must not be: the shared
|
||||||
|
# instance lives on vps2 and holds real user data.
|
||||||
|
TG_DB_NAME=thermograph
|
||||||
|
TG_DB_USER=thermograph
|
||||||
|
# LOOPBACK ONLY. vps1 is a public VPS and dev runs whatever branch is in
|
||||||
|
# flight, including unreviewed ones, so the stack must never publish on
|
||||||
|
# 0.0.0.0 the way it did on the operator's LAN box.
|
||||||
|
#
|
||||||
|
# It was briefly mesh-only (10.10.0.2). It is now fronted by Caddy at
|
||||||
|
# dev.thermograph.org — TLS, HTTP basic auth, and X-Robots-Tag: noindex
|
||||||
|
# (see deploy/Caddyfile.vps1) — so the only reachable path is through that
|
||||||
|
# site block, and binding loopback is what guarantees it: nothing can
|
||||||
|
# reach the app without passing the password, not even from the mesh.
|
||||||
|
TG_BIND_ADDR=127.0.0.1
|
||||||
|
# Dev NEVER layers common.yaml — that file is the internet-facing fleet's
|
||||||
|
# shared production credential set (both S3 keypairs, the VAPID signing
|
||||||
|
# key, REGISTRY_TOKEN, the metrics token), and dev runs whatever branch is
|
||||||
|
# in flight on the same box as Forgejo and its CI runner. This lives here
|
||||||
|
# rather than only in deploy-dev.sh so the protection is a property of the
|
||||||
|
# ENVIRONMENT, not of which entry point happened to be used. See the long
|
||||||
|
# note in render-secrets.sh.
|
||||||
|
TG_SKIP_COMMON=1
|
||||||
|
TG_DB_SERVICE=db
|
||||||
|
# Dev warms nothing and pings nothing: it must never spend the upstream
|
||||||
|
# archive quota, and it must never tell a search engine it exists.
|
||||||
|
TG_POST_DEPLOY=0
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "!! unknown environment '$env_name' (expected dev|beta|prod)" >&2
|
||||||
|
return 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# Derived, never hand-set: keeping these next to the definitions above is what
|
||||||
|
# stops a second environment on one host from sharing a lock or a tag file.
|
||||||
|
TG_LOCK_FILE="$TG_APP_DIR/infra/deploy/.deploy.lock"
|
||||||
|
if [ "$TG_DEPLOY_MODE" = stack ]; then
|
||||||
|
TG_TAGS_FILE="$TG_APP_DIR/infra/deploy/.stack-image-tags.env"
|
||||||
|
else
|
||||||
|
TG_TAGS_FILE="$TG_APP_DIR/infra/deploy/.image-tags.env"
|
||||||
|
fi
|
||||||
|
|
||||||
|
export TG_ENV TG_HOST TG_APP_DIR TG_BRANCH TG_DEPLOY_MODE
|
||||||
|
export TG_STACK_NAME TG_STACK_FILE TG_LB_CONFIG TG_COMPOSE_PROJECT TG_COMPOSE_FILE
|
||||||
|
export TG_ENV_FILE TG_STACK_ENV_FILE TG_LB_NAME
|
||||||
|
export TG_LB_HTTP_PORT TG_LB_FE_PORT TG_DB_NAME TG_DB_USER
|
||||||
|
export TG_TAGS_FILE TG_LOCK_FILE TG_BIND_ADDR TG_SKIP_COMMON
|
||||||
|
export TG_SVC_PREFIX TG_DATA_NETWORK TG_DB_SERVICE TG_POST_DEPLOY
|
||||||
|
export TG_SECRETS_BACKEND TG_BAO_APPROLE
|
||||||
|
export TG_SSH_HOST TG_SSH_TARGET
|
||||||
|
}
|
||||||
|
|
||||||
|
# thermograph_env_name
|
||||||
|
#
|
||||||
|
# Which environment is this run for? Explicit input first (the deploy workflows
|
||||||
|
# pass THERMOGRAPH_ENV, and on vps2 it is the ONLY thing distinguishing a beta
|
||||||
|
# deploy from a prod one), then the host marker for a by-hand run on a
|
||||||
|
# single-environment box. Empty output means "cannot tell" and the caller must
|
||||||
|
# fail rather than guess — guessing wrong on vps2 means deploying beta's image
|
||||||
|
# tag onto prod.
|
||||||
|
thermograph_env_name() {
|
||||||
|
local marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
|
||||||
|
if [ -n "${THERMOGRAPH_ENV:-}" ]; then
|
||||||
|
printf '%s\n' "$THERMOGRAPH_ENV"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
cat "$marker" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
|
@ -1,22 +1,24 @@
|
||||||
# Forgejo on the Swarm cluster
|
# Forgejo on the Swarm cluster
|
||||||
|
|
||||||
Runs as `deploy/forgejo/docker-stack.yml` — the only Swarm-scheduled workload
|
Runs as `deploy/forgejo/docker-stack.yml` — the only Swarm-scheduled workload
|
||||||
this cluster carries (the Thermograph app itself stays on the
|
this cluster carries (prod and beta's app stacks are separate `docker stack
|
||||||
Terraform-managed `docker compose` deploys; see `terraform/README.md`).
|
deploy`s that happen to run on the manager node, vps2 — see
|
||||||
Pinned to the **beta** node (old VPS) via the `role=forge` label from
|
`deploy/stack/README` context in `DEPLOY.md`). Pinned to the **vps1** node
|
||||||
|
(`75.119.132.91`) via the `role=forge` label from
|
||||||
`deploy/swarm/label-forge-node.sh`.
|
`deploy/swarm/label-forge-node.sh`.
|
||||||
|
|
||||||
The Actions **runner** is deliberately *not* part of this stack — it runs on
|
The Actions **runner** is deliberately *not* part of this stack — see
|
||||||
the **desktop** as a plain systemd service (`register-lan-runner.sh` below),
|
`DEPLOY-DEV.md` and the note in `register-lan-runner.sh`'s own header for
|
||||||
per `thermograph-docs/runbooks/implementation-handoff.md` Track B step 5. That's the
|
where it runs today. That's the canonical placement per
|
||||||
canonical placement; an earlier revision of this stack ran the runner as a
|
`thermograph-docs/runbooks/implementation-handoff.md` Track B step 5; an
|
||||||
Swarm-scheduled Docker-in-Docker sidecar pinned to beta, which is gone now.
|
earlier revision of this stack ran the runner as a Swarm-scheduled
|
||||||
|
Docker-in-Docker sidecar pinned to the Forgejo node, which is gone now.
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
1. All three nodes have joined the swarm (`deploy/swarm/`) and beta is
|
1. All three nodes have joined the swarm (`deploy/swarm/`) and vps1 is
|
||||||
labeled `role=forge`.
|
labeled `role=forge`.
|
||||||
2. `docker node ls` (from the manager) shows all three `Ready`.
|
2. `docker node ls` (from the manager, vps2) shows all three `Ready`.
|
||||||
|
|
||||||
## One-time setup: Swarm secret
|
## One-time setup: Swarm secret
|
||||||
|
|
||||||
|
|
@ -43,27 +45,27 @@ its first health check just fails harmlessly until it is.
|
||||||
|
|
||||||
This stack has **no auto-deploy trigger** — nothing in `.forgejo/workflows/`
|
This stack has **no auto-deploy trigger** — nothing in `.forgejo/workflows/`
|
||||||
redeploys it on push. A change to `docker-stack.yml` only takes effect once
|
redeploys it on push. A change to `docker-stack.yml` only takes effect once
|
||||||
someone re-runs `docker stack deploy` by hand on the manager (prod).
|
someone re-runs `docker stack deploy` by hand on the manager (vps2).
|
||||||
|
|
||||||
`db`/`forgejo` both carry `resources.limits` (defaults: db 1 CPU/1g, forgejo 2
|
`db`/`forgejo` both carry `resources.limits` (defaults: db 1 CPU/1g, forgejo 2
|
||||||
CPU/2g — several times observed steady-state usage), overridable with
|
CPU/2g — several times observed steady-state usage), overridable with
|
||||||
`FORGEJO_DB_CPUS`/`FORGEJO_DB_MEMORY`/`FORGEJO_CPUS`/`FORGEJO_MEMORY` env vars
|
`FORGEJO_DB_CPUS`/`FORGEJO_DB_MEMORY`/`FORGEJO_CPUS`/`FORGEJO_MEMORY` env vars
|
||||||
before `docker stack deploy`, same convention as the app stack.
|
before `docker stack deploy`, same convention as the app stack.
|
||||||
|
|
||||||
## DNS + TLS: reusing beta's existing Caddy, not a second reverse proxy
|
## DNS + TLS: reusing vps1's existing Caddy, not a second reverse proxy
|
||||||
|
|
||||||
Forgejo is pinned to beta (`role=forge`) — but beta is also **today's live
|
Forgejo is pinned to vps1 (`role=forge`) — the same box that also runs
|
||||||
thermograph.org host**, and its Caddy already owns ports 80/443
|
Grafana/Loki/Alloy and the `emigriffith.dev` portfolio site, each fronted by
|
||||||
(`/etc/caddy/Caddyfile` on that box). A second ingress (Traefik) trying to
|
that host's own Caddy. A second ingress (Traefik) trying to bind the same
|
||||||
bind the same ports would collide with it. So there's no Traefik in this
|
ports would collide with it. So there's no Traefik in this stack: `forgejo`'s
|
||||||
stack: `forgejo`'s web port publishes to `127.0.0.1:3080` only (host-local),
|
web port publishes to `127.0.0.1:3080` only (host-local), and vps1's
|
||||||
and beta's *existing* Caddy gets one more site block reverse-proxying to it —
|
*existing* Caddy gets one more site block reverse-proxying to it — same
|
||||||
same pattern as its `thermograph.org` block, same automatic-HTTPS.
|
pattern as its other site blocks, same automatic-HTTPS.
|
||||||
|
|
||||||
1. Point the Forgejo domain (default `git.thermograph.org`; override with
|
1. Point the Forgejo domain (default `git.thermograph.org`; override with
|
||||||
`FORGEJO_DOMAIN=...` before `docker stack deploy`) at **beta's** public IP
|
`FORGEJO_DOMAIN=...` before `docker stack deploy`) at **vps1's** public IP
|
||||||
— that's where the task actually runs, not prod's or the desktop's.
|
— that's where the task actually runs, not vps2's or the desktop's.
|
||||||
2. Append `deploy/forgejo/caddy-git.conf` to beta's `/etc/caddy/Caddyfile`,
|
2. Append `deploy/forgejo/caddy-git.conf` to vps1's `/etc/caddy/Caddyfile`,
|
||||||
adjusting the domain if you didn't use the default, then `systemctl reload
|
adjusting the domain if you didn't use the default, then `systemctl reload
|
||||||
caddy`.
|
caddy`.
|
||||||
3. That file also resolves the registry-exposure hazard (#15 in
|
3. That file also resolves the registry-exposure hazard (#15 in
|
||||||
|
|
@ -75,28 +77,28 @@ same pattern as its `thermograph.org` block, same automatic-HTTPS.
|
||||||
|
|
||||||
## Registry access from mesh clients
|
## Registry access from mesh clients
|
||||||
|
|
||||||
Any node that needs `docker login`/push/pull against the registry (the
|
Any node that needs `docker login`/push/pull against the registry (the CI
|
||||||
desktop's CI runner building/pushing images, later any Swarm node pulling
|
runner building/pushing images, any Swarm node pulling them, prod or beta on
|
||||||
them) must reach `git.thermograph.org` **over the WireGuard tunnel**, not
|
vps2 pulling app images) must reach `git.thermograph.org` **over the
|
||||||
beta's public IP — otherwise Caddy's `/v2/*` block above refuses the
|
WireGuard tunnel**, not vps1's public IP — otherwise Caddy's `/v2/*` block
|
||||||
connection. Public DNS resolves the domain to beta's public IP, so add a
|
above refuses the connection. Public DNS resolves the domain to vps1's public
|
||||||
`/etc/hosts` override on each such node pinning it to beta's WireGuard
|
IP, so add a `/etc/hosts` override on each such node pinning it to vps1's
|
||||||
address instead:
|
WireGuard address instead:
|
||||||
|
|
||||||
```
|
```
|
||||||
echo "10.10.0.2 git.thermograph.org" | sudo tee -a /etc/hosts
|
echo "10.10.0.2 git.thermograph.org" | sudo tee -a /etc/hosts
|
||||||
```
|
```
|
||||||
|
|
||||||
(`10.10.0.2` is beta's WG address per `deploy/swarm/README.md`'s peer
|
(`10.10.0.2` is vps1's WG address per `deploy/swarm/README.md`'s peer
|
||||||
numbering — adjust if you assigned it differently.) The git/web UI keeps
|
numbering — adjust if you assigned it differently.) The git/web UI keeps
|
||||||
working normally for everyone else since only `/v2/*` is restricted.
|
working normally for everyone else since only `/v2/*` is restricted.
|
||||||
|
|
||||||
## Register the Actions runner (on the desktop, not through Swarm)
|
## Register the Actions runner
|
||||||
|
|
||||||
Once Forgejo answers at its domain:
|
Once Forgejo answers at its domain:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# On the desktop:
|
# On the runner host (see DEPLOY-DEV.md for where that is today):
|
||||||
# Forgejo web UI -> Site Administration -> Actions -> Runners -> Create new Runner
|
# Forgejo web UI -> Site Administration -> Actions -> Runners -> Create new Runner
|
||||||
# (or, for a repo-scoped runner: <repo> -> Settings -> Actions -> Runners)
|
# (or, for a repo-scoped runner: <repo> -> Settings -> Actions -> Runners)
|
||||||
# copy the registration token, then:
|
# copy the registration token, then:
|
||||||
|
|
@ -104,8 +106,8 @@ bash deploy/forgejo/register-lan-runner.sh https://<forgejo-domain> <token>
|
||||||
```
|
```
|
||||||
|
|
||||||
See that script's header for exactly what it replaces (the pre-Forgejo GitHub
|
See that script's header for exactly what it replaces (the pre-Forgejo GitHub
|
||||||
self-hosted runner on this same machine) and why it registers with two
|
self-hosted runner) and why it registers with two labels where there used to
|
||||||
labels where there used to be two separate runners.
|
be two separate runners.
|
||||||
|
|
||||||
`config.yaml`'s `runner.capacity` is raised from the tool's default of 1 to 8
|
`config.yaml`'s `runner.capacity` is raised from the tool's default of 1 to 8
|
||||||
(override with `CAPACITY=`) — a single PR push fires `pr-build`,
|
(override with `CAPACITY=`) — a single PR push fires `pr-build`,
|
||||||
|
|
@ -113,18 +115,16 @@ labels where there used to be two separate runners.
|
||||||
no `needs:` between them), so capacity 1 serializes work that could run in
|
no `needs:` between them), so capacity 1 serializes work that could run in
|
||||||
parallel, and even capacity 3 (an earlier, undocumented hand-tune) leaves
|
parallel, and even capacity 3 (an earlier, undocumented hand-tune) leaves
|
||||||
`build-backend`/`build-frontend`/`validate-observability` queued behind
|
`build-backend`/`build-frontend`/`validate-observability` queued behind
|
||||||
those three before they get a slot. The desktop has 16 cores / 34GB free
|
those three before they get a slot.
|
||||||
today; 8 concurrent jobs is comfortable headroom without starving LAN dev's
|
|
||||||
own compose stack.
|
|
||||||
|
|
||||||
**Adding more runner capacity should mean raising this number, or adding a
|
**Adding more runner capacity should mean raising this number, or adding a
|
||||||
second runner on the desktop itself — not putting a runner on prod or
|
second runner alongside it — not putting a runner on prod or beta (vps2).**
|
||||||
beta.** `container.docker_host: automount` gives job containers the *host's*
|
`container.docker_host: automount` gives job containers the *host's* Docker
|
||||||
Docker socket; on prod or beta that would mean any CI job has root-equivalent
|
socket; on vps2 that would mean any CI job has root-equivalent access to both
|
||||||
access to whatever else is running there (the live app stack, or Forgejo
|
the prod and beta stacks running there. An earlier revision of this stack
|
||||||
itself). An earlier revision of this stack actually did run the runner as a
|
actually ran the runner as a Swarm-hosted container on the box Forgejo was
|
||||||
Swarm-hosted container on beta and was deliberately reverted to the desktop
|
pinned to and was deliberately reverted for this same class of reason — see
|
||||||
for this reason — see the note at the top of `docker-stack.yml`.
|
the note at the top of `docker-stack.yml`.
|
||||||
|
|
||||||
## Custom CI job image (`ci-runner/`)
|
## Custom CI job image (`ci-runner/`)
|
||||||
|
|
||||||
|
|
@ -155,8 +155,8 @@ docker push git.thermograph.org/emi/thermograph/ci-runner:vN
|
||||||
|
|
||||||
`register-lan-runner.sh`'s `LABELS` default points at the current tag, so
|
`register-lan-runner.sh`'s `LABELS` default points at the current tag, so
|
||||||
fresh registrations pick it up automatically. The live runner is cut over by
|
fresh registrations pick it up automatically. The live runner is cut over by
|
||||||
editing the `labels` array in `~/forgejo-runner/.runner` on the desktop (same
|
editing the `labels` array in `~/forgejo-runner/.runner` on the runner host
|
||||||
runner id/token, no re-registration needed) and restarting the service —
|
(same runner id/token, no re-registration needed) and restarting the service —
|
||||||
**verify a real job runs green under the new image before relying on it**,
|
**verify a real job runs green under the new image before relying on it**,
|
||||||
same way v1's break was caught. Only after that verification should the
|
same way v1's break was caught. Only after that verification should the
|
||||||
now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed
|
now-redundant `apt-get install docker.io` / `python3-yaml` steps be removed
|
||||||
|
|
@ -165,7 +165,7 @@ still running on the stock `node:20-bookworm` image.
|
||||||
|
|
||||||
## Why Postgres here and not the Thermograph app's TimescaleDB
|
## Why Postgres here and not the Thermograph app's TimescaleDB
|
||||||
|
|
||||||
Separate instance, separate network (`forgejo_net`, not the app's compose
|
Separate instance, separate network (`forgejo_net`, not the app's overlay
|
||||||
network), separate volume. Forgejo is a distinct product with its own schema
|
network), separate volume. Forgejo is a distinct product with its own schema
|
||||||
and its own backup/restore lifecycle — sharing a database with the app would
|
and its own backup/restore lifecycle — sharing a database with the app would
|
||||||
couple two things that should be able to fail, migrate, and restore
|
couple two things that should be able to fail, migrate, and restore
|
||||||
|
|
@ -177,7 +177,7 @@ independently.
|
||||||
docker service ls # forgejo_db, forgejo_forgejo both Running, 1/1
|
docker service ls # forgejo_db, forgejo_forgejo both Running, 1/1
|
||||||
curl -I https://git.thermograph.org/ # 200, valid cert (Caddy's, not a new one)
|
curl -I https://git.thermograph.org/ # 200, valid cert (Caddy's, not a new one)
|
||||||
curl -I https://git.thermograph.org/v2/ # 403 from anywhere off the WireGuard mesh
|
curl -I https://git.thermograph.org/v2/ # 403 from anywhere off the WireGuard mesh
|
||||||
# On the desktop, after registering the runner:
|
# On the runner host, after registering the runner:
|
||||||
systemctl --user status forgejo-runner # active, both labels registered
|
systemctl --user status forgejo-runner # active, both labels registered
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -3,10 +3,11 @@
|
||||||
# ends up as a confusing orphaned comment in production config with no
|
# ends up as a confusing orphaned comment in production config with no
|
||||||
# surrounding context (docker-stack.yml isn't visible from there).
|
# surrounding context (docker-stack.yml isn't visible from there).
|
||||||
#
|
#
|
||||||
# Target: /etc/caddy/Caddyfile on beta (the box `forgejo` is pinned to — see
|
# Target: /etc/caddy/Caddyfile on vps1 (the box `forgejo` is pinned to — see
|
||||||
# docker-stack.yml). Reuses beta's existing Caddy instead of a second reverse
|
# docker-stack.yml; this is the box also called "vps1", 75.119.132.91).
|
||||||
# proxy/ACME flow: Forgejo publishes its web UI to 127.0.0.1:3080 only
|
# Reuses vps1's existing Caddy instead of a second reverse proxy/ACME flow:
|
||||||
# (host-local), and this block is the only thing that ever talks to it.
|
# Forgejo publishes its web UI to 127.0.0.1:3080 only (host-local), and this
|
||||||
|
# block is the only thing that ever talks to it.
|
||||||
#
|
#
|
||||||
# Registry exposure (hazard #15, see docker-stack.yml's header comment):
|
# Registry exposure (hazard #15, see docker-stack.yml's header comment):
|
||||||
# git.thermograph.org/v2/* is the built-in OCI registry API. It's blocked
|
# git.thermograph.org/v2/* is the built-in OCI registry API. It's blocked
|
||||||
|
|
@ -15,7 +16,7 @@
|
||||||
# else (web UI, git-over-HTTP, PR pages) stays public like the rest of the
|
# else (web UI, git-over-HTTP, PR pages) stays public like the rest of the
|
||||||
# site.
|
# site.
|
||||||
#
|
#
|
||||||
# Update the domain below to match whatever you actually pointed at beta's
|
# Update the domain below to match whatever you actually pointed at vps1's
|
||||||
# IP, if not git.thermograph.org.
|
# IP, if not git.thermograph.org.
|
||||||
|
|
||||||
# --- copy from here down into /etc/caddy/Caddyfile ---
|
# --- copy from here down into /etc/caddy/Caddyfile ---
|
||||||
|
|
|
||||||
|
|
@ -9,18 +9,18 @@
|
||||||
# (deploy/forgejo/register-lan-runner.sh) — the same place the pre-Forgejo
|
# (deploy/forgejo/register-lan-runner.sh) — the same place the pre-Forgejo
|
||||||
# GitHub self-hosted runner already lived, not a Swarm-scheduled container.
|
# GitHub self-hosted runner already lived, not a Swarm-scheduled container.
|
||||||
#
|
#
|
||||||
# No Traefik here. Forgejo is pinned to beta (node.labels.role == forge)
|
# No Traefik here. Forgejo is pinned to vps1 (node.labels.role == forge)
|
||||||
# because that's what was chosen, but beta is ALSO today's live thermograph.org
|
# because that's what was chosen, and vps1 is ALSO the emigriffith.dev portfolio
|
||||||
# host — Caddy already owns its ports 80/443 (see /etc/caddy/Caddyfile on that
|
# host — Caddy already owns its ports 80/443 (see /etc/caddy/Caddyfile on that
|
||||||
# box). A second reverse proxy binding those same ports would either fail to
|
# box). A second reverse proxy binding those same ports would either fail to
|
||||||
# start or fight Caddy. Instead: forgejo's web port publishes to
|
# start or fight Caddy. Instead: forgejo's web port publishes to
|
||||||
# 127.0.0.1:3080 only (host-local, mode: host), and beta's existing Caddy gets
|
# 127.0.0.1:3080 only (host-local, mode: host), and vps1's existing Caddy gets
|
||||||
# a new site block reverse-proxying git.thermograph.org -> 127.0.0.1:3080,
|
# a new site block reverse-proxying git.thermograph.org -> 127.0.0.1:3080,
|
||||||
# same pattern as its thermograph.org block. TLS is Caddy's existing
|
# same pattern as its other blocks. TLS is Caddy's existing automatic-HTTPS
|
||||||
# automatic-HTTPS (HTTP-01), not a second ACME flow.
|
# (HTTP-01), not a second ACME flow.
|
||||||
#
|
#
|
||||||
# Deploy from the manager node (prod), after all three nodes (prod, beta,
|
# Deploy from the manager node (vps2, which runs prod), after all three nodes
|
||||||
# desktop) have joined the swarm and beta is labeled role=forge:
|
# (vps1, vps2, desktop) have joined the swarm and vps1 is labeled role=forge:
|
||||||
#
|
#
|
||||||
# docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo
|
# docker stack deploy -c deploy/forgejo/docker-stack.yml forgejo
|
||||||
#
|
#
|
||||||
|
|
@ -32,7 +32,7 @@
|
||||||
# firewall the /v2/ registry API paths so only WireGuard-mesh clients can
|
# firewall the /v2/ registry API paths so only WireGuard-mesh clients can
|
||||||
# reach them. That's implemented in the Caddy site block (see
|
# reach them. That's implemented in the Caddy site block (see
|
||||||
# deploy/forgejo/README.md), not here — nothing in this stack file is
|
# deploy/forgejo/README.md), not here — nothing in this stack file is
|
||||||
# registry-specific, the restriction lives entirely in Caddy's config on beta.
|
# registry-specific, the restriction lives entirely in Caddy's config on vps1.
|
||||||
|
|
||||||
services:
|
services:
|
||||||
db:
|
db:
|
||||||
|
|
@ -53,7 +53,19 @@ services:
|
||||||
cpus: "${FORGEJO_DB_CPUS:-1}"
|
cpus: "${FORGEJO_DB_CPUS:-1}"
|
||||||
memory: ${FORGEJO_DB_MEMORY:-1g}
|
memory: ${FORGEJO_DB_MEMORY:-1g}
|
||||||
restart_policy:
|
restart_policy:
|
||||||
condition: on-failure
|
# `any`, NOT `on-failure`. This took Forgejo down for 27 hours on 2026-07-29:
|
||||||
|
# Postgres hit an invalid data-directory lock file ("could not open file
|
||||||
|
# postmaster.pid ... performing immediate shutdown") and exited **0**. A clean
|
||||||
|
# exit is not a failure, so Swarm considered the task Complete, dropped the
|
||||||
|
# service to 0/1 replicas, and never rescheduled it. Forgejo itself stayed Up
|
||||||
|
# and served its homepage while every repo page, the API and all CI returned
|
||||||
|
# 500 with `dial tcp: lookup db ... no such host`.
|
||||||
|
#
|
||||||
|
# `on-failure` is the wrong policy for any always-on stateful service: it
|
||||||
|
# cannot distinguish "finished successfully" from "shut itself down and should
|
||||||
|
# be restarted", and Postgres does the latter with status 0 on several paths.
|
||||||
|
condition: any
|
||||||
|
delay: 5s
|
||||||
|
|
||||||
forgejo:
|
forgejo:
|
||||||
image: codeberg.org/forgejo/forgejo:9-rootless
|
image: codeberg.org/forgejo/forgejo:9-rootless
|
||||||
|
|
@ -96,9 +108,10 @@ services:
|
||||||
FORGEJO__service__ENABLE_PASSWORD_SIGNIN_FORM: "false"
|
FORGEJO__service__ENABLE_PASSWORD_SIGNIN_FORM: "false"
|
||||||
FORGEJO__service__REGISTER_MANUAL_CONFIRM: "true"
|
FORGEJO__service__REGISTER_MANUAL_CONFIRM: "true"
|
||||||
FORGEJO__service__DEFAULT_USER_IS_RESTRICTED: "true"
|
FORGEJO__service__DEFAULT_USER_IS_RESTRICTED: "true"
|
||||||
# Outbound mail via prod's Postfix null client over the WireGuard mesh
|
# Outbound mail via vps2's Postfix null client (prod's box) over the
|
||||||
# (10.10.0.1:25 — mynetworks permits beta); self-signed cert on :25, so
|
# WireGuard mesh (10.10.0.1:25 — mynetworks permits vps1, 10.10.0.2,
|
||||||
# trust it. Send-only; used for admin/approval and notification mail.
|
# where this Forgejo runs); self-signed cert on :25, so trust it.
|
||||||
|
# Send-only; used for admin/approval and notification mail.
|
||||||
FORGEJO__mailer__ENABLED: "true"
|
FORGEJO__mailer__ENABLED: "true"
|
||||||
FORGEJO__mailer__PROTOCOL: "smtp"
|
FORGEJO__mailer__PROTOCOL: "smtp"
|
||||||
FORGEJO__mailer__SMTP_ADDR: "10.10.0.1"
|
FORGEJO__mailer__SMTP_ADDR: "10.10.0.1"
|
||||||
|
|
@ -117,17 +130,17 @@ services:
|
||||||
networks: [forgejo_net]
|
networks: [forgejo_net]
|
||||||
ports:
|
ports:
|
||||||
# SSH for git@ clones — published on whichever node the task lands on
|
# SSH for git@ clones — published on whichever node the task lands on
|
||||||
# (pinned to beta by the placement constraint below, so effectively
|
# (pinned to vps1 by the placement constraint below, so effectively
|
||||||
# always beta's public IP, port 2222).
|
# always vps1's public IP, port 2222).
|
||||||
- target: 2222
|
- target: 2222
|
||||||
published: 2222
|
published: 2222
|
||||||
protocol: tcp
|
protocol: tcp
|
||||||
mode: host
|
mode: host
|
||||||
# Web UI/API. Swarm's port schema has no host_ip scoping, so this binds
|
# Web UI/API. Swarm's port schema has no host_ip scoping, so this binds
|
||||||
# 0.0.0.0:3080 on beta — NOT actually localhost-only by itself. A
|
# 0.0.0.0:3080 on vps1 — NOT actually localhost-only by itself. A
|
||||||
# DOCKER-USER iptables rule (see deploy/forgejo/README.md) is what
|
# DOCKER-USER iptables rule (see deploy/forgejo/README.md) is what
|
||||||
# actually restricts it, since Docker's own iptables rules bypass ufw
|
# actually restricts it, since Docker's own iptables rules bypass ufw
|
||||||
# for published ports. beta's Caddy reverse-proxies to 127.0.0.1:3080.
|
# for published ports. vps1's Caddy reverse-proxies to 127.0.0.1:3080.
|
||||||
- target: 3000
|
- target: 3000
|
||||||
published: 3080
|
published: 3080
|
||||||
protocol: tcp
|
protocol: tcp
|
||||||
|
|
|
||||||
|
|
@ -4,17 +4,26 @@
|
||||||
# runner (see DEPLOY-DEV.md) — this replaces that runner, it doesn't add a
|
# runner (see DEPLOY-DEV.md) — this replaces that runner, it doesn't add a
|
||||||
# second one. Sudo-free, systemd --user, same pattern as the app service.
|
# second one. Sudo-free, systemd --user, same pattern as the app service.
|
||||||
#
|
#
|
||||||
# This is THE runner (docs/runbooks/implementation-handoff.md Track B step 5
|
# It registers with BOTH labels the workflows historically needed: general
|
||||||
# — canonical placement is the desktop, not a Swarm-hosted container), so it
|
# CI/build/deploy jobs (`docker`, containerized via this machine's own
|
||||||
# registers with BOTH labels the workflows need, where one runner used to be
|
# already-installed Docker — no Docker-in-Docker sidecar needed, since a real
|
||||||
# two: general CI/build/deploy.yml jobs (`docker`, containerized via this
|
# host with a real Docker install needs no such indirection) and the LAN-deploy
|
||||||
# machine's own already-installed Docker — no Docker-in-Docker sidecar needed,
|
# job (`thermograph-lan`, bare/host-native — it wrote to ~/thermograph-dev and
|
||||||
# unlike the Swarm-hosted design this replaces, since a real host with a real
|
# restarted a systemd --user service, which only worked running directly on the
|
||||||
# Docker install needs no such indirection) and the LAN-deploy job
|
|
||||||
# (`thermograph-lan`, bare/host-native — it writes to ~/thermograph-dev and
|
|
||||||
# restarts a systemd --user service, which only works running directly on the
|
|
||||||
# host, not inside a container).
|
# host, not inside a container).
|
||||||
#
|
#
|
||||||
|
# `thermograph-lan` IS NOW OBSOLETE. Dev moved off this machine to vps1 and is
|
||||||
|
# deployed over SSH by the same `Deploy` workflow that ships beta and prod;
|
||||||
|
# there is no host-native LAN deploy job left for that label to serve. Nothing
|
||||||
|
# breaks by keeping it registered — no workflow requests it — but do not build
|
||||||
|
# anything new on it.
|
||||||
|
#
|
||||||
|
# TODO(cutover): whether this machine keeps serving the `docker` label at all is
|
||||||
|
# an open call. It is no longer the only runner (the Swarm-hosted one is
|
||||||
|
# always-on), and the box's new job is AI model hosting plus flex Swarm-worker
|
||||||
|
# capacity. Keeping it is fine — it is real CI capacity — but the estate no
|
||||||
|
# longer depends on it, so decide deliberately rather than by inertia.
|
||||||
|
#
|
||||||
# bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
|
# bash deploy/forgejo/register-lan-runner.sh <forgejo_url> <registration_token>
|
||||||
#
|
#
|
||||||
# Get <registration_token> from the Forgejo web UI:
|
# Get <registration_token> from the Forgejo web UI:
|
||||||
|
|
|
||||||
|
|
@ -184,6 +184,10 @@ credit; keep it in place.
|
||||||
|
|
||||||
## 9. Not on dev/beta
|
## 9. Not on dev/beta
|
||||||
|
|
||||||
This overlay runs only on the self-hosting host (prod). Dev and beta leave
|
This overlay runs only for prod specifically — not for every environment on
|
||||||
`THERMOGRAPH_ARCHIVE_URL` unset and use the public Open-Meteo archive API — do
|
prod's host. Beta now shares vps2 with prod, but that doesn't extend the
|
||||||
not bring the overlay up there.
|
self-hosted archive to it: beta is its own Swarm stack
|
||||||
|
(`thermograph-beta-stack.yml`), which never sets `THERMOGRAPH_ARCHIVE_URL`, so
|
||||||
|
it reaches the public Open-Meteo archive API like dev does. Dev leaves it
|
||||||
|
unset for the same reason on vps1. Do not bring this overlay up for beta or
|
||||||
|
dev, and don't assume co-residency with prod on vps2 changes that.
|
||||||
|
|
|
||||||
|
|
@ -1,30 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
# One-time bootstrap for the Thermograph LAN dev server on THIS machine.
|
|
||||||
#
|
|
||||||
# Sudo-free: the app runs as a Docker Compose stack (see deploy-dev.sh), and
|
|
||||||
# `linger` keeps the runner/services running across logout/reboot. Re-runnable
|
|
||||||
# (idempotent).
|
|
||||||
#
|
|
||||||
# bash deploy/provision-dev-lan.sh
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
APP_DIR="${APP_DIR:-$HOME/thermograph-dev}"
|
|
||||||
REPO_URL="${REPO_URL:-http://10.10.0.2:3080/emi/thermograph-infra.git}"
|
|
||||||
BRANCH="${BRANCH:-dev}"
|
|
||||||
here="$(cd "$(dirname "$0")" && pwd)"
|
|
||||||
|
|
||||||
echo "==> Enabling linger so the service survives logout/reboot"
|
|
||||||
loginctl enable-linger "$USER" \
|
|
||||||
|| echo " (couldn't enable linger; the service still runs while you're logged in)"
|
|
||||||
|
|
||||||
echo "==> Cloning/refreshing $APP_DIR and starting the service"
|
|
||||||
APP_DIR="$APP_DIR" REPO_URL="$REPO_URL" BRANCH="$BRANCH" bash "$here/deploy-dev.sh"
|
|
||||||
|
|
||||||
cat <<EOF
|
|
||||||
|
|
||||||
Done. The dev server runs as the 'thermograph-dev' systemd --user service.
|
|
||||||
status: systemctl --user status thermograph-dev
|
|
||||||
logs: journalctl --user -u thermograph-dev -f
|
|
||||||
restart: systemctl --user restart thermograph-dev
|
|
||||||
stop: systemctl --user stop thermograph-dev
|
|
||||||
EOF
|
|
||||||
80
infra/deploy/provision-dev.sh
Executable file
80
infra/deploy/provision-dev.sh
Executable file
|
|
@ -0,0 +1,80 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# One-time bootstrap of the DEV environment on vps1.
|
||||||
|
#
|
||||||
|
# sudo bash infra/deploy/provision-dev.sh
|
||||||
|
#
|
||||||
|
# Replaces provision-dev-lan.sh, which bootstrapped dev on the operator's
|
||||||
|
# desktop as a sudo-free systemd --user stack with `linger`. Dev is a normal
|
||||||
|
# fleet environment now: a checkout at /opt/thermograph-dev, secrets rendered
|
||||||
|
# from the SOPS vault at deploy time, deployed over SSH by CI like beta and
|
||||||
|
# prod. The desktop hosts no Thermograph environment at all.
|
||||||
|
#
|
||||||
|
# What dev keeps that beta and prod do not:
|
||||||
|
# - It renders dev.yaml ALONE, never layering common.yaml (the fleet's shared
|
||||||
|
# production credentials). vps1 also runs Forgejo and its CI, and dev runs
|
||||||
|
# whatever branch is in flight — see the long note in render-secrets.sh.
|
||||||
|
# - It binds LOOPBACK, never 0.0.0.0. Caddy fronts it at dev.thermograph.org
|
||||||
|
# with TLS, basic auth and X-Robots-Tag: noindex (deploy/Caddyfile.vps1).
|
||||||
|
# Binding loopback is what makes that guard total rather than decorative:
|
||||||
|
# the site block is the only route in.
|
||||||
|
#
|
||||||
|
# Idempotent; re-run it after changing the branch or repointing the remote.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SELF_DIR=$(cd "$(dirname "$0")" && pwd)
|
||||||
|
# shellcheck source=infra/deploy/env-topology.sh
|
||||||
|
. "$SELF_DIR/env-topology.sh"
|
||||||
|
thermograph_topology dev
|
||||||
|
|
||||||
|
REPO_URL="${REPO_URL:-http://10.10.0.2:3080/emi/thermograph.git}"
|
||||||
|
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
|
||||||
|
BRANCH="${BRANCH:-$TG_BRANCH}"
|
||||||
|
|
||||||
|
if [ "$(id -u)" -ne 0 ]; then
|
||||||
|
echo "!! run this as root (it writes /opt and /etc/thermograph)" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Checkout: $APP_DIR on $BRANCH"
|
||||||
|
if [ -d "$APP_DIR/.git" ]; then
|
||||||
|
git -C "$APP_DIR" remote set-url origin "$REPO_URL"
|
||||||
|
git -C "$APP_DIR" fetch --prune origin "$BRANCH"
|
||||||
|
git -C "$APP_DIR" checkout -B "$BRANCH" "origin/$BRANCH"
|
||||||
|
else
|
||||||
|
git clone --branch "$BRANCH" "$REPO_URL" "$APP_DIR"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Marking this checkout's environment"
|
||||||
|
mkdir -p /etc/thermograph
|
||||||
|
# The host marker still exists for by-hand runs. vps1 runs exactly one
|
||||||
|
# environment, so a marker is sufficient here — unlike vps2, where beta and prod
|
||||||
|
# share a box and THERMOGRAPH_ENV must be passed explicitly.
|
||||||
|
printf 'dev\n' > /etc/thermograph/secrets-env
|
||||||
|
# Dev is compose, not Swarm. env-topology.sh is what actually decides this; the
|
||||||
|
# marker is only the fallback for a checkout that predates it.
|
||||||
|
rm -f /etc/thermograph/deploy-mode
|
||||||
|
|
||||||
|
if [ ! -f /etc/thermograph/age.key ]; then
|
||||||
|
cat >&2 <<'EOF'
|
||||||
|
!! No age key at /etc/thermograph/age.key.
|
||||||
|
!! Dev cannot render its vault without it, and deploy-dev.sh will fall back to
|
||||||
|
!! the un-vaulted defaults. Copy the key over (0400 root:root), then re-run:
|
||||||
|
!! install -m 0400 -o root -g root age.key /etc/thermograph/age.key
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat <<EOF
|
||||||
|
|
||||||
|
Done. Dev is checked out at $APP_DIR on $BRANCH.
|
||||||
|
|
||||||
|
Deploy it:
|
||||||
|
SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> \\
|
||||||
|
$APP_DIR/infra/deploy/deploy-dev.sh
|
||||||
|
|
||||||
|
Reach it:
|
||||||
|
https://dev.thermograph.org (basic auth; see deploy/Caddyfile.vps1)
|
||||||
|
|
||||||
|
Confirm the app itself is NOT directly exposed — this must show
|
||||||
|
${TG_BIND_ADDR}:8137 and never 0.0.0.0:8137, so Caddy is the only way in:
|
||||||
|
ss -ltnp | grep 8137
|
||||||
|
EOF
|
||||||
|
|
@ -35,6 +35,12 @@
|
||||||
# sudo MAIL_DOMAIN=thermograph.org bash deploy/provision-mail.sh
|
# sudo MAIL_DOMAIN=thermograph.org bash deploy/provision-mail.sh
|
||||||
# sudo MAIL_DOMAIN=thermograph.org RELAYHOST='[smtp.provider.com]:587' \
|
# sudo MAIL_DOMAIN=thermograph.org RELAYHOST='[smtp.provider.com]:587' \
|
||||||
# RELAY_USER=apikey RELAY_PASSWORD=secret bash deploy/provision-mail.sh
|
# RELAY_USER=apikey RELAY_PASSWORD=secret bash deploy/provision-mail.sh
|
||||||
|
#
|
||||||
|
# MAIL_ENV picks which environment's deploy mode (env-topology.sh) sizes the
|
||||||
|
# Docker mail gateway below -- see the "WHICH gateway" comment. Defaults to
|
||||||
|
# prod; only matters if this box ever runs an environment in compose mode
|
||||||
|
# (it doesn't today -- prod and beta are both Swarm on this box; compose-mode
|
||||||
|
# dev lives on the other box entirely and doesn't run Postfix).
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
MAIL_DOMAIN="${MAIL_DOMAIN:-thermograph.org}"
|
MAIL_DOMAIN="${MAIL_DOMAIN:-thermograph.org}"
|
||||||
|
|
@ -43,6 +49,12 @@ RELAYHOST="${RELAYHOST:-}"
|
||||||
RELAY_USER="${RELAY_USER:-}"
|
RELAY_USER="${RELAY_USER:-}"
|
||||||
RELAY_PASSWORD="${RELAY_PASSWORD:-}"
|
RELAY_PASSWORD="${RELAY_PASSWORD:-}"
|
||||||
|
|
||||||
|
MAIL_ENV="${MAIL_ENV:-prod}"
|
||||||
|
SELF_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||||
|
# shellcheck source=infra/deploy/env-topology.sh
|
||||||
|
. "$SELF_DIR/env-topology.sh"
|
||||||
|
thermograph_topology "$MAIL_ENV"
|
||||||
|
|
||||||
if [[ $EUID -ne 0 ]]; then
|
if [[ $EUID -ne 0 ]]; then
|
||||||
echo "run as root (sudo)" >&2
|
echo "run as root (sudo)" >&2
|
||||||
exit 1
|
exit 1
|
||||||
|
|
@ -68,17 +80,21 @@ postconf -e "myorigin = ${MAIL_DOMAIN}"
|
||||||
# docker-compose.yml). Set DOCKER_MAIL_GATEWAY="" for a pure loopback-only null
|
# docker-compose.yml). Set DOCKER_MAIL_GATEWAY="" for a pure loopback-only null
|
||||||
# client (app running natively on the host, not in a container).
|
# client (app running natively on the host, not in a container).
|
||||||
#
|
#
|
||||||
# WHICH gateway depends on the host's deploy mode: plain compose (beta, LAN)
|
# WHICH gateway to listen on comes from MAIL_ENV's deploy mode (env-topology.sh,
|
||||||
# uses the pinned compose bridge (172.19.0.1/172.19.0.0/16, the defaults);
|
# TG_DEPLOY_MODE) -- not a hardcoded beta/prod split. Beta moved onto this SAME
|
||||||
# Swarm-stack mode (prod) uses the docker_gwbridge gateway instead --
|
# box as prod and is Swarm too now, so "compose" no longer means "beta"; it
|
||||||
# overlay tasks have no compose-bridge gateway -- so prod is provisioned with
|
# means dev, which lives on the other box entirely and never runs this script.
|
||||||
# DOCKER_MAIL_GATEWAY=172.18.0.1 DOCKER_MAIL_SUBNET=172.18.0.0/16 (plus its
|
# Compose mode uses the pinned compose bridge (172.19.0.1/172.19.0.0/16, the
|
||||||
# MESH_MAIL_* listener below). Do NOT list an address that doesn't exist on
|
# defaults); stack mode uses the docker_gwbridge gateway instead -- overlay
|
||||||
# the host: Postfix's master fails to bind and takes ALL listeners down --
|
# tasks have no compose-bridge gateway -- so a stack-mode MAIL_ENV (prod or
|
||||||
# exactly what happened when the compose bridge (172.19.0.1) vanished at the
|
# beta; same box, same gateway) gets DOCKER_MAIL_GATEWAY=172.18.0.1
|
||||||
# stack cutover while still listed in inet_interfaces. Also note: postfix on
|
# DOCKER_MAIL_SUBNET=172.18.0.0/16 (plus the MESH_MAIL_* listener below). Do
|
||||||
# this distro is an umbrella unit; restart `postfix@-`, not `postfix`, for
|
# NOT list an address that doesn't exist on the host: Postfix's master fails
|
||||||
# inet_interfaces changes to take effect.
|
# to bind and takes ALL listeners down -- exactly what happened when the
|
||||||
|
# compose bridge (172.19.0.1) vanished at the stack cutover while still listed
|
||||||
|
# in inet_interfaces. Also note: postfix on this distro is an umbrella unit;
|
||||||
|
# restart `postfix@-`, not `postfix`, for inet_interfaces changes to take
|
||||||
|
# effect.
|
||||||
#
|
#
|
||||||
# THE SAME TRAP FIRES AT BOOT, NOT JUST ON RENUMBERING (prod outage 2026-07-24).
|
# THE SAME TRAP FIRES AT BOOT, NOT JUST ON RENUMBERING (prod outage 2026-07-24).
|
||||||
# A Docker bridge address does not exist until dockerd creates it, and the stock
|
# A Docker bridge address does not exist until dockerd creates it, and the stock
|
||||||
|
|
@ -92,15 +108,23 @@ postconf -e "myorigin = ${MAIL_DOMAIN}"
|
||||||
# it orders postfix@ after docker.service and wg-quick@wg0.service and retries
|
# it orders postfix@ after docker.service and wg-quick@wg0.service and retries
|
||||||
# on failure. If you add an address here that some other daemon creates, add
|
# on failure. If you add an address here that some other daemon creates, add
|
||||||
# that daemon to the drop-in too.
|
# that daemon to the drop-in too.
|
||||||
DOCKER_MAIL_GATEWAY="${DOCKER_MAIL_GATEWAY-172.19.0.1}"
|
if [ "$TG_DEPLOY_MODE" = stack ]; then
|
||||||
DOCKER_MAIL_SUBNET="${DOCKER_MAIL_SUBNET-172.19.0.0/16}"
|
DOCKER_MAIL_GATEWAY="${DOCKER_MAIL_GATEWAY-172.18.0.1}"
|
||||||
# Optional WireGuard-mesh listener: other mesh nodes (e.g. beta's Forgejo, whose
|
DOCKER_MAIL_SUBNET="${DOCKER_MAIL_SUBNET-172.18.0.0/16}"
|
||||||
|
else
|
||||||
|
DOCKER_MAIL_GATEWAY="${DOCKER_MAIL_GATEWAY-172.19.0.1}"
|
||||||
|
DOCKER_MAIL_SUBNET="${DOCKER_MAIL_SUBNET-172.19.0.0/16}"
|
||||||
|
fi
|
||||||
|
# Optional WireGuard-mesh listener: other mesh nodes (e.g. vps1's Forgejo, whose
|
||||||
# mailer posts to 10.10.0.1:25 — see deploy/forgejo/docker-stack.yml) can relay
|
# mailer posts to 10.10.0.1:25 — see deploy/forgejo/docker-stack.yml) can relay
|
||||||
# through this box. Prod runs with MESH_MAIL_LISTEN=10.10.0.1 and
|
# through this box. This host (vps2 -- prod AND beta) runs with
|
||||||
# MESH_MAIL_PEERS=10.10.0.2/32; both default OFF so a plain run stays a strict
|
# MESH_MAIL_LISTEN=10.10.0.1 and MESH_MAIL_PEERS=10.10.0.2/32; both default OFF
|
||||||
# null client. Without these, re-running this script on prod would silently drop
|
# so a plain run stays a strict null client. 10.10.0.2 is vps1 (Forgejo,
|
||||||
# the mesh listener and break Forgejo's outbound mail — the live config was
|
# Grafana, dev) -- mesh IPs did not move in the vps1/vps2 split, only which
|
||||||
# originally hand-applied and this script is the source of truth for it now.
|
# environment runs where, so this is NOT "beta's" address. Without these,
|
||||||
|
# re-running this script on vps2 would silently drop the mesh listener and
|
||||||
|
# break Forgejo's outbound mail — the live config was originally hand-applied
|
||||||
|
# and this script is the source of truth for it now.
|
||||||
MESH_MAIL_LISTEN="${MESH_MAIL_LISTEN-}"
|
MESH_MAIL_LISTEN="${MESH_MAIL_LISTEN-}"
|
||||||
MESH_MAIL_PEERS="${MESH_MAIL_PEERS-}"
|
MESH_MAIL_PEERS="${MESH_MAIL_PEERS-}"
|
||||||
postconf -e "inet_protocols = ipv4"
|
postconf -e "inet_protocols = ipv4"
|
||||||
|
|
|
||||||
|
|
@ -1,16 +1,37 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Provision a host to render secrets from the SOPS vault at deploy time. Idempotent;
|
# Provision a NAMED ENVIRONMENT to render secrets from the SOPS vault at deploy
|
||||||
# run once per box (prod/beta) as a sudo-capable user.
|
# time. Idempotent; run once per environment (not once per box) as a
|
||||||
|
# sudo-capable user — vps2 carries two (prod AND beta), so it needs this run
|
||||||
|
# TWICE, each pointing SECRETS_ENV_FILE at that environment's own marker.
|
||||||
#
|
#
|
||||||
# SECRETS_ENV=prod AGE_KEY_SRC=/path/to/age.key bash deploy/provision-secrets.sh
|
# SECRETS_ENV=prod AGE_KEY_SRC=/path/to/age.key bash deploy/provision-secrets.sh
|
||||||
|
# SECRETS_ENV=beta AGE_KEY_SRC=/path/to/age.key \
|
||||||
|
# SECRETS_ENV_FILE=/etc/thermograph/beta-secrets-env bash deploy/provision-secrets.sh
|
||||||
#
|
#
|
||||||
# Installs `sops` + `age`, then installs the age PRIVATE key at /etc/thermograph/age.key
|
# Installs `sops` + `age` (idempotent, and shared across every environment on the
|
||||||
# (0400) and writes /etc/thermograph/secrets-env. After this, deploy.sh renders
|
# box — one age key decrypts every deploy/secrets/<env>.yaml, since they're all
|
||||||
# /etc/thermograph.env from deploy/secrets/*.yaml. See deploy/secrets/README.md.
|
# encrypted to the same recipient), then installs the age PRIVATE key at
|
||||||
|
# /etc/thermograph/age.key (0400) and writes SECRETS_ENV_FILE (default
|
||||||
|
# /etc/thermograph/secrets-env — right for a single-environment host, dev
|
||||||
|
# included). After this, deploy.sh renders that environment's own
|
||||||
|
# /etc/thermograph*.env from deploy/secrets/*.yaml. See deploy/secrets/README.md.
|
||||||
|
#
|
||||||
|
# SECRETS_ENV_FILE only matters for a by-hand run with THERMOGRAPH_ENV unset:
|
||||||
|
# env-topology.sh's thermograph_env_name() falls back to reading it to decide
|
||||||
|
# which environment a bare `deploy.sh` means, and a host running two
|
||||||
|
# environments (vps2) cannot answer that from one shared marker — a by-hand
|
||||||
|
# deploy of beta there must export THERMOGRAPH_SECRETS_ENV_FILE to match
|
||||||
|
# whatever path this script wrote. CI-driven deploys pass THERMOGRAPH_ENV
|
||||||
|
# explicitly and never consult this file at all.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SECRETS_ENV="${SECRETS_ENV:?set SECRETS_ENV=prod|beta}"
|
SECRETS_ENV="${SECRETS_ENV:?set SECRETS_ENV=prod|beta|dev}"
|
||||||
AGE_KEY_SRC="${AGE_KEY_SRC:?set AGE_KEY_SRC=/path/to/the/age/private/key}"
|
AGE_KEY_SRC="${AGE_KEY_SRC:?set AGE_KEY_SRC=/path/to/the/age/private/key}"
|
||||||
|
# Where to write the env marker. Default matches the original, single-environment
|
||||||
|
# behavior; a SECOND environment on the same box (vps2's beta, alongside prod's
|
||||||
|
# default marker) must pass a distinct path so provisioning one can never
|
||||||
|
# overwrite the other's marker.
|
||||||
|
SECRETS_ENV_FILE="${SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
|
||||||
SOPS_VERSION="${SOPS_VERSION:-v3.13.2}"
|
SOPS_VERSION="${SOPS_VERSION:-v3.13.2}"
|
||||||
AGE_VERSION="${AGE_VERSION:-v1.3.1}"
|
AGE_VERSION="${AGE_VERSION:-v1.3.1}"
|
||||||
BIN="${BIN:-/usr/local/bin}"
|
BIN="${BIN:-/usr/local/bin}"
|
||||||
|
|
@ -35,11 +56,16 @@ install_age() {
|
||||||
install_sops
|
install_sops
|
||||||
install_age
|
install_age
|
||||||
|
|
||||||
echo "==> Installing age key -> /etc/thermograph/age.key (0400) and marker (${SECRETS_ENV})"
|
echo "==> Installing age key -> /etc/thermograph/age.key (0400) and marker ${SECRETS_ENV_FILE} (${SECRETS_ENV})"
|
||||||
sudo mkdir -p /etc/thermograph
|
sudo mkdir -p /etc/thermograph "$(dirname "$SECRETS_ENV_FILE")"
|
||||||
sudo install -m 0400 "$AGE_KEY_SRC" /etc/thermograph/age.key
|
sudo install -m 0400 "$AGE_KEY_SRC" /etc/thermograph/age.key
|
||||||
printf '%s\n' "$SECRETS_ENV" | sudo tee /etc/thermograph/secrets-env >/dev/null
|
printf '%s\n' "$SECRETS_ENV" | sudo tee "$SECRETS_ENV_FILE" >/dev/null
|
||||||
sudo chmod 0644 /etc/thermograph/secrets-env
|
sudo chmod 0644 "$SECRETS_ENV_FILE"
|
||||||
|
|
||||||
echo "==> Done. Verify: sops --version && ls -l /etc/thermograph/"
|
echo "==> Done. Verify: sops --version && ls -l /etc/thermograph/"
|
||||||
echo " Next: seed deploy/secrets/*.yaml, dry-run render, then deploy (see README)."
|
echo " Next: seed deploy/secrets/*.yaml, dry-run render, then deploy (see README)."
|
||||||
|
if [ "$SECRETS_ENV_FILE" != /etc/thermograph/secrets-env ]; then
|
||||||
|
echo " NOTE: non-default marker -- a by-hand deploy of ${SECRETS_ENV} on this box"
|
||||||
|
echo " must export THERMOGRAPH_SECRETS_ENV_FILE=${SECRETS_ENV_FILE} (CI-driven"
|
||||||
|
echo " deploys pass THERMOGRAPH_ENV explicitly and never consult this file)."
|
||||||
|
fi
|
||||||
|
|
|
||||||
493
infra/deploy/render-secrets-openbao.sh
Normal file
493
infra/deploy/render-secrets-openbao.sh
Normal file
|
|
@ -0,0 +1,493 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# OpenBao source backend for render-secrets.sh. Sourced, never run directly.
|
||||||
|
#
|
||||||
|
# Functions:
|
||||||
|
# thermograph_openbao_source <env> -> merged dotenv on STDOUT
|
||||||
|
# thermograph_render_openbao <env> <out> -> source, then write <out>
|
||||||
|
# thermograph_render_openbao_centralis <env> [out] -> quoted /etc/centralis.env
|
||||||
|
#
|
||||||
|
# The first two serve the app stack (/etc/thermograph.env, raw unquoted dotenv). The
|
||||||
|
# third serves Centralis, whose file is SOURCED by bash and therefore single-quoted —
|
||||||
|
# a different shape for a different consumer, not an inconsistency.
|
||||||
|
#
|
||||||
|
# render_thermograph_secrets in render-secrets.sh dispatches to the second when
|
||||||
|
# TG_SECRETS_BACKEND=openbao, and is otherwise completely untouched — so the SOPS path
|
||||||
|
# keeps byte-identical behaviour and all new risk is confined to this file. See the
|
||||||
|
# note above thermograph_render_openbao on why the write path is duplicated here
|
||||||
|
# rather than shared, and when to consolidate.
|
||||||
|
#
|
||||||
|
# WHY OUTPUT SHAPE IS FROZEN: /etc/thermograph.env must stay raw, UNQUOTED KEY=value.
|
||||||
|
# Centralis' secrets_inventory / secrets_render compare that file textually
|
||||||
|
# (deploy/secrets/README.md:206-210), compose reads it via `env_file:`, and the Swarm
|
||||||
|
# stack's env-entrypoint.sh parses it line by line. Quoting it "properly" would be an
|
||||||
|
# improvement in isolation and a silent break in context.
|
||||||
|
|
||||||
|
# thermograph_openbao_source <env_name>
|
||||||
|
#
|
||||||
|
# Reads the common set then the per-environment set and merges them with the
|
||||||
|
# environment winning — the same last-wins semantic as the SOPS concatenation, except
|
||||||
|
# performed explicitly here because OpenBao returns a map per path rather than a
|
||||||
|
# stream that can simply be appended.
|
||||||
|
thermograph_openbao_source() {
|
||||||
|
local env_name="${1:?env name required}"
|
||||||
|
local addr="${THERMOGRAPH_BAO_ADDR:-https://10.10.0.1:8200}"
|
||||||
|
local mount="${THERMOGRAPH_BAO_MOUNT:-thermograph}"
|
||||||
|
# Explicit override first, then the per-environment value env-topology.sh derives,
|
||||||
|
# then the conventional path. Same precedence shape as render-secrets.sh:44's
|
||||||
|
# THERMOGRAPH_SECRETS_BACKEND / TG_SECRETS_BACKEND pair, for the same reason: the
|
||||||
|
# THERMOGRAPH_-prefixed name is the by-hand escape hatch, the TG_ one is what the
|
||||||
|
# deploy path sets. Falling straight through to the bare default is what made beta
|
||||||
|
# authenticate as tg-prod and get denied its own path.
|
||||||
|
local approle="${THERMOGRAPH_BAO_APPROLE:-${TG_BAO_APPROLE:-/etc/thermograph/openbao-approle}}"
|
||||||
|
local ca="${THERMOGRAPH_BAO_CACERT:-/etc/thermograph/openbao-ca.crt}"
|
||||||
|
|
||||||
|
command -v bao >/dev/null 2>&1 || {
|
||||||
|
echo "!! bao CLI not installed but this host is configured for the openbao backend" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
command -v python3 >/dev/null 2>&1 || {
|
||||||
|
echo "!! python3 required to render dotenv from OpenBao JSON" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# AppRole credentials: role_id and secret_id, one per line, 0400 root — the same
|
||||||
|
# protection level as /etc/thermograph/age.key today. Read directly if we can, else
|
||||||
|
# via sudo, mirroring how render-secrets.sh lifts the age key.
|
||||||
|
local creds
|
||||||
|
if [ -r "$approle" ]; then
|
||||||
|
creds=$(cat "$approle")
|
||||||
|
else
|
||||||
|
creds=$(sudo cat "$approle" 2>/dev/null || true)
|
||||||
|
fi
|
||||||
|
[ -n "$creds" ] || {
|
||||||
|
echo "!! cannot read OpenBao AppRole credentials at $approle (need sudo)" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
local role_id secret_id
|
||||||
|
role_id=$(printf '%s\n' "$creds" | sed -n '1p')
|
||||||
|
secret_id=$(printf '%s\n' "$creds" | sed -n '2p')
|
||||||
|
[ -n "$role_id" ] && [ -n "$secret_id" ] || {
|
||||||
|
echo "!! $approle malformed: expected role_id on line 1, secret_id on line 2" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
export BAO_ADDR="$addr"
|
||||||
|
[ -f "$ca" ] && export BAO_CACERT="$ca"
|
||||||
|
|
||||||
|
# Exchange the AppRole for a short-lived token. The token TTL is set on the role
|
||||||
|
# (see bootstrap.sh) and is deliberately measured in minutes: it only has to
|
||||||
|
# outlive one render.
|
||||||
|
local token
|
||||||
|
token=$(bao write -field=token auth/approle/login \
|
||||||
|
role_id="$role_id" secret_id="$secret_id" 2>/dev/null) || {
|
||||||
|
echo "!! OpenBao AppRole login failed for env='${env_name}'" >&2
|
||||||
|
echo "!! check that OpenBao at $addr is reachable and UNSEALED" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
export BAO_TOKEN="$token"
|
||||||
|
|
||||||
|
# Fetch. `common` is skipped for dev by the same flag the SOPS path uses — but note
|
||||||
|
# that on the OpenBao path the flag is belt, not braces: dev's AppRole is bound to
|
||||||
|
# the tg-host-dev policy, which DENIES thermograph/data/common outright. If this
|
||||||
|
# flag is ever forgotten at a call site, dev gets a 403 rather than a file full of
|
||||||
|
# production credentials. That inversion — from "the caller must remember" to "the
|
||||||
|
# store refuses" — is the main reason for this migration.
|
||||||
|
local common_json="" env_json
|
||||||
|
if [ "${THERMOGRAPH_SECRETS_SKIP_COMMON:-0}" != 1 ]; then
|
||||||
|
common_json=$(bao kv get -format=json -mount="$mount" common 2>/dev/null) || {
|
||||||
|
echo "!! cannot read ${mount}/common for env='${env_name}'" >&2
|
||||||
|
echo "!! if this is dev, THERMOGRAPH_SECRETS_SKIP_COMMON=1 was not set and the" >&2
|
||||||
|
echo "!! tg-host-dev policy correctly refused — that is the guard working." >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
fi
|
||||||
|
|
||||||
|
env_json=$(bao kv get -format=json -mount="$mount" "env/${env_name}" 2>/dev/null) || {
|
||||||
|
echo "!! cannot read ${mount}/env/${env_name}" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Merge and emit. Python rather than jq because jq is not installed on these hosts
|
||||||
|
# and python3 is (the Centralis renderer already relies on it).
|
||||||
|
#
|
||||||
|
# This step also enforces the two invariants the dotenv format cannot express:
|
||||||
|
# * a value containing a newline is REJECTED, because it would silently become
|
||||||
|
# two lines and the second would parse as a bogus KEY=value or be dropped;
|
||||||
|
# * a value containing a shell metacharacter is REJECTED, because
|
||||||
|
# /etc/thermograph.env is sourced by bash in six places (deploy.sh:133,
|
||||||
|
# deploy-stack.sh:98, both daemon entrypoints, ops-cron.yml:85, terraform), so
|
||||||
|
# `$(...)` or a backtick in a secret is a live command-execution path on the
|
||||||
|
# deploy host. Refusing here turns a latent code-execution bug into a loud
|
||||||
|
# render failure.
|
||||||
|
#
|
||||||
|
# An earlier version of this comment claimed the SOPS path had never tripped
|
||||||
|
# this "only because every current value happens to be alphanumeric". That is
|
||||||
|
# false: CENTRALIS_TOKENS is JSON and full of double quotes. It has never
|
||||||
|
# tripped THESE paths because it is not in them — it lives at
|
||||||
|
# centralis/<env> and renders to /etc/centralis.env, which is single-quoted
|
||||||
|
# precisely because raw dotenv cannot carry it. See
|
||||||
|
# thermograph_render_openbao_centralis at the end of this file.
|
||||||
|
THERMOGRAPH_BAO_COMMON="$common_json" \
|
||||||
|
THERMOGRAPH_BAO_ENV="$env_json" \
|
||||||
|
python3 - <<'PY'
|
||||||
|
import json, os, re, sys
|
||||||
|
|
||||||
|
def load(raw):
|
||||||
|
if not raw:
|
||||||
|
return {}
|
||||||
|
doc = json.loads(raw)
|
||||||
|
return doc.get("data", {}).get("data", {}) or {}
|
||||||
|
|
||||||
|
merged = {}
|
||||||
|
# common first, environment second: last-wins, matching the SOPS concatenation order.
|
||||||
|
for src in ("THERMOGRAPH_BAO_COMMON", "THERMOGRAPH_BAO_ENV"):
|
||||||
|
merged.update(load(os.environ.get(src, "")))
|
||||||
|
|
||||||
|
KEY_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||||
|
# Characters that change meaning when the file is sourced by bash.
|
||||||
|
UNSAFE = set("`$\\\"'")
|
||||||
|
|
||||||
|
bad = []
|
||||||
|
lines = []
|
||||||
|
# Validate EVERYTHING before emitting a single byte. Printing as we go would leave
|
||||||
|
# partial output on stdout when a later key fails, and render-secrets.sh's whole
|
||||||
|
# discipline is that a failed source never produces a partial env file.
|
||||||
|
for key in sorted(merged):
|
||||||
|
if not KEY_RE.match(key):
|
||||||
|
bad.append(f"{key}: not a valid env var name")
|
||||||
|
continue
|
||||||
|
val = merged[key]
|
||||||
|
if val is None:
|
||||||
|
val = ""
|
||||||
|
if not isinstance(val, str):
|
||||||
|
val = json.dumps(val, separators=(",", ":"))
|
||||||
|
if "\n" in val or "\r" in val:
|
||||||
|
bad.append(f"{key}: value contains a newline (would corrupt the dotenv)")
|
||||||
|
continue
|
||||||
|
if UNSAFE & set(val):
|
||||||
|
bad.append(
|
||||||
|
f"{key}: value contains a shell metacharacter (one of ` $ \\ \" ') and "
|
||||||
|
f"/etc/thermograph.env is sourced by bash — refusing"
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
lines.append(f"{key}={val}")
|
||||||
|
|
||||||
|
if bad:
|
||||||
|
sys.stderr.write("!! refusing to render; unsafe values:\n")
|
||||||
|
for line in bad:
|
||||||
|
sys.stderr.write(f"!! {line}\n")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
if not lines:
|
||||||
|
sys.stderr.write("!! refusing to render: OpenBao returned zero keys\n")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
sys.stdout.write("\n".join(lines) + "\n")
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# thermograph_render_openbao <env_name> <out_file>
|
||||||
|
#
|
||||||
|
# The OpenBao equivalent of render_thermograph_secrets: source, then write.
|
||||||
|
#
|
||||||
|
# ON THE DUPLICATED WRITE PATH. The write logic below mirrors
|
||||||
|
# render-secrets.sh:130-154 rather than being factored out and shared. That is a
|
||||||
|
# deliberate, temporary choice, not an oversight:
|
||||||
|
#
|
||||||
|
# render-secrets.sh is on the deploy path for all three environments and there is no
|
||||||
|
# test suite in front of it (infra/CLAUDE.md: "Shell here runs as root over SSH
|
||||||
|
# against live hosts with no test suite in front of it"). Extracting the write path
|
||||||
|
# would mean the SOPS render — which currently works — starts flowing through newly
|
||||||
|
# moved code that cannot be tested end-to-end from here. An early-return branch keeps
|
||||||
|
# the SOPS path byte-identical and confines all new risk to the new backend.
|
||||||
|
#
|
||||||
|
# CONSOLIDATE THESE after prod has been on the OpenBao backend for a full release
|
||||||
|
# cycle and the SOPS path is being deleted anyway. Until then, a change to one write
|
||||||
|
# path must be made in both — which is exactly the cost being accepted here.
|
||||||
|
thermograph_render_openbao() {
|
||||||
|
local env_name="${1:?env name required}"
|
||||||
|
local out="${2:?output path required}"
|
||||||
|
|
||||||
|
echo "==> Rendering $out from OpenBao (env=${env_name})"
|
||||||
|
|
||||||
|
local tmp; tmp=$(mktemp)
|
||||||
|
# Holds DECRYPTED secrets. Removed on every exit path. Deliberately not a
|
||||||
|
# `trap ... RETURN` — see render-secrets.sh:82-91 for why that is actively wrong in
|
||||||
|
# a sourced function (the trap persists into the caller and re-fires on its next
|
||||||
|
# `source`, where the local is unset and `set -u` makes it fatal and silent).
|
||||||
|
if ! thermograph_openbao_source "$env_name" > "$tmp"; then
|
||||||
|
rm -f "$tmp"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# An empty render must never reach $out. The source function already refuses a
|
||||||
|
# zero-key result, so this is a second line of defence rather than the only one:
|
||||||
|
# a truncated /etc/thermograph.env is the single worst outcome available here,
|
||||||
|
# because thermograph.service uses `EnvironmentFile=-` (missing is NOT fatal) and
|
||||||
|
# the app self-generates AUTH_SECRET and the VAPID pair when they are absent. The
|
||||||
|
# result would be a green deploy that silently invalidated every session and
|
||||||
|
# deleted every push subscription.
|
||||||
|
if [ ! -s "$tmp" ]; then
|
||||||
|
echo "!! OpenBao render produced an empty file; refusing to write $out" >&2
|
||||||
|
rm -f "$tmp"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
local rc=0
|
||||||
|
if [ -f "$out" ] && [ -w "$out" ]; then
|
||||||
|
cat "$tmp" > "$out" || rc=1
|
||||||
|
elif install -m 0640 "$tmp" "$out" 2>/dev/null; then :
|
||||||
|
elif sudo install -m 0640 -o "$(id -un)" -g "$(id -gn)" "$tmp" "$out" 2>/dev/null; then :
|
||||||
|
else
|
||||||
|
echo "!! cannot write $out (need file write access or passwordless sudo)" >&2
|
||||||
|
rc=1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$rc" = 0 ] && [ -n "${THERMOGRAPH_SECRETS_ENV_FILE_MIRROR:-}" ]; then
|
||||||
|
if ! mkdir -p "$(dirname "$THERMOGRAPH_SECRETS_ENV_FILE_MIRROR")" \
|
||||||
|
|| ! install -m 0600 "$tmp" "$THERMOGRAPH_SECRETS_ENV_FILE_MIRROR"; then
|
||||||
|
echo "!! cannot write mirror at $THERMOGRAPH_SECRETS_ENV_FILE_MIRROR" >&2
|
||||||
|
rc=1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
rm -f "$tmp"
|
||||||
|
return "$rc"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# CENTRALIS
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# /etc/centralis.env is a different file, with a different consumer, and therefore a
|
||||||
|
# different output shape. render-secrets.sh:206-233 carries the full argument; the
|
||||||
|
# short version is that it is consumed by exactly one thing and that thing is a shell:
|
||||||
|
#
|
||||||
|
# sudo bash -c 'set -a && . /etc/centralis.env && set +a && docker compose up'
|
||||||
|
#
|
||||||
|
# `set -a; . file` runs every line through bash's whole expansion pipeline — quote
|
||||||
|
# removal, parameter expansion, command substitution, field splitting. Raw dotenv is
|
||||||
|
# therefore not a safe representation here. CENTRALIS_TOKENS is JSON: rendered
|
||||||
|
# unquoted, bash strips its quotes, and Centralis fails CLOSED on the malformed result
|
||||||
|
# by serving a single `shared` identity — indistinguishable from the file never having
|
||||||
|
# been written. That is the 2026-07-24 incident. So this path emits POSIX
|
||||||
|
# single-quoted assignments and PROVES them by sourcing the result in a clean shell
|
||||||
|
# before anything touches $dest.
|
||||||
|
#
|
||||||
|
# This mirrors render_centralis_secrets rather than sharing with it, for the same
|
||||||
|
# reason given above thermograph_render_openbao: the SOPS path stays byte-identical
|
||||||
|
# and all new risk stays in this file. Consolidate the two when the SOPS path is being
|
||||||
|
# deleted anyway, not before.
|
||||||
|
|
||||||
|
# _thermograph_openbao_login <approle_file> <addr> <ca> -> token on STDOUT
|
||||||
|
#
|
||||||
|
# Separate from the login inline in thermograph_openbao_source deliberately: that
|
||||||
|
# function is now exercised by prod and beta parity and is left untouched.
|
||||||
|
_thermograph_openbao_login() {
|
||||||
|
local approle="$1" addr="$2" ca="$3"
|
||||||
|
local creds role_id secret_id
|
||||||
|
if [ -r "$approle" ]; then
|
||||||
|
creds=$(cat "$approle")
|
||||||
|
else
|
||||||
|
creds=$(sudo cat "$approle" 2>/dev/null || true)
|
||||||
|
fi
|
||||||
|
[ -n "$creds" ] || {
|
||||||
|
echo "!! cannot read OpenBao AppRole credentials at $approle (need sudo)" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
role_id=$(printf '%s\n' "$creds" | sed -n '1p')
|
||||||
|
secret_id=$(printf '%s\n' "$creds" | sed -n '2p')
|
||||||
|
[ -n "$role_id" ] && [ -n "$secret_id" ] || {
|
||||||
|
echo "!! $approle malformed: expected role_id on line 1, secret_id on line 2" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
export BAO_ADDR="$addr"
|
||||||
|
[ -f "$ca" ] && export BAO_CACERT="$ca"
|
||||||
|
bao write -field=token auth/approle/login \
|
||||||
|
role_id="$role_id" secret_id="$secret_id" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
# thermograph_render_openbao_centralis <env_name> [dest]
|
||||||
|
thermograph_render_openbao_centralis() {
|
||||||
|
local env_name="${1:?env name required}"
|
||||||
|
local dest="${2:-${CENTRALIS_RENDER_DEST:-/etc/centralis.env}}"
|
||||||
|
local addr="${THERMOGRAPH_BAO_ADDR:-https://10.10.0.1:8200}"
|
||||||
|
local mount="${THERMOGRAPH_BAO_MOUNT:-thermograph}"
|
||||||
|
local ca="${THERMOGRAPH_BAO_CACERT:-/etc/thermograph/openbao-ca.crt}"
|
||||||
|
local approle="${THERMOGRAPH_BAO_CENTRALIS_APPROLE:-/etc/thermograph/openbao-approle-centralis}"
|
||||||
|
|
||||||
|
command -v bao >/dev/null 2>&1 || {
|
||||||
|
echo "!! bao CLI not installed but this host is configured for the openbao backend" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
command -v python3 >/dev/null 2>&1 || {
|
||||||
|
echo "!! python3 not installed; needed to quote values safely for a sourced file" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "==> Rendering ${dest} from OpenBao (${mount}/centralis/${env_name})"
|
||||||
|
|
||||||
|
local token
|
||||||
|
token=$(_thermograph_openbao_login "$approle" "$addr" "$ca") || return 1
|
||||||
|
[ -n "$token" ] || {
|
||||||
|
echo "!! OpenBao AppRole login failed for centralis" >&2
|
||||||
|
echo "!! check that OpenBao at $addr is reachable and UNSEALED" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
export BAO_TOKEN="$token"
|
||||||
|
|
||||||
|
# ONE 0700 temp directory holding plaintext, so cleanup is a single rm on every
|
||||||
|
# path. Deliberately not a `trap ... RETURN`: this file is SOURCED, and such a trap
|
||||||
|
# persists into the caller's shell and re-fires on its next `source`.
|
||||||
|
local work rc=0
|
||||||
|
work=$(mktemp -d) || { echo "!! mktemp -d failed" >&2; return 1; }
|
||||||
|
chmod 700 "$work"
|
||||||
|
|
||||||
|
# Subshell so no failure path can skip the cleanup, and so umask cannot leak out.
|
||||||
|
# Every step carries its own `|| exit 1`: `set -e` is DISABLED inside a compound
|
||||||
|
# command that is the left operand of `||`, which this subshell is.
|
||||||
|
(
|
||||||
|
umask 077
|
||||||
|
|
||||||
|
bao kv get -format=json -mount="$mount" "centralis/${env_name}" \
|
||||||
|
> "$work/kv.json" 2>/dev/null || {
|
||||||
|
echo "!! cannot read ${mount}/centralis/${env_name}" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
python3 - "$work/kv.json" "$work/plain.json" <<'UNWRAP' || exit 1
|
||||||
|
import json, sys
|
||||||
|
doc = json.load(open(sys.argv[1], encoding="utf-8"))
|
||||||
|
data = doc.get("data", {}).get("data", {}) or {}
|
||||||
|
if not data:
|
||||||
|
sys.stderr.write("!! OpenBao returned zero keys for centralis\n")
|
||||||
|
sys.exit(1)
|
||||||
|
json.dump(data, open(sys.argv[2], "w", encoding="utf-8"))
|
||||||
|
UNWRAP
|
||||||
|
|
||||||
|
python3 - "$work/plain.json" "$work/out.env" "$work/want.json" <<'EMIT' || exit 1
|
||||||
|
import json, re, sys
|
||||||
|
|
||||||
|
src, out_env, want_json = sys.argv[1], sys.argv[2], sys.argv[3]
|
||||||
|
NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
|
||||||
|
|
||||||
|
|
||||||
|
def shell_quote(v):
|
||||||
|
"""POSIX single-quoting. Inside '...' every byte is literal except ' itself,
|
||||||
|
which is closed, backslash-escaped and reopened. There is no escape sequence
|
||||||
|
to get wrong and no character class to keep up to date: it is total, and
|
||||||
|
total is the only property worth having for a file bash will expand."""
|
||||||
|
return "'" + v.replace("'", "'\\''") + "'"
|
||||||
|
|
||||||
|
|
||||||
|
def scalar(k, v):
|
||||||
|
if isinstance(v, str):
|
||||||
|
return v
|
||||||
|
if isinstance(v, bool): # before int: bool is an int in Python
|
||||||
|
return "true" if v else "false"
|
||||||
|
if isinstance(v, (int, float)):
|
||||||
|
return json.dumps(v)
|
||||||
|
if v is None:
|
||||||
|
return ""
|
||||||
|
sys.stderr.write(
|
||||||
|
"!! %s is a %s, but an env file holds only scalars\n" % (k, type(v).__name__))
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
data = json.load(open(src, encoding="utf-8"))
|
||||||
|
want, lines = {}, []
|
||||||
|
for k, v in data.items():
|
||||||
|
if not NAME.match(k):
|
||||||
|
sys.stderr.write("!! %r is not a usable shell variable name\n" % (k,))
|
||||||
|
sys.exit(1)
|
||||||
|
want[k] = scalar(k, v)
|
||||||
|
lines.append("%s=%s\n" % (k, shell_quote(want[k])))
|
||||||
|
|
||||||
|
with open(out_env, "w", encoding="utf-8") as fh:
|
||||||
|
fh.write("# Rendered by infra/deploy/render-secrets-openbao.sh from OpenBao.\n")
|
||||||
|
fh.write("# DO NOT EDIT BY HAND: the next render overwrites this file, and a\n")
|
||||||
|
fh.write("# hand-edit is how the token registry was lost on 2026-07-24.\n")
|
||||||
|
fh.write("# Rotate with `bao kv put` against thermograph/centralis/<env> instead.\n")
|
||||||
|
fh.writelines(lines)
|
||||||
|
with open(want_json, "w", encoding="utf-8") as fh:
|
||||||
|
json.dump(want, fh)
|
||||||
|
sys.stderr.write(" %d keys quoted\n" % len(want))
|
||||||
|
EMIT
|
||||||
|
|
||||||
|
# PROVE IT. Source the rendered file exactly the way the deploy does — clean
|
||||||
|
# environment, `set -a`, nothing inherited that could mask a dropped key — and
|
||||||
|
# read every value back. This is the check the 2026-07-24 file could not pass.
|
||||||
|
# shellcheck disable=SC2016 # single quotes are the point: "$1" must be expanded
|
||||||
|
# by the inner `bash -c` against the argument after `_`, not by this shell.
|
||||||
|
env -i PATH="$PATH" bash -c \
|
||||||
|
'set -a; . "$1"; set +a; exec python3 -c "
|
||||||
|
import json, os, sys
|
||||||
|
sys.stdout.write(json.dumps(dict(os.environ)))"' _ "$work/out.env" \
|
||||||
|
> "$work/got.json" || exit 1
|
||||||
|
|
||||||
|
python3 - "$work/want.json" "$work/got.json" <<'VERIFY' || exit 1
|
||||||
|
import json, sys
|
||||||
|
|
||||||
|
want = json.load(open(sys.argv[1], encoding="utf-8"))
|
||||||
|
got = json.load(open(sys.argv[2], encoding="utf-8"))
|
||||||
|
|
||||||
|
bad = []
|
||||||
|
for k, v in want.items():
|
||||||
|
if k not in got:
|
||||||
|
bad.append("%s: not set at all after sourcing" % k)
|
||||||
|
elif got[k] != v:
|
||||||
|
# NEVER print either value. The difference is the finding; the content is
|
||||||
|
# not, and this runs inside a deploy log.
|
||||||
|
bad.append("%s: survives sourcing as a DIFFERENT value (len %d -> %d)"
|
||||||
|
% (k, len(v), len(got[k])))
|
||||||
|
if bad:
|
||||||
|
sys.stderr.write("!! the rendered file does not round-trip through bash:\n")
|
||||||
|
for b in bad:
|
||||||
|
sys.stderr.write("!! %s\n" % b)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
# CENTRALIS_TOKENS is the reason for all of the above: JSON, full of double quotes,
|
||||||
|
# in a file bash expands. Centralis fails CLOSED on malformed JSON — it drops to the
|
||||||
|
# single `shared` identity, which looks exactly like the registry never having been
|
||||||
|
# configured. Refuse to write a file that would do that.
|
||||||
|
raw = got.get("CENTRALIS_TOKENS", "")
|
||||||
|
if raw:
|
||||||
|
try:
|
||||||
|
reg = json.loads(raw)
|
||||||
|
except ValueError as exc:
|
||||||
|
sys.stderr.write("!! CENTRALIS_TOKENS is not valid JSON after sourcing: %s\n" % exc)
|
||||||
|
sys.exit(1)
|
||||||
|
if not isinstance(reg, dict) or not reg:
|
||||||
|
sys.stderr.write("!! CENTRALIS_TOKENS must be a non-empty JSON object\n")
|
||||||
|
sys.exit(1)
|
||||||
|
if not all(isinstance(s, str) and isinstance(t, str) and t for s, t in reg.items()):
|
||||||
|
sys.stderr.write("!! CENTRALIS_TOKENS must map subject -> non-empty token string\n")
|
||||||
|
sys.exit(1)
|
||||||
|
# Subjects are names on audit lines, not credentials. Printing them is the whole
|
||||||
|
# point: it is how an operator sees at a glance that nobody was lost.
|
||||||
|
sys.stderr.write(" CENTRALIS_TOKENS parses: %d subject(s) — %s\n"
|
||||||
|
% (len(reg), ", ".join(sorted(reg))))
|
||||||
|
|
||||||
|
sys.stderr.write(" %d keys survive `set -a; . <file>` byte-for-byte\n" % len(want))
|
||||||
|
VERIFY
|
||||||
|
) || rc=1
|
||||||
|
|
||||||
|
# Only now, with a file read back through bash and matched value for value, does
|
||||||
|
# anything touch $dest. 0600: four bearer credentials for the estate's control
|
||||||
|
# plane, with no group that needs them.
|
||||||
|
if [ "$rc" -eq 0 ]; then
|
||||||
|
if [ -f "$dest" ] && [ -w "$dest" ]; then
|
||||||
|
cat "$work/out.env" > "$dest" || rc=1
|
||||||
|
elif install -m 0600 "$work/out.env" "$dest" 2>/dev/null; then :
|
||||||
|
elif sudo install -m 0600 -o "$(id -un)" -g "$(id -gn)" "$work/out.env" "$dest" 2>/dev/null; then :
|
||||||
|
else
|
||||||
|
echo "!! cannot write ${dest} (need file write access or passwordless sudo)" >&2
|
||||||
|
rc=1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "!! render aborted; ${dest} left untouched" >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
rm -rf "$work"
|
||||||
|
return "$rc"
|
||||||
|
}
|
||||||
|
|
@ -15,12 +15,53 @@
|
||||||
# (prod|beta|dev). A host without them keeps whatever /etc/thermograph.env it already
|
# (prod|beta|dev). A host without them keeps whatever /etc/thermograph.env it already
|
||||||
# has — so merging this is a safe no-op until a host is deliberately migrated. See
|
# has — so merging this is a safe no-op until a host is deliberately migrated. See
|
||||||
# deploy/secrets/README.md.
|
# deploy/secrets/README.md.
|
||||||
|
# render_thermograph_secrets <repo> [env_name] [out_file]
|
||||||
|
#
|
||||||
|
# env_name and out_file are explicit since vps2 began running two environments.
|
||||||
|
# A host-wide marker file cannot name two environments, and one output path
|
||||||
|
# cannot hold two renders: on vps2 prod renders prod.yaml to /etc/thermograph.env
|
||||||
|
# while beta renders beta.yaml to /etc/thermograph-beta.env. Both arguments keep
|
||||||
|
# their pre-split defaults (marker file, /etc/thermograph.env), so a
|
||||||
|
# single-environment host and a by-hand run behave exactly as before.
|
||||||
render_thermograph_secrets() {
|
render_thermograph_secrets() {
|
||||||
local repo="${1:-.}" # repo root holding deploy/secrets
|
local repo="${1:-.}" # repo root holding deploy/secrets
|
||||||
local key="${THERMOGRAPH_AGE_KEY:-/etc/thermograph/age.key}"
|
local key="${THERMOGRAPH_AGE_KEY:-/etc/thermograph/age.key}"
|
||||||
local marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
|
local marker="${THERMOGRAPH_SECRETS_ENV_FILE:-/etc/thermograph/secrets-env}"
|
||||||
local env_name
|
local env_name="${2:-}"
|
||||||
env_name=$(cat "$marker" 2>/dev/null || true)
|
local out="${3:-/etc/thermograph.env}"
|
||||||
|
[ -n "$env_name" ] || env_name=$(cat "$marker" 2>/dev/null || true)
|
||||||
|
|
||||||
|
# Backend dispatch. TG_SECRETS_BACKEND comes from env-topology.sh (per environment,
|
||||||
|
# because vps2 runs two); THERMOGRAPH_SECRETS_BACKEND overrides it for a by-hand run.
|
||||||
|
# Default is sops, so an unmigrated host and an unset caller both behave exactly as
|
||||||
|
# before this branch existed.
|
||||||
|
#
|
||||||
|
# The OpenBao path returns early rather than threading a conditional through the
|
||||||
|
# rest of this function. That is on purpose: everything below — the age-key sudo
|
||||||
|
# lift, the two-file last-wins concatenation, the three-way write, the mirror — stays
|
||||||
|
# on precisely the code path it has always been on, so migrating cannot regress the
|
||||||
|
# backend that is still authoritative for prod.
|
||||||
|
local backend="${THERMOGRAPH_SECRETS_BACKEND:-${TG_SECRETS_BACKEND:-sops}}"
|
||||||
|
if [ "$backend" = openbao ]; then
|
||||||
|
if [ -z "$env_name" ]; then
|
||||||
|
echo "!! TG_SECRETS_BACKEND=openbao but no environment name was resolved" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
local bao_lib="${repo}/deploy/render-secrets-openbao.sh"
|
||||||
|
if [ ! -f "$bao_lib" ]; then
|
||||||
|
echo "!! TG_SECRETS_BACKEND=openbao but $bao_lib is missing" >&2
|
||||||
|
echo "!! (a checkout predating the OpenBao backend cannot render this way)" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
. "$bao_lib"
|
||||||
|
thermograph_render_openbao "$env_name" "$out"
|
||||||
|
return $?
|
||||||
|
fi
|
||||||
|
if [ "$backend" != sops ]; then
|
||||||
|
echo "!! unknown secrets backend '${backend}' (expected sops|openbao)" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
# The per-host file is required; common.yaml is optional so the initial cutover can
|
# The per-host file is required; common.yaml is optional so the initial cutover can
|
||||||
# seed each host as an exact copy of its live env (byte-identical render, no value
|
# seed each host as an exact copy of its live env (byte-identical render, no value
|
||||||
|
|
@ -31,7 +72,7 @@ render_thermograph_secrets() {
|
||||||
# (a) Not configured for SOPS at all — no env marker, or no age key. The LAN
|
# (a) Not configured for SOPS at all — no env marker, or no age key. The LAN
|
||||||
# dev box is legitimately in this state. Saying so and succeeding is right.
|
# dev box is legitimately in this state. Saying so and succeeding is right.
|
||||||
if [ -z "$env_name" ] || [ ! -f "$key" ]; then
|
if [ -z "$env_name" ] || [ ! -f "$key" ]; then
|
||||||
echo "==> SOPS secrets not configured here; using existing /etc/thermograph.env"
|
echo "==> SOPS secrets not configured here; using existing $out"
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|
@ -57,7 +98,7 @@ render_thermograph_secrets() {
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo "==> Rendering /etc/thermograph.env from deploy/secrets (common + ${env_name})"
|
echo "==> Rendering $out from deploy/secrets (common + ${env_name})"
|
||||||
# The age private key is root-owned (0400). Read it directly if we can, else via
|
# The age private key is root-owned (0400). Read it directly if we can, else via
|
||||||
# sudo into SOPS_AGE_KEY — so the key never has to be readable by the deploy user.
|
# sudo into SOPS_AGE_KEY — so the key never has to be readable by the deploy user.
|
||||||
# (The render needs sudo to write /etc/thermograph.env below anyway.)
|
# (The render needs sudo to write /etc/thermograph.env below anyway.)
|
||||||
|
|
@ -119,12 +160,12 @@ render_thermograph_secrets() {
|
||||||
# failed in-place `cat` write to status 0. Capture the status instead, so a
|
# failed in-place `cat` write to status 0. Capture the status instead, so a
|
||||||
# half-written /etc/thermograph.env fails loudly rather than deploying stale.
|
# half-written /etc/thermograph.env fails loudly rather than deploying stale.
|
||||||
local rc=0
|
local rc=0
|
||||||
if [ -f /etc/thermograph.env ] && [ -w /etc/thermograph.env ]; then
|
if [ -f "$out" ] && [ -w "$out" ]; then
|
||||||
cat "$tmp" > /etc/thermograph.env || rc=1
|
cat "$tmp" > "$out" || rc=1
|
||||||
elif install -m 0640 "$tmp" /etc/thermograph.env 2>/dev/null; then :
|
elif install -m 0640 "$tmp" "$out" 2>/dev/null; then :
|
||||||
elif sudo install -m 0640 -o "$(id -un)" -g "$(id -gn)" "$tmp" /etc/thermograph.env 2>/dev/null; then :
|
elif sudo install -m 0640 -o "$(id -un)" -g "$(id -gn)" "$tmp" "$out" 2>/dev/null; then :
|
||||||
else
|
else
|
||||||
echo "!! cannot write /etc/thermograph.env (need file write access or passwordless sudo)" >&2
|
echo "!! cannot write $out (need file write access or passwordless sudo)" >&2
|
||||||
rc=1
|
rc=1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|
@ -231,6 +272,30 @@ render_centralis_secrets() {
|
||||||
echo "!! no environment name at ${marker} — cannot tell which Centralis vault to render" >&2
|
echo "!! no environment name at ${marker} — cannot tell which Centralis vault to render" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Same early-return dispatch as render_thermograph_secrets, for the same reason
|
||||||
|
# given there: everything below — the age-key sudo lift, the JSON decrypt, the
|
||||||
|
# quote-and-prove pipeline, the three-way write — stays on exactly the code path it
|
||||||
|
# has always been on, so this cannot regress the backend still authoritative for
|
||||||
|
# Centralis. The OpenBao path needs no age key, so it returns before that check.
|
||||||
|
local backend="${THERMOGRAPH_SECRETS_BACKEND:-${TG_SECRETS_BACKEND:-sops}}"
|
||||||
|
if [ "$backend" = openbao ]; then
|
||||||
|
local bao_lib="${repo}/deploy/render-secrets-openbao.sh"
|
||||||
|
if [ ! -f "$bao_lib" ]; then
|
||||||
|
echo "!! TG_SECRETS_BACKEND=openbao but $bao_lib is missing" >&2
|
||||||
|
echo "!! (a checkout predating the OpenBao backend cannot render this way)" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
. "$bao_lib"
|
||||||
|
thermograph_render_openbao_centralis "$env_name" "$dest"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
if [ "$backend" != sops ]; then
|
||||||
|
echo "!! unknown secrets backend '${backend}' (expected sops|openbao)" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
if [ ! -f "$key" ]; then
|
if [ ! -f "$key" ]; then
|
||||||
echo "!! no age key at ${key}; this host cannot decrypt the vault" >&2
|
echo "!! no age key at ${key}; this host cannot decrypt the vault" >&2
|
||||||
return 1
|
return 1
|
||||||
|
|
|
||||||
|
|
@ -16,8 +16,8 @@ no per-host duplication.
|
||||||
|------|-------|------------|
|
|------|-------|------------|
|
||||||
| `common.yaml` | The 16 values identical on **prod and beta** — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. **Not layered under `dev`** — see below | ✅ |
|
| `common.yaml` | The 16 values identical on **prod and beta** — VAPID keypair, metrics token, IndexNow key, `REGISTRY_TOKEN`, S3 endpoint/bucket and both S3 keypairs, plus shared non-secret config. **Not layered under `dev`** — see below | ✅ |
|
||||||
| `prod.yaml` | Prod's own — the three held-back credentials below, sizing (`APP_CPUS`, `DB_CPUS`, `DB_MEMORY`, `WORKERS`), `THERMOGRAPH_BASE_URL`, and the Discord + mail credentials that exist nowhere else | ✅ |
|
| `prod.yaml` | Prod's own — the three held-back credentials below, sizing (`APP_CPUS`, `DB_CPUS`, `DB_MEMORY`, `WORKERS`), `THERMOGRAPH_BASE_URL`, and the Discord + mail credentials that exist nowhere else | ✅ |
|
||||||
| `beta.yaml` | Beta's own — the three held-back credentials, sizing, `THERMOGRAPH_BASE_URL` | ✅ |
|
| `beta.yaml` | Beta's own — the three held-back credentials, `THERMOGRAPH_BASE_URL`, `APP_CPUS`/`WORKERS`, plus `THERMOGRAPH_MAIL_BACKEND=console` and `THERMOGRAPH_DISCORD_BOT=0`. Its `POSTGRES_PASSWORD` and `THERMOGRAPH_DATABASE_URL` name beta's **own role and database** (`thermograph_beta`) on the instance it shares with prod. `DB_CPUS`/`DB_MEMORY` were **removed**: beta no longer runs a database of its own, and leaving them would have implied it sized one — the shared instance is sized by `prod.yaml`'s values (see `deploy/stack/thermograph-stack.yml`'s `db` service) | ✅ |
|
||||||
| `dev.yaml` | The LAN dev box's own — **self-contained**, 12 values, no production credential among them | ✅ |
|
| `dev.yaml` | Dev's own (vps1) — **self-contained**, 12 values, no production credential among them | ✅ |
|
||||||
| `centralis.prod.yaml` | **Centralis's** nine variables, rendered to `/etc/centralis.env` on prod. A different service, a different output file, a different renderer — see below | ✅ |
|
| `centralis.prod.yaml` | **Centralis's** nine variables, rendered to `/etc/centralis.env` on prod. A different service, a different output file, a different renderer — see below | ✅ |
|
||||||
| `example.yaml` | Format reference / CI fixture (fake values) | ✅ |
|
| `example.yaml` | Format reference / CI fixture (fake values) | ✅ |
|
||||||
| `../../.sops.yaml` | Which age recipient files are encrypted to (plaintext config) | — |
|
| `../../.sops.yaml` | Which age recipient files are encrypted to (plaintext config) | — |
|
||||||
|
|
@ -25,28 +25,55 @@ no per-host duplication.
|
||||||
### Three credentials are deliberately NOT in `common.yaml`
|
### Three credentials are deliberately NOT in `common.yaml`
|
||||||
|
|
||||||
`POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET` and `THERMOGRAPH_DATABASE_URL`
|
`POSTGRES_PASSWORD`, `THERMOGRAPH_AUTH_SECRET` and `THERMOGRAPH_DATABASE_URL`
|
||||||
hold **identical values on prod and beta today**, so by the mechanical rule they
|
hold **different values on prod and beta today** (beta has its own database
|
||||||
belong in `common.yaml`. They are kept per-host anyway.
|
role/password on the shared instance, and its own auth secret), so unlike the
|
||||||
|
rest of `common.yaml` they were never candidates for sharing on values alone.
|
||||||
|
They stay per-environment for a reason that matters more now than it used to:
|
||||||
|
|
||||||
These are the credentials that let one environment act as another: with them, a
|
**These are the credentials that let one environment act as another.** With
|
||||||
foothold on beta is a foothold on prod's database and prod's session signing.
|
matching values, a foothold in one environment's rendered env file is a
|
||||||
Beta is the *more* exposed box — it serves public Forgejo and Grafana. They match
|
foothold on the other's database and the other's session signing. That used
|
||||||
only because beta was seeded from prod, not because the two are meant to be one
|
to matter because beta and prod were separate boxes; it matters for a
|
||||||
system.
|
different reason now that they're co-resident on vps2 — keeping the
|
||||||
|
credentials themselves separate is what stops co-location on one host from
|
||||||
|
also becoming equivalence at the credential level. `deploy/db/provision-env-db.sh`
|
||||||
|
enforces the same boundary from the database side: beta's role
|
||||||
|
(`thermograph_beta`) is `NOSUPERUSER`/`NOCREATEDB`/`NOCREATEROLE`, owns only
|
||||||
|
its own database, and `CONNECT` is revoked from `PUBLIC` on it — so even a
|
||||||
|
leaked beta credential cannot reach prod's data, and the reverse.
|
||||||
|
|
||||||
Keeping them per-host costs nothing now (same values, one extra line each) and
|
**What actually separates beta and prod now, stated plainly:** they share a
|
||||||
buys the ability to diverge: rotating prod's database password stops implying
|
host by design (see `deploy/env-topology.sh`'s header), so a host-level
|
||||||
"and beta's too". Moving them into `common.yaml` would encode the equivalence as
|
compromise of vps2 is shared between them — separate SSH credentials no
|
||||||
intentional and turn breaking it into a migration rather than an edit.
|
longer buy a host boundary between beta and prod, the way they once did when
|
||||||
|
beta and prod were different machines. What remains is isolation at the
|
||||||
|
**database** level (separate roles, separate databases, `CONNECT` revoked from
|
||||||
|
`PUBLIC`) and the **file** level (separate checkouts, separate rendered env
|
||||||
|
files, separate stack env files, separate loopback ports). Keeping
|
||||||
|
`POSTGRES_PASSWORD`/`THERMOGRAPH_AUTH_SECRET`/`THERMOGRAPH_DATABASE_URL` out of
|
||||||
|
a shared file is part of that file-level isolation: nothing on vps2 would ever
|
||||||
|
hand you prod's auth secret merely because you have beta's.
|
||||||
|
|
||||||
If you ever *do* want them to differ, change one file. That is the whole point.
|
**vps1 is the box that must never hold prod credentials.** It runs Forgejo,
|
||||||
|
Forgejo's CI runner, Grafana, and the `dev` environment — which runs whatever
|
||||||
|
branch is currently in flight, reviewed or not. That is the actual reason
|
||||||
|
`dev` renders `dev.yaml` **alone** and never layers `common.yaml` (see below):
|
||||||
|
not squeamishness about a dev box, but that vps1 is uniquely positioned to
|
||||||
|
leak whatever it's handed, and `common.yaml` is the fleet's shared production
|
||||||
|
credential set.
|
||||||
|
|
||||||
`dev.yaml` is the case that already differs: all three of its copies are its own,
|
Keeping the three held-back credentials per-environment costs nothing (one
|
||||||
generated on the desktop. The rule earned its keep the first time it was used.
|
line each) and buys the ability for them to diverge further, or for a future
|
||||||
|
environment to be added without inheriting anyone else's values by accident.
|
||||||
|
|
||||||
|
`dev.yaml` is the case that already differs completely: all three of its
|
||||||
|
values are its own, generated on vps1 and never shared with prod or beta.
|
||||||
|
|
||||||
At deploy the renderer concatenates `common.yaml` then `<env>.yaml`, so a **host
|
At deploy the renderer concatenates `common.yaml` then `<env>.yaml`, so a **host
|
||||||
value wins** (env_file / `source` take the last occurrence of a duplicate key).
|
value wins** (env_file / `source` take the last occurrence of a duplicate key).
|
||||||
`<env>` comes from `/etc/thermograph/secrets-env` on the box (`prod`/`beta`/`dev`).
|
`<env>` comes from an explicit `THERMOGRAPH_ENV`/argument on vps2 (which runs
|
||||||
|
two environments) or `/etc/thermograph/secrets-env` as a fallback elsewhere —
|
||||||
|
see `deploy/env-topology.sh`.
|
||||||
|
|
||||||
### `dev` is vaulted, but renders `dev.yaml` **alone**
|
### `dev` is vaulted, but renders `dev.yaml` **alone**
|
||||||
|
|
||||||
|
|
@ -65,17 +92,17 @@ is a **plaintext concatenation**, so layering it would put every one of those in
|
||||||
`/etc/thermograph.env` on the dev box and, through `env_file:`, into the
|
`/etc/thermograph.env` on the dev box and, through `env_file:`, into the
|
||||||
environment of every container in the dev stack.
|
environment of every container in the dev stack.
|
||||||
|
|
||||||
The dev box is the operator's desktop. It is also the Forgejo CI runner, whose
|
The dev box is **vps1** — the same box that runs Forgejo and its CI. Its
|
||||||
`docker`-labelled jobs get the host docker socket automounted, and the stack it
|
`docker`-labelled runner jobs get the host docker socket automounted, and the
|
||||||
runs is the `dev` branch — code that has not been through the gate that guards
|
stack it runs is the `dev` branch — code that has not been through the gate
|
||||||
prod. Handing that the estate's read-write object-storage keys and the push
|
that guards prod. Handing that the estate's read-write object-storage keys and
|
||||||
signing key is a strictly larger blast radius than anything the vault buys back.
|
the push signing key is a strictly larger blast radius than anything the vault
|
||||||
An override in `dev.yaml` would not help: last-wins governs *consumers*, but the
|
buys back. An override in `dev.yaml` would not help: last-wins governs
|
||||||
production value is still physically a line in the file.
|
*consumers*, but the production value is still physically a line in the file.
|
||||||
|
|
||||||
One value in `common.yaml` is also simply **wrong** for dev:
|
One value in `common.yaml` is also simply **wrong** for dev:
|
||||||
`THERMOGRAPH_COOKIE_SECURE=1` is right for the TLS hosts and silently breaks login
|
`THERMOGRAPH_COOKIE_SECURE=1` is right for the TLS hosts and silently breaks login
|
||||||
over dev's plain-HTTP LAN URL. `dev.yaml` sets `0`.
|
over dev's plain-HTTP mesh-only URL. `dev.yaml` sets `0`.
|
||||||
|
|
||||||
So `dev.yaml` is self-contained. Where the value is shared-but-harmless
|
So `dev.yaml` is self-contained. Where the value is shared-but-harmless
|
||||||
(`PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`) it carries its own copy — a
|
(`PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`) it carries its own copy — a
|
||||||
|
|
@ -87,8 +114,8 @@ duplicated constant is the cheap half of this trade.
|
||||||
|-----|-----|
|
|-----|-----|
|
||||||
| `POSTGRES_PASSWORD`, `THERMOGRAPH_DATABASE_URL` | dev's own. Deliberately the weak, well-known `thermograph-dev` — the value dev's `pgdata` volume is already initialized with, and the DB is never published off the compose network. Kept per-host for the reason below, and *not* shared with prod |
|
| `POSTGRES_PASSWORD`, `THERMOGRAPH_DATABASE_URL` | dev's own. Deliberately the weak, well-known `thermograph-dev` — the value dev's `pgdata` volume is already initialized with, and the DB is never published off the compose network. Kept per-host for the reason below, and *not* shared with prod |
|
||||||
| `THERMOGRAPH_AUTH_SECRET` | dev's **own**, freshly generated. Fixes a live crash loop: the `daemon` service refuses to start without it (it derives the `/internal/*` token from it), so dev's daemon had been restarting every 60s. Prod's value must never be here — it signs sessions and verification links |
|
| `THERMOGRAPH_AUTH_SECRET` | dev's **own**, freshly generated. Fixes a live crash loop: the `daemon` service refuses to start without it (it derives the `/internal/*` token from it), so dev's daemon had been restarting every 60s. Prod's value must never be here — it signs sessions and verification links |
|
||||||
| `THERMOGRAPH_BASE_URL` | `http://10.0.1.216:8137`. Unset, the app defaults to `https://thermograph.org`, so dev's IndexNow pings and Discord links claimed to be prod |
|
| `THERMOGRAPH_BASE_URL` | `http://10.10.0.2:8137` (vps1's mesh address). Unset, the app defaults to `https://thermograph.org`, so dev's IndexNow pings and Discord links claimed to be prod |
|
||||||
| `THERMOGRAPH_COOKIE_SECURE=0` | plain HTTP on the LAN |
|
| `THERMOGRAPH_COOKIE_SECURE=0` | plain HTTP on the mesh-only URL |
|
||||||
| `THERMOGRAPH_MAIL_BACKEND=console`, `THERMOGRAPH_DISCORD_BOT=0` | explicit fail-safes. Both are already the code defaults; stating them means dev cannot start mailing or open a Discord gateway by inheriting a value |
|
| `THERMOGRAPH_MAIL_BACKEND=console`, `THERMOGRAPH_DISCORD_BOT=0` | explicit fail-safes. Both are already the code defaults; stating them means dev cannot start mailing or open a Discord gateway by inheriting a value |
|
||||||
| `PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`, `DB_MEMORY`, `WORKERS` | non-secret config that would otherwise have come from `common.yaml` |
|
| `PORT`, `THERMOGRAPH_BASE`, `TIMESCALEDB_TAG`, `DB_MEMORY`, `WORKERS` | non-secret config that would otherwise have come from `common.yaml` |
|
||||||
|
|
||||||
|
|
@ -104,8 +131,8 @@ duplicated constant is the cheap half of this trade.
|
||||||
(`backend/indexnow.py`). Prod's key is what authenticates URL submissions *for
|
(`backend/indexnow.py`). Prod's key is what authenticates URL submissions *for
|
||||||
thermograph.org*.
|
thermograph.org*.
|
||||||
- **`THERMOGRAPH_METRICS_TOKEN`.** With no token `/api/v2/metrics` is
|
- **`THERMOGRAPH_METRICS_TOKEN`.** With no token `/api/v2/metrics` is
|
||||||
direct-loopback-only, which is exactly right on a LAN box; dev's Alloy agent
|
direct-loopback-only, which is exactly right on a mesh-only box; dev's Alloy
|
||||||
ships logs, not metrics, so nothing scrapes it.
|
agent ships logs, not metrics, so nothing scrapes it.
|
||||||
- **`REGISTRY_TOKEN`.** dev pulls with the persistent `docker login` credential
|
- **`REGISTRY_TOKEN`.** dev pulls with the persistent `docker login` credential
|
||||||
already in the host's docker config, like the prod/beta CI paths. A vault copy
|
already in the host's docker config, like the prod/beta CI paths. A vault copy
|
||||||
would only duplicate a credential the box already has into a more readable file.
|
would only duplicate a credential the box already has into a more readable file.
|
||||||
|
|
@ -224,12 +251,44 @@ files), so layering would only push the VAPID private key, both S3 keypairs and
|
||||||
- `/etc/thermograph/age.key` (`0400`) on each VPS that renders at deploy time.
|
- `/etc/thermograph/age.key` (`0400`) on each VPS that renders at deploy time.
|
||||||
- **Back it up** in the password manager. The public recipient is in `.sops.yaml`.
|
- **Back it up** in the password manager. The public recipient is in `.sops.yaml`.
|
||||||
|
|
||||||
On the **dev desktop** those two are the same file, on purpose. `render-secrets.sh`
|
### The two on-host copies are a deliberate quorum
|
||||||
falls back to `sudo cat` for a root-owned key, the desktop has no passwordless sudo,
|
|
||||||
and the Forgejo runner is a `systemd --user` service with no tty — a `/etc` copy
|
`/etc/thermograph/age.key` on **vps1** and **vps2** is not provisioning residue and
|
||||||
would make every CI-triggered dev deploy hang on a prompt it cannot answer.
|
must not be tidied away. Together they are the recovery quorum for the entire
|
||||||
`deploy-dev.sh` points `THERMOGRAPH_AGE_KEY` at the operator's keyring instead, so
|
estate: five SOPS vaults, and every off-box backup, since `ops-cron.yml:142,197`
|
||||||
the box keeps **one** copy of the recovery root rather than two.
|
pipe those dumps through `age -r` to this same recipient.
|
||||||
|
|
||||||
|
That is not theoretical. On **2026-07-30** the operator's copy was shredded from the
|
||||||
|
desktop *before* it had been copied to its replacement machine, leaving zero
|
||||||
|
operator-side copies. These two were the only surviving material, and the key was
|
||||||
|
restored from vps2. The ordering rule below exists because of that hour.
|
||||||
|
|
||||||
|
- **Never remove the last operator-side copy** until its replacement has been
|
||||||
|
verified against all five vaults (`sops -d … | grep -c '='` → 16 / 12 / 8 / 16 / 9).
|
||||||
|
Deleting first and verifying second is unrecoverable if the copy is bad.
|
||||||
|
- **Both hosts must hold byte-identical copies.** Divergence is as bad as absence:
|
||||||
|
two different keys means one host renders secrets the other cannot read, and a
|
||||||
|
restore silently picks the wrong one.
|
||||||
|
- **Verify it, don't assume it** — `deploy/secrets/verify-age-quorum.sh` asserts
|
||||||
|
presence, permissions and hash equality across both hosts. It prints SHA-256
|
||||||
|
digests and modes only, never the key, so it is safe in a CI log and suitable as
|
||||||
|
a cron gate. Deletion and divergence are both silent failures; nothing else in the
|
||||||
|
estate would notice either until a restore.
|
||||||
|
|
||||||
|
Verified 2026-07-30: vps1 `0440 root:deploy`, vps2 `0400 root:root`, digests equal.
|
||||||
|
Note the asymmetry — on vps1 the group `deploy` can read the key that decrypts
|
||||||
|
`common.yaml` and `prod.yaml`, on the box that also runs Forgejo, its CI runner and
|
||||||
|
dev's unreviewed branches. The dev render does not need that group bit: it runs as
|
||||||
|
`agent`, which is not in group `deploy` and reaches the key through its passwordless
|
||||||
|
`sudo` instead (`/opt/thermograph-dev` is `agent:agent`). Tightening vps1 to `0400
|
||||||
|
root:root` is therefore expected to be safe and is the obvious follow-up; it is
|
||||||
|
called out rather than done here because it changes a live host on the deploy path.
|
||||||
|
|
||||||
|
This weakens — in practice, not in intent — the rule in `infra/CLAUDE.md` that "vps1
|
||||||
|
must never hold prod credentials". Withholding `common.yaml` from dev's *render* does
|
||||||
|
not withhold the key that decrypts it from the *box*, because every vault is
|
||||||
|
encrypted to a single recipient. Fixing that properly means a second age identity for
|
||||||
|
dev, not a policy line.
|
||||||
|
|
||||||
## Prerequisites (once per machine)
|
## Prerequisites (once per machine)
|
||||||
|
|
||||||
|
|
@ -246,8 +305,10 @@ go install github.com/getsops/sops/v3/cmd/sops@latest # if you have Go
|
||||||
```sh
|
```sh
|
||||||
sops deploy/secrets/common.yaml # opens DECRYPTED in $EDITOR; re-encrypts on save
|
sops deploy/secrets/common.yaml # opens DECRYPTED in $EDITOR; re-encrypts on save
|
||||||
git commit -am "rotate <thing>" && git push forgejo
|
git commit -am "rotate <thing>" && git push forgejo
|
||||||
# beta auto-deploys on push to main. prod (no CI) — one command:
|
# beta and prod both auto-deploy on push (main -> beta, release -> prod) via
|
||||||
ssh agent@169.58.46.181 'cd /opt/thermograph && git pull && deploy/deploy.sh'
|
# the Deploy workflow. To force it by hand on vps2, say which environment:
|
||||||
|
ssh agent@169.58.46.181 'cd /opt/thermograph && git pull && THERMOGRAPH_ENV=prod deploy/deploy.sh'
|
||||||
|
ssh agent@169.58.46.181 'cd /opt/thermograph-beta && git pull && THERMOGRAPH_ENV=beta deploy/deploy.sh'
|
||||||
```
|
```
|
||||||
|
|
||||||
A host-specific value (e.g. `POSTGRES_PASSWORD`) is the same, editing `prod.yaml` /
|
A host-specific value (e.g. `POSTGRES_PASSWORD`) is the same, editing `prod.yaml` /
|
||||||
|
|
@ -264,10 +325,13 @@ commit, and the next `deploy-dev.sh` run (merge to `dev`, or by hand) renders it
|
||||||
|
|
||||||
## Setting up a new dev machine
|
## Setting up a new dev machine
|
||||||
|
|
||||||
The desktop is hands-on: Centralis cannot SSH to it, so this is typed, not
|
Dev is provisioned like any fleet host now (`deploy/provision-dev.sh`, run as
|
||||||
automated. `provision-secrets.sh` is **not** the right tool here — it installs the
|
root on vps1), not the sudo-free desktop bootstrap this predates. The
|
||||||
age key root-owned at `/etc/thermograph/age.key`, which the runner cannot read
|
secrets-specific steps below are still worth doing by hand rather than
|
||||||
(above).
|
folding into that script: `provision-secrets.sh` is **not** the right tool
|
||||||
|
here — it installs the age key root-owned at `/etc/thermograph/age.key`,
|
||||||
|
which the CI-runner-driven render may not be able to read (see the age-key
|
||||||
|
note above).
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# 1. sops + age on PATH, and the age private key in the operator's keyring:
|
# 1. sops + age on PATH, and the age private key in the operator's keyring:
|
||||||
|
|
@ -293,17 +357,17 @@ SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt \
|
||||||
|
|
||||||
# 5. Deploy. deploy-dev.sh refuses to run if render-secrets.sh in the checkout
|
# 5. Deploy. deploy-dev.sh refuses to run if render-secrets.sh in the checkout
|
||||||
# cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, rather than leak.
|
# cannot honour THERMOGRAPH_SECRETS_SKIP_COMMON, rather than leak.
|
||||||
APP_DIR=~/thermograph-dev bash deploy/deploy-dev.sh
|
APP_DIR=/opt/thermograph-dev bash deploy/deploy-dev.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
To take a dev box **back** off the vault, delete `/etc/thermograph/secrets-env`:
|
To take a dev box **back** off the vault, delete `/etc/thermograph/secrets-env`:
|
||||||
presence detection turns rendering off and `deploy-dev.sh` falls back to its
|
presence detection turns rendering off and `deploy-dev.sh` falls back to its
|
||||||
built-in `POSTGRES_PASSWORD` default.
|
built-in `POSTGRES_PASSWORD` default.
|
||||||
|
|
||||||
A *different* dev machine (a second desktop, a laptop) needs its own values, not a
|
A *different* dev machine (a laptop running `make dev-up` locally, or a future
|
||||||
copy of this one's: give it its own `<env>.yaml` and its own marker name, so the two
|
second dev host) needs its own values, not a copy of vps1's: give it its own
|
||||||
can never be confused for one host — the same argument as the three held-back
|
`<env>.yaml` and its own marker name, so the two can never be confused for one
|
||||||
credentials above.
|
host — the same argument as the three held-back credentials above.
|
||||||
|
|
||||||
## Add a new secret
|
## Add a new secret
|
||||||
|
|
||||||
|
|
@ -332,17 +396,24 @@ git commit -am "rotate age identity" && push + deploy
|
||||||
|
|
||||||
## First-time bootstrap (seed from the live boxes)
|
## First-time bootstrap (seed from the live boxes)
|
||||||
|
|
||||||
|
**Historical** — this ran once, before the vps1/vps2 split, when beta was its
|
||||||
|
own box at `75.119.132.91`. That address is **vps1** now (dev + Forgejo +
|
||||||
|
Grafana), not beta; re-running the beta line below as written would seed
|
||||||
|
`beta.yaml` from the wrong host. Kept for the shape of the process, which is
|
||||||
|
unchanged if a new environment is ever bootstrapped the same way — just point
|
||||||
|
`seed-from-live.sh` at wherever that environment actually lives today.
|
||||||
|
|
||||||
Order matters — values must match the live env exactly so `AUTH_SECRET` / VAPID /
|
Order matters — values must match the live env exactly so `AUTH_SECRET` / VAPID /
|
||||||
`POSTGRES_PASSWORD` don't rotate unintentionally:
|
`POSTGRES_PASSWORD` don't rotate unintentionally:
|
||||||
|
|
||||||
1. Install `sops`+`age` and drop `/etc/thermograph/age.key` (0400) +
|
1. Install `sops`+`age` and drop `/etc/thermograph/age.key` (0400) +
|
||||||
`/etc/thermograph/secrets-env` on prod & beta (`provision-secrets.sh`).
|
`/etc/thermograph/secrets-env` on the target hosts (`provision-secrets.sh`).
|
||||||
2. Seed each host's file from its live env with the helper (pulls over SSH, encrypts,
|
2. Seed each host's file from its live env with the helper (pulls over SSH, encrypts,
|
||||||
and verifies the render round-trips — run it on your own machine, it reads live
|
and verifies the render round-trips — run it on your own machine, it reads live
|
||||||
secrets):
|
secrets), e.g. as originally run:
|
||||||
```sh
|
```sh
|
||||||
deploy/secrets/seed-from-live.sh prod agent@169.58.46.181
|
deploy/secrets/seed-from-live.sh prod agent@169.58.46.181
|
||||||
deploy/secrets/seed-from-live.sh beta agent@75.119.132.91
|
deploy/secrets/seed-from-live.sh beta agent@75.119.132.91 # beta's address AT THE TIME; now vps1's
|
||||||
```
|
```
|
||||||
That writes exact per-host copies (`prod.yaml`/`beta.yaml`); factor shared values
|
That writes exact per-host copies (`prod.yaml`/`beta.yaml`); factor shared values
|
||||||
into `common.yaml` later. Commit.
|
into `common.yaml` later. Commit.
|
||||||
|
|
|
||||||
|
|
@ -1,23 +1,23 @@
|
||||||
APP_CPUS: ENC[AES256_GCM,data:Wg==,iv:9kB0WruqLCIkJp7i+t5kGB4qv8NzHwgTQlFmYwhTEXc=,tag:q6cKDI86YrFJjGRnEkdweA==,type:str]
|
APP_CPUS: ENC[AES256_GCM,data:qg==,iv:T3eyF+9ssUCWDiFr8Wg0tRpgeV3BSLIjOSBE2i7WOlc=,tag:wisw0gQ6E8QCNom/Zd3mWQ==,type:str]
|
||||||
DB_CPUS: ENC[AES256_GCM,data:7w==,iv:xDOeZo3dJ5gnXIEGyOBGZm4RUTNyzinMkvxWoY/EbQs=,tag:ikmAjzeM/U4w5HDZ+C7mgA==,type:str]
|
POSTGRES_PASSWORD: ENC[AES256_GCM,data:r5ygtwgycLjjGXSehpLt/RpUkhLyZ4lHPt6KiqNd+ng=,iv:qgqfU1mtm7ZuqSbeUEB9cgT56bOAkc0UFb3BMxED+xg=,tag:WiEoVHrA96WNFROqmFyElA==,type:str]
|
||||||
DB_MEMORY: ENC[AES256_GCM,data:apg=,iv:4+GHbccVBFUGtrP12a2oEya7Hz0KUhJAFzdwpqQJ2sc=,tag:X0h2nYza80c9PjeNgqX7Uw==,type:str]
|
THERMOGRAPH_AUTH_SECRET: ENC[AES256_GCM,data:aZySO3pAVB24S7uCsblRtd65U+OkBatK2tmuzeALdlPsqRhMqRioXQLgMQ==,iv:5zxNwj39E/AOJXzV/AEX9w1n3LSnl9PnS3cp5wVRi2E=,tag:jQRGr2436E4qqBE53iltqQ==,type:str]
|
||||||
POSTGRES_PASSWORD: ENC[AES256_GCM,data:X5HmOvHVtm2mjeLZO7FaEWP0Qq767D0quycr2iI3/Mw=,iv:BQsjJaDPW9RxnMldE1rCU3yP5WBOXvc9VjnTim3kMZs=,tag:yfdi+T7lBbBK544bSvEZtQ==,type:str]
|
THERMOGRAPH_BASE_URL: ENC[AES256_GCM,data:9IxftZqwSBG/1MY+jcALQeeTbIk+tJ/8GXGUgQ==,iv:EvsDUExWQHpidriMDDj0flM8Rz25iFycVPJLrK240+s=,tag:7rKr5KUDMe4J7U4OHUvPRg==,type:str]
|
||||||
THERMOGRAPH_AUTH_SECRET: ENC[AES256_GCM,data:YjgknnUhAU0gW6YuMCmlZYfVhOgye52MLhVF7nKMuc0ToGn0OTqqoiXN0A==,iv:Q9XCoKa5Y/7V9b72tktGZ12jgPumC4kJKjrtCM39ihw=,tag:3kEpMjLCEFQcpSir3N9f6w==,type:str]
|
THERMOGRAPH_DATABASE_URL: ENC[AES256_GCM,data:SnD/achZHduuWwwrQYOseqw0mquV1B4k4c2eom81dqtib6SxGj1OabHYj0rJd+o9Y5sH+kTh+zkFHpBi0J7c5GEGzhBqLctt530uMlOstvu1Hh7qL+DareyBF+SHwG4=,iv:qPFUnfRpQ6gIfBx6phyl5RWsNgEFYdNqW8oVzDZgXU8=,tag:djaqFRpDetNoH7Ux1PmQ3Q==,type:str]
|
||||||
THERMOGRAPH_BASE_URL: ENC[AES256_GCM,data:bus0/ibfLohpsszHZKKoGe5P0/Su5i36Hk2nzg==,iv:QI3crdbipse62xdakCGxPIA4wmdq/T4KzZSvi/NE1Xg=,tag:/S4Bzk2aDoA6ahDlW7qG8w==,type:str]
|
WORKERS: ENC[AES256_GCM,data:gQ==,iv:jkwIzTpiEtEtbt7acP8DWRbl51baTFuQiw4oJxb3eyA=,tag:X2Gqy3icYRKrI6F99lnuyQ==,type:str]
|
||||||
THERMOGRAPH_DATABASE_URL: ENC[AES256_GCM,data:xMd8m6VajAjtwQy8DDqTs4VbyASG8ua8FUZHfPmmEzZEb4cTtYILJ+z6LXhcyvdenDsPg2/GJx5BApvVAKSrWrnSzNWnp+pAs7x+VnNzZ2giZ9meiQ==,iv:MOU+SB/taEc/ExRc07ChF1ksOQiaDj7TPMQq+aV4Ltg=,tag:XP2h9KBGJE7JzripO7yjcg==,type:str]
|
THERMOGRAPH_MAIL_BACKEND: ENC[AES256_GCM,data:OrchUc9tpw==,iv:SmAZ+USGesaCyaOdKopqKmPeG7y/WtDry9G3aDVgP/Q=,tag:EGh5foV6uClrIJCdkjP3Hg==,type:str]
|
||||||
WORKERS: ENC[AES256_GCM,data:BQ==,iv:XYgQ+Sn9OnbUR8LVW+4r9vhnQQJZmliVomfyRmUqMsw=,tag:JzM3JsNR08z1pePSTl/dhw==,type:str]
|
THERMOGRAPH_DISCORD_BOT: ENC[AES256_GCM,data:Yw==,iv:jsHid5lQ75shgfIJkrsLnlhmcJ1aJWuxkUyhmpg52Dc=,tag:UqtZjQpYwQH5uvEYC9mp7w==,type:str]
|
||||||
sops:
|
sops:
|
||||||
age:
|
age:
|
||||||
- enc: |
|
- enc: |
|
||||||
-----BEGIN AGE ENCRYPTED FILE-----
|
-----BEGIN AGE ENCRYPTED FILE-----
|
||||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFWHVZU2xSS1pmcy91S2w5
|
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqdFZ3MWVCSDdvdnVLWlZH
|
||||||
MWNhQW90TDdQdDdJV1BHM2Y0a1d6aFJOS2hnCjRodHZlY0orODdpS3QwTVdTdXM5
|
My9JMzF2TmhiSlJjbWRYbUIzSm0zb1dtdTI4CmI4N2o0YzR1NGNoTmI5OXBJM1Bj
|
||||||
Z3RJOFhyR3AzbjljaExkQjN0YUJ2ckUKLS0tIGpua1Z2TDJrRzE3WFZlMWNpL1Ja
|
OVF3WWEvYmlWczR2dUJpYTROR2pvK2sKLS0tIDM5blFFdnZwRHpDOEZ6aHhyL0dn
|
||||||
VG45eVM0MldLejRNd0pweWNna3JYWlUKWuNU+6PqKlbr7F0ckrNxsMF2OyXh1fMu
|
d1BDWVV6YXNQNWo3dWRqN2FVUC85UmMK8jjGvQxKDsnlr7i95Ar509nihDgRb/JW
|
||||||
cLFBIQg/7vO7O7PJ0VIy0Ugfq6gj2Gv91qKJUGeOXOw3tv1Y6HHbjA==
|
qzLM5fwMU16HOPMiAPd/zH3rUyKcZxMtITR6QLcuSNLByURg4ZMBnQ==
|
||||||
-----END AGE ENCRYPTED FILE-----
|
-----END AGE ENCRYPTED FILE-----
|
||||||
recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
|
recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
|
||||||
lastmodified: "2026-07-25T00:38:24Z"
|
lastmodified: "2026-07-25T21:43:52Z"
|
||||||
mac: ENC[AES256_GCM,data:lTpEmA8gweS+QlAakIziXxAEiHBUsNDk5lGwVG+uaXNhifIYWbu+QWkklP4wES9o3pu4TNCzFboRWkBbnzhW8HYRglkSChsmmFXDChz67IYlaveCFwczxffp39iypnV/EKaBCa8m1eDJ5kugzr2s7+blagCH0TwpNXBqZShyGNE=,iv:KBF6CreB4oTJGwNetTj27Grox30X3s3qEEYb1cJXmIE=,tag:YfRb5yAoY8K/TXspoZYdcA==,type:str]
|
mac: ENC[AES256_GCM,data:/+kmyMP20n7fQRO2FdiHWaLl8MAfnMjtF8OE5HEUCLpRGet3s+IqYb8DajFIF9SCibW7aEqb6ir9m0KBx0IRW4TTsvK6JYFyRZKKtt4oiCzRaGwUMwhI75DE6xYdHxEbdAC6Q+qFUKGtRcIGhSX1LrlGEloSJggdEFSgfTlJLNg=,iv:UX1zMFw70ODiPolKSjEj8adKxa49Kf5dzfDA7ePtxmE=,tag:dA0OqIPTvXBlMPNH4oCNtg==,type:str]
|
||||||
unencrypted_suffix: _unencrypted
|
unencrypted_suffix: _unencrypted
|
||||||
version: 3.13.2
|
version: 3.13.2
|
||||||
|
|
|
||||||
|
|
@ -4,13 +4,13 @@ THERMOGRAPH_BASE: ENC[AES256_GCM,data:sQ==,iv:Pop6S2qU0o2+2xMKWQD2P3Vjdh4rUafWF7
|
||||||
THERMOGRAPH_COOKIE_SECURE: ENC[AES256_GCM,data:CA==,iv:TM6XiygrDQn2qps4lO6hY2ldjW8LWlO3D/R/DBPqvGQ=,tag:+thBT7fMzH+DigecgxF4CQ==,type:str]
|
THERMOGRAPH_COOKIE_SECURE: ENC[AES256_GCM,data:CA==,iv:TM6XiygrDQn2qps4lO6hY2ldjW8LWlO3D/R/DBPqvGQ=,tag:+thBT7fMzH+DigecgxF4CQ==,type:str]
|
||||||
THERMOGRAPH_INDEXNOW_KEY: ENC[AES256_GCM,data:XZEMMJfmrSjc/sKEePaigTqEkh3ob/E25R25DvOAWwY=,iv:fqdaegzG1Na0zW+UgOh92RS4FP22pxStdD1/Jq5DRjs=,tag:/8ORtVkGUWjqgGXG/xZHmg==,type:str]
|
THERMOGRAPH_INDEXNOW_KEY: ENC[AES256_GCM,data:XZEMMJfmrSjc/sKEePaigTqEkh3ob/E25R25DvOAWwY=,iv:fqdaegzG1Na0zW+UgOh92RS4FP22pxStdD1/Jq5DRjs=,tag:/8ORtVkGUWjqgGXG/xZHmg==,type:str]
|
||||||
THERMOGRAPH_LAKE_S3_ACCESS_KEY: ENC[AES256_GCM,data:5pL+8tZbs8TZ5ezJpkPUmMss0gLcte73lU+au/476Ws=,iv:ujU3G+rbPZQx/igg50GeIPQxqgd+szIsjc3AqGhGLBU=,tag:hrkgn2DPkmBGThlsdMplqg==,type:str]
|
THERMOGRAPH_LAKE_S3_ACCESS_KEY: ENC[AES256_GCM,data:5pL+8tZbs8TZ5ezJpkPUmMss0gLcte73lU+au/476Ws=,iv:ujU3G+rbPZQx/igg50GeIPQxqgd+szIsjc3AqGhGLBU=,tag:hrkgn2DPkmBGThlsdMplqg==,type:str]
|
||||||
THERMOGRAPH_LAKE_S3_SECRET_KEY: ENC[AES256_GCM,data:XG+6wizXMVVFixhqLhg72HoKw9kEmH+Tty8jYMycvVk=,iv:tNz/WrkMYip2QyAr6bRy6PaIee56ZZ64BELBgvidVT0=,tag:HiQ55DGbeDAOSF0vpNWpWw==,type:str]
|
THERMOGRAPH_LAKE_S3_SECRET_KEY: ENC[AES256_GCM,data:Ui5dcmhzz84nTp1gYhJ89VOJ2eOBFyLjf0WkZ7Y3sI0=,iv:WI0QqbUW7BHo7Zz0082zxpYNIYpVIhq1NeBGbQeDc/I=,tag:urNsE8DPaldJcw9aqX1PYA==,type:str]
|
||||||
THERMOGRAPH_METRICS_TOKEN: ENC[AES256_GCM,data:DmOZU3HWQTuoSvQMb7iPkClJbDH48NB3OcRIYRkqNz8=,iv:acgGh0w2mc5ZD7rB7m/GW3Awsx3RWK+02L6YkAv5Rw0=,tag:x3U9F0Dekw/r9euRMjVQAA==,type:str]
|
THERMOGRAPH_METRICS_TOKEN: ENC[AES256_GCM,data:DmOZU3HWQTuoSvQMb7iPkClJbDH48NB3OcRIYRkqNz8=,iv:acgGh0w2mc5ZD7rB7m/GW3Awsx3RWK+02L6YkAv5Rw0=,tag:x3U9F0Dekw/r9euRMjVQAA==,type:str]
|
||||||
THERMOGRAPH_S3_ACCESS_KEY: ENC[AES256_GCM,data:hTSSUIfOlY0GYI8gZ7FcOi7c89iye0Ym73l4M22zkNk=,iv:aPdbXKb4pOqigo9J3a7oM3qTip6HQcHUQDlAwg6XfoE=,tag:C4p+Ob/iSElNdF4m9E+M7A==,type:str]
|
THERMOGRAPH_S3_ACCESS_KEY: ENC[AES256_GCM,data:hTSSUIfOlY0GYI8gZ7FcOi7c89iye0Ym73l4M22zkNk=,iv:aPdbXKb4pOqigo9J3a7oM3qTip6HQcHUQDlAwg6XfoE=,tag:C4p+Ob/iSElNdF4m9E+M7A==,type:str]
|
||||||
THERMOGRAPH_S3_BUCKET: ENC[AES256_GCM,data:vGJDaElN6JRUawlD55Vr5Q==,iv:4ZAQ5NXb9xuMN+qPspM17W6zm9NIDXLX9hTJ/etSL0E=,tag:Ss7xaUl+kOiVF7oDs9FiYw==,type:str]
|
THERMOGRAPH_S3_BUCKET: ENC[AES256_GCM,data:vGJDaElN6JRUawlD55Vr5Q==,iv:4ZAQ5NXb9xuMN+qPspM17W6zm9NIDXLX9hTJ/etSL0E=,tag:Ss7xaUl+kOiVF7oDs9FiYw==,type:str]
|
||||||
THERMOGRAPH_S3_ENDPOINT: ENC[AES256_GCM,data:PtZg52Uu70Ndx7L/dRTCZTmqv80NXiohOg+gfrzG,iv:hKQTy3Twd5uWwnS4UIs9oqT4NzpYUWQxWO1lUqrzpAc=,tag:kYwYp58fxkzRe5yQPSXa3g==,type:str]
|
THERMOGRAPH_S3_ENDPOINT: ENC[AES256_GCM,data:PtZg52Uu70Ndx7L/dRTCZTmqv80NXiohOg+gfrzG,iv:hKQTy3Twd5uWwnS4UIs9oqT4NzpYUWQxWO1lUqrzpAc=,tag:kYwYp58fxkzRe5yQPSXa3g==,type:str]
|
||||||
THERMOGRAPH_S3_SECRET_KEY: ENC[AES256_GCM,data:1xuZRaC4p+ZM/oFAUuGDZb6AAu2PbjFrHQY3zmczRpI=,iv:xzI24qlUOvYiThBLOue0odvopuxdHCwDlwI4JwiP9EU=,tag:yRS9GjGLBEYk/kZ6miFudg==,type:str]
|
THERMOGRAPH_S3_SECRET_KEY: ENC[AES256_GCM,data:4u5d3RHHBeIDfRIwQVDldWUhH7PyTcNP9s3xTrL4N74=,iv:RXWu+4ZdTmWx3PGw6nWhN2587byy4wqjY74Ldu4xvuk=,tag:dwS6htP2Dv2OQyNzmjZZIA==,type:str]
|
||||||
THERMOGRAPH_VAPID_CONTACT: ENC[AES256_GCM,data:N9mPLx6Mcgqm5TlgqIMFhNuZsBZxVPXf3LplO9k+VA==,iv:y5+MLI0zC5Kb6pRdORTMVJ1iMMoFrVRfv99mKWMRjp8=,tag:7VvKCLU+1TJtQcHWlDwybg==,type:str]
|
THERMOGRAPH_VAPID_CONTACT: ENC[AES256_GCM,data:hg5HqBua81dG3d4KaxLxljRkeGnt4h4=,iv:q+4tQGrlsQaRGUTJuqByqss+twR6v6NQrI/qSPjCbsQ=,tag:2D1+hv/KSDJu+4+J9S0hww==,type:str]
|
||||||
THERMOGRAPH_VAPID_PRIVATE_KEY: ENC[AES256_GCM,data:hTxKlgPWeW4HGBf08SxdAdK+ce6T5Fl9f+5tSGV6WpS+za5EI3zsK8BgRw==,iv:l+omFaCyL2+PHWdchadkmED2gIAoj5T0u/Ltzaw8mok=,tag:AQ3wT/wD5JAouX1M+Tjjmw==,type:str]
|
THERMOGRAPH_VAPID_PRIVATE_KEY: ENC[AES256_GCM,data:hTxKlgPWeW4HGBf08SxdAdK+ce6T5Fl9f+5tSGV6WpS+za5EI3zsK8BgRw==,iv:l+omFaCyL2+PHWdchadkmED2gIAoj5T0u/Ltzaw8mok=,tag:AQ3wT/wD5JAouX1M+Tjjmw==,type:str]
|
||||||
THERMOGRAPH_VAPID_PUBLIC_KEY: ENC[AES256_GCM,data:bgtqZFfTzYz5llh/YhAyGwGtRedK3/ze8SAWj8YdldY6s4nt2F3XeTANVO2Y9+NzvXc16PB3WWPnYEzCxnrG2KciIeYpOYZMoPpGldENPBLJtLa76Ej9,iv:HuZ7XtlXIt4wyJ79k3IYjHfWrQp7lnak4uCspyzS4BE=,tag:uPZvba8LVugDy76pKat6TA==,type:str]
|
THERMOGRAPH_VAPID_PUBLIC_KEY: ENC[AES256_GCM,data:bgtqZFfTzYz5llh/YhAyGwGtRedK3/ze8SAWj8YdldY6s4nt2F3XeTANVO2Y9+NzvXc16PB3WWPnYEzCxnrG2KciIeYpOYZMoPpGldENPBLJtLa76Ej9,iv:HuZ7XtlXIt4wyJ79k3IYjHfWrQp7lnak4uCspyzS4BE=,tag:uPZvba8LVugDy76pKat6TA==,type:str]
|
||||||
TIMESCALEDB_TAG: ENC[AES256_GCM,data:3mDU/k5fx0894+E=,iv:/iw3BZPF3mZ9M3bFNJSTzP9t15YZWCJwzoVXYL9Vj9o=,tag:Y4ez3+XfLJN1PtJyZlJj+g==,type:str]
|
TIMESCALEDB_TAG: ENC[AES256_GCM,data:3mDU/k5fx0894+E=,iv:/iw3BZPF3mZ9M3bFNJSTzP9t15YZWCJwzoVXYL9Vj9o=,tag:Y4ez3+XfLJN1PtJyZlJj+g==,type:str]
|
||||||
|
|
@ -25,7 +25,7 @@ sops:
|
||||||
bncE6yn10QK5kVcF9dDvpQ6RbaG+ESpNZTnuhSL5PDqrKtjnsV0Iqw==
|
bncE6yn10QK5kVcF9dDvpQ6RbaG+ESpNZTnuhSL5PDqrKtjnsV0Iqw==
|
||||||
-----END AGE ENCRYPTED FILE-----
|
-----END AGE ENCRYPTED FILE-----
|
||||||
recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
|
recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
|
||||||
lastmodified: "2026-07-25T00:38:24Z"
|
lastmodified: "2026-07-31T04:23:12Z"
|
||||||
mac: ENC[AES256_GCM,data:C12PnFE7MO0Ge5dCAHCcyXh650hFtRuexl5d3DL0IqTf8nLyNtRJzrPyl9whIs+S3HulsXh3xJOwwPdg6aTqdCfGSQ4Kp5qV+OHxH9lfpHs2yH+exa/eX+7Khpjk0OfvX5beC0GSPpYf9c01buaUTWcIh3pJXpDMdqr+aLC+Zd0=,iv:6VeaaEEkWU5rdTHy75rc2emb2u/HmOmLzO11D38ZXHc=,tag:iGRevzdtqZF9kyH0wxqG3w==,type:str]
|
mac: ENC[AES256_GCM,data:KkItyA0iw2j4w9pf/efFcpyOP8RmO8XzfMoQhJcRXLZbrUC3/xltbxkfTIe2hRgAqKyz5d7EuAHBPwwSCubue0O2gzlG0nCnRQH7YyO4cUrP24XYb+aI+tBM2jdZIuZrGDNcq04cwKiecC6MyOOp8uQPZ41AOPH3sONGWj1VH2s=,iv:+j5uv4QbuWpXAyElvbYGKWhe9vmGTfs3Hpy5I5FetJo=,tag:wcfPuugefeQAyxdb2HV1Fg==,type:str]
|
||||||
unencrypted_suffix: _unencrypted
|
unencrypted_suffix: _unencrypted
|
||||||
version: 3.13.2
|
version: 3.13.2
|
||||||
|
|
|
||||||
|
|
@ -1,27 +1,27 @@
|
||||||
PORT: ENC[AES256_GCM,data:0LjaWw==,iv:3e8Q71QFlMAhQStgoiVh/c2ELI3+7UkWseAFv8BnElU=,tag:/wYfKmqErJpsLBjg1gs4Tw==,type:str]
|
PORT: ENC[AES256_GCM,data:VCheDw==,iv:u9IC7psxzcsLuuWhU89fUJfVsmCI7lECbjxzKmC/nGM=,tag:ZTG/kG2vSNvUUlItAjsCHg==,type:str]
|
||||||
THERMOGRAPH_BASE: ENC[AES256_GCM,data:Ww==,iv:e2yNIAAw4dtlahfbA+PCM/jyODKFwWFNfVpVJ1JoREc=,tag:QzoGwDgIwdA9OFAMIPej7Q==,type:str]
|
THERMOGRAPH_BASE: ENC[AES256_GCM,data:Aw==,iv:Pcmbm6D126jncQnmBquZ/t6qMaaUF7UDnQV6q85CsrY=,tag:SGakFybX3TNKAWvFo9gsEw==,type:str]
|
||||||
TIMESCALEDB_TAG: ENC[AES256_GCM,data:7Oj7OoxoaG51whg=,iv:PPbfdn4DHMxTPA5FUqv+lEY29wQsVX+g0aVuWvraYrU=,tag:NAyJZlRXU1iKo4jwBctUtw==,type:str]
|
TIMESCALEDB_TAG: ENC[AES256_GCM,data:ghewNqRFuYJfBgE=,iv:q1GsQM3QZBW4A95U9NY90RR9WRmPV6miIxGLgLmPTps=,tag:eJARZNyqnJbHhtXXUI1AWw==,type:str]
|
||||||
DB_MEMORY: ENC[AES256_GCM,data:/3U=,iv:jFI/9NQaz8oIqBWAKydWUEEI7bI3SmLnsoBlVgfKw7I=,tag:oBoFANuTgZV0jXC9qR5IHQ==,type:str]
|
DB_MEMORY: ENC[AES256_GCM,data:k2U=,iv:ZRCjRUHbNvVrCdGWqmOoJBtsJLeLVSVm32szpQ23HN4=,tag:KE5jUadHZnKFPis0friNbg==,type:str]
|
||||||
WORKERS: ENC[AES256_GCM,data:VQ==,iv:+d84ByEU22XVQi43NcRn8cHmbW9XudJMj0guidsHxBQ=,tag:XM//7y0VQCwini4CGL7pSg==,type:str]
|
WORKERS: ENC[AES256_GCM,data:yQ==,iv:6VE9mjS7QLuOWazReqyt1DhAhamkvWSm92B2d/ZCZYo=,tag:hDl1/E11xwUyPVhFNhzOuQ==,type:str]
|
||||||
THERMOGRAPH_BASE_URL: ENC[AES256_GCM,data:FK2kOaZG8ESwDi3VMOlo0XvBKvdADQ==,iv:c6GvCrAnAf+ZFh53xg3ZOlf934Ls+PyR/+CXRi933lE=,tag:cYLXvxP5hk7/6gmz/CCP0A==,type:str]
|
THERMOGRAPH_BASE_URL: ENC[AES256_GCM,data:/sOgmM0eCf7dWldK0iZaDWE0W4aN,iv:dpiLeHLanO2zwtMmPMbsqR4aTVaXfT+V2Ux9K8Alh3w=,tag:PDd/L2nt4X750xCKjHnSPw==,type:str]
|
||||||
THERMOGRAPH_COOKIE_SECURE: ENC[AES256_GCM,data:zg==,iv:sVr33CvZoZmn0LcHlP+FVA8esjGy0aKe1A2iPrrrT0I=,tag:3YneNF7m05fCRGfMgYzByA==,type:str]
|
THERMOGRAPH_COOKIE_SECURE: ENC[AES256_GCM,data:Bw==,iv:dsLcLu126A3d7dTqBgQY1SOvJfKOBT/ye4ucIB4w6B4=,tag:/G7Qvlzb6KKVVjqmn3d/Nw==,type:str]
|
||||||
THERMOGRAPH_MAIL_BACKEND: ENC[AES256_GCM,data:pRBtOu6cTw==,iv:88QgyY6rwwp/6Y32r6Us1M+b7YpSLPfKIsCL2j/YuIY=,tag:FrchZZrMcCSeOFRWbePM4Q==,type:str]
|
THERMOGRAPH_MAIL_BACKEND: ENC[AES256_GCM,data:ryWhCkSdxg==,iv:Lpl1CIjTINV8rF8BW5MHZD4Ji802i96zcyqF3xRMLcY=,tag:I0nbb6IP6LM/Y+KaHjBDpg==,type:str]
|
||||||
THERMOGRAPH_DISCORD_BOT: ENC[AES256_GCM,data:NQ==,iv:Zl888EtEoQQd8FdqGb4/N7gGMT4yEymQfWd2fZZCNVA=,tag:Xspr8/taeQwh1qHhfFbLfw==,type:str]
|
THERMOGRAPH_DISCORD_BOT: ENC[AES256_GCM,data:Rw==,iv:o5tr5lQGXFkB6kTJ/dDARBWECDYRdofFlZ7v/HdMe5k=,tag:Lv97fH6u/nIKZlJr5HjbZw==,type:str]
|
||||||
POSTGRES_PASSWORD: ENC[AES256_GCM,data:l076hUutOAKw7+X6wboX,iv:9WVz3SVUikWREVjB6GsJ+EXWO3PvX+g+OaZNGN9f+jU=,tag:OkVuEqEbTSikpCNEySPX5A==,type:str]
|
POSTGRES_PASSWORD: ENC[AES256_GCM,data:IBVel17GKtrW5yzqDjBi,iv:yEPSSYR/Ng9rjSvSl0veNRDPLayF0SvWZ1T8BSY1l50=,tag:hbB4Uo6YluQPgl9XyvgfKw==,type:str]
|
||||||
THERMOGRAPH_DATABASE_URL: ENC[AES256_GCM,data:kXCYmZXewa8JSY5iJd0cHDtsKrxyKEl5tSVwWeQEd8u7XU+/dn4BExT4RDB3vUWVFVoHXYIXedMnDO8h0QfDCxSI13s=,iv:s1bDjdH4E4T+kF126saOrKlx3AnKEkRWF68KEPcCUe0=,tag:nCzD9U/7g28bJmUXkiYh3Q==,type:str]
|
THERMOGRAPH_DATABASE_URL: ENC[AES256_GCM,data:KjsSo+IglfY5MhtEsX3HJpExAtBwzmSu3E+O7Y22B2WuIrTKGDxG81WwvQGABYfKVel6kzzv8uKczq1qEtOwHRdMsRI=,iv:VPJNe1GrJa7BeAA/N/fqyBjBF+ciF8btE4j/TwrV3cA=,tag:tkyzkR1LxY8z28bjb0bJ7w==,type:str]
|
||||||
THERMOGRAPH_AUTH_SECRET: ENC[AES256_GCM,data:EjwKGKh1od5vgvm8LLv1l6TXO7nshUwh5ihXDpETl+3ldtF8L4H3CWszsw==,iv:4mqwEXWpGpLYbIUotHj9Qf7FSD0ATwkxwhfuL7pG+Zc=,tag:0iLU0ee86H0CgmgPCCtatQ==,type:str]
|
THERMOGRAPH_AUTH_SECRET: ENC[AES256_GCM,data:ccEQxcNxMEn8/Gf2knQaVMSoBp7BRJ7zWhr9KmawaSMrwHGU7kADDWCoMw==,iv:nDpMHhP0ubjzz2LIY0QMGFLGP1M10zOIJb6pPgri0PI=,tag:+Z2j5XLdV+LuSK7hWf5r0Q==,type:str]
|
||||||
sops:
|
sops:
|
||||||
age:
|
age:
|
||||||
- enc: |
|
- enc: |
|
||||||
-----BEGIN AGE ENCRYPTED FILE-----
|
-----BEGIN AGE ENCRYPTED FILE-----
|
||||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzV0lrcUpjeU81TmJDRVpa
|
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyb2dCeVgrbGxGSnNpaUE3
|
||||||
elppeU1mMUlpc3lKSTVtU3MxUi83bzN2dVVRCmxZbDJFUkJQVjhCSmljNFpPaWVu
|
RlovN0RrejVmREY0ckNGaExpNWkvSjcyRW1RCjYvc1IrMWwvdmJGU2g5bU52Mmgw
|
||||||
dkNqTkJTd1ZEaHZsZitpTWlJdElCK28KLS0tIEdSTXN6S1Y2eE12U3o2RGxBNXNN
|
SVY1dnFHd0cyNkM1ZWhJaWlIZldzZlEKLS0tIGtLZDRSMS82UjRFRVdZU3J1ckMz
|
||||||
bDBlYkNxMDVTVEFLYVIyVkhIRnNwOGMKv/TMt307XqvzBLCOCA5C4kXFV9iJeVBn
|
T0g1WnRUUGFFRTVyMlF6WnZiSEoweEEKPLwcTNiE4s24J0kComnNJj5jXIBRBzMh
|
||||||
7gyzb5MRbHbDoNAY/5ckU3341uWUXUQ+IrPvVt2snAdXIlWghm+HwA==
|
PVktsQm9m1ojaG0hgCZsFaMOOnE5O0Vi41jgG7WWLSNZRaVggvRbJg==
|
||||||
-----END AGE ENCRYPTED FILE-----
|
-----END AGE ENCRYPTED FILE-----
|
||||||
recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
|
recipient: age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2
|
||||||
lastmodified: "2026-07-25T00:59:09Z"
|
lastmodified: "2026-07-25T21:44:03Z"
|
||||||
mac: ENC[AES256_GCM,data:Ga1lMooQrBhU+YasAbN3THZvjrct21KRbuxOoXf+72K5mrgPmqZyFDdB71B8dOUzmz4cqTPw3+jQx2s2ZrZCEiR3qmb897I8hWbTiFWnURSC7rjytxIe6kLhDBjCtso8ThKUMqNjAglqjkytHXljP7Cg4zCuGLEgczoakEJVB7s=,iv:NFHljXdG619XJMjy+1OuUknu+QaG5nH68fvo/zAf/TE=,tag:MyFf/edujKzypOwloVUGAA==,type:str]
|
mac: ENC[AES256_GCM,data:ok66XdbCyLyU/2UxQ6Fo63S16r6VETT7Qk/KVJ2b4rrdd4hfOMMAvmUfFmBNkxclxBp+UMOlGJ3XpGRZRzcmWR9o1eXpFlussgDbWHU+KLoIVqAVkUsltVM9aDkg+3dl6/vInftWGw24LbfDh1WhYQ9+OwyKxZntj0sNHAShc8E=,iv:eBTeVtkAv6qcldT0x+zyO2yWVrCm0nhRUJjzaAfxLBM=,tag:BtgIWfkLSoZDcbJML9uO7g==,type:str]
|
||||||
unencrypted_suffix: _unencrypted
|
unencrypted_suffix: _unencrypted
|
||||||
version: 3.13.2
|
version: 3.13.2
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Seed deploy/secrets/<env>.yaml by encrypting a box's live /etc/thermograph.env
|
# Seed deploy/secrets/<env>.yaml by encrypting an environment's live env file
|
||||||
# ENTIRELY ON THE BOX. The plaintext never leaves the box; only the already-encrypted
|
# ENTIRELY ON THE BOX. The plaintext never leaves the box; only the already-encrypted
|
||||||
# ciphertext is written back here. Encryption uses the age PUBLIC key (safe to hardcode)
|
# ciphertext is written back here. Encryption uses the age PUBLIC key (safe to hardcode)
|
||||||
# so the machine you run this from needs nothing but SSH access + the agent key — no
|
# so the machine you run this from needs nothing but SSH access + the agent key — no
|
||||||
|
|
@ -7,7 +7,13 @@
|
||||||
#
|
#
|
||||||
# Run from any checkout of this branch that can SSH to the target with the agent key:
|
# Run from any checkout of this branch that can SSH to the target with the agent key:
|
||||||
# deploy/secrets/seed-encrypt-on-host.sh prod agent@169.58.46.181
|
# deploy/secrets/seed-encrypt-on-host.sh prod agent@169.58.46.181
|
||||||
# deploy/secrets/seed-encrypt-on-host.sh beta agent@75.119.132.91
|
# deploy/secrets/seed-encrypt-on-host.sh beta agent@169.58.46.181
|
||||||
|
# deploy/secrets/seed-encrypt-on-host.sh dev agent@75.119.132.91
|
||||||
|
#
|
||||||
|
# prod and beta share a box (vps2); they differ only by which env FILE is read
|
||||||
|
# (/etc/thermograph.env vs /etc/thermograph-beta.env), resolved from
|
||||||
|
# deploy/env-topology.sh. Hardcoding that path meant "seed beta" would encrypt
|
||||||
|
# PROD's live secrets into beta.yaml.
|
||||||
# Then commit + push the resulting deploy/secrets/<env>.yaml. Verify faithfulness with
|
# Then commit + push the resulting deploy/secrets/<env>.yaml. Verify faithfulness with
|
||||||
# the key-gaps skill after. See README.md.
|
# the key-gaps skill after. See README.md.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
@ -17,14 +23,18 @@ ENV_NAME="${1:?usage: seed-encrypt-on-host.sh <env> <ssh-target> [ssh-key]}"
|
||||||
SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}"
|
SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}"
|
||||||
SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}"
|
SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}"
|
||||||
OUT="deploy/secrets/${ENV_NAME}.yaml"
|
OUT="deploy/secrets/${ENV_NAME}.yaml"
|
||||||
|
# shellcheck source=infra/deploy/env-topology.sh
|
||||||
|
. deploy/env-topology.sh
|
||||||
|
thermograph_topology "$ENV_NAME"
|
||||||
|
REMOTE_ENV_FILE="$TG_ENV_FILE"
|
||||||
|
|
||||||
# Public age recipient — NOT a secret (matches .sops.yaml). The private key never
|
# Public age recipient — NOT a secret (matches .sops.yaml). The private key never
|
||||||
# leaves the operator's machine / the hosts' /etc/thermograph/age.key.
|
# leaves the operator's machine / the hosts' /etc/thermograph/age.key.
|
||||||
AGE_PUB="age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2"
|
AGE_PUB="age1xx4dzs0dxlwvkv9sjuqzsphl7lfrxannkfken374yu2qvvcte9sqzktqt2"
|
||||||
SOPS_URL="https://github.com/getsops/sops/releases/download/v3.13.2/sops-v3.13.2.linux.amd64"
|
SOPS_URL="https://github.com/getsops/sops/releases/download/v3.13.2/sops-v3.13.2.linux.amd64"
|
||||||
|
|
||||||
echo "==> Encrypting ${SSH_TARGET}:/etc/thermograph.env on-host -> ${OUT}"
|
echo "==> Encrypting ${SSH_TARGET}:${REMOTE_ENV_FILE} on-host -> ${OUT}"
|
||||||
ssh -i "$SSH_KEY" "$SSH_TARGET" "AGE_PUB='$AGE_PUB' SOPS_URL='$SOPS_URL' bash -s" > "$OUT" <<'REMOTE'
|
ssh -i "$SSH_KEY" "$SSH_TARGET" "AGE_PUB='$AGE_PUB' SOPS_URL='$SOPS_URL' ENV_FILE='$REMOTE_ENV_FILE' bash -s" > "$OUT" <<'REMOTE'
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
exec 3>&1 # real stdout carries ONLY the ciphertext
|
exec 3>&1 # real stdout carries ONLY the ciphertext
|
||||||
{ # setup noise -> stderr, so it can't corrupt the file
|
{ # setup noise -> stderr, so it can't corrupt the file
|
||||||
|
|
@ -34,7 +44,9 @@ exec 3>&1 # real stdout carries ONLY the ciphertext
|
||||||
fi
|
fi
|
||||||
} >&2
|
} >&2
|
||||||
tmp="$(mktemp)"; tmy="$(mktemp)"; trap 'rm -f "$tmp" "$tmy"' EXIT
|
tmp="$(mktemp)"; tmy="$(mktemp)"; trap 'rm -f "$tmp" "$tmy"' EXIT
|
||||||
sudo cat /etc/thermograph.env > "$tmp" # plaintext stays on this box only
|
# $ENV_FILE comes from the ssh command line above (env-topology.sh resolved it
|
||||||
|
# for the named environment) — NOT hardcoded, because vps2 holds two of them.
|
||||||
|
sudo cat "$ENV_FILE" > "$tmp" # plaintext stays on this box only
|
||||||
python3 -c '
|
python3 -c '
|
||||||
import json, sys
|
import json, sys
|
||||||
for line in open(sys.argv[1], encoding="utf-8"):
|
for line in open(sys.argv[1], encoding="utf-8"):
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,19 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Seed (or re-seed) an encrypted secrets file from a box's live /etc/thermograph.env.
|
# Seed (or re-seed) an encrypted secrets file from an environment's live env file.
|
||||||
# Run on the operator's machine — it reads production secrets, so it is deliberately
|
# Run on the operator's machine — it reads production secrets, so it is deliberately
|
||||||
# NOT something the agent runs for you. Requires: sops + age on PATH, the age private
|
# NOT something the agent runs for you. Requires: sops + age on PATH, the age private
|
||||||
# key at ~/.config/sops/age/keys.txt (or SOPS_AGE_KEY_FILE), and SSH access to the box.
|
# key at ~/.config/sops/age/keys.txt (or SOPS_AGE_KEY_FILE), and SSH access to the box.
|
||||||
#
|
#
|
||||||
# deploy/secrets/seed-from-live.sh prod agent@169.58.46.181 ~/.ssh/thermograph_agent_ed25519
|
# deploy/secrets/seed-from-live.sh prod agent@169.58.46.181 ~/.ssh/thermograph_agent_ed25519
|
||||||
# deploy/secrets/seed-from-live.sh beta agent@75.119.132.91 ~/.ssh/thermograph_agent_ed25519
|
# deploy/secrets/seed-from-live.sh beta agent@169.58.46.181 ~/.ssh/thermograph_agent_ed25519
|
||||||
|
# deploy/secrets/seed-from-live.sh dev agent@75.119.132.91 ~/.ssh/thermograph_agent_ed25519
|
||||||
|
#
|
||||||
|
# Note prod and beta share a box (vps2) and therefore differ only by which env
|
||||||
|
# FILE is read: /etc/thermograph.env vs /etc/thermograph-beta.env. That path is
|
||||||
|
# resolved from deploy/env-topology.sh rather than hardcoded — hardcoding it
|
||||||
|
# meant "seed beta" would read PROD's live secrets and write them into
|
||||||
|
# beta.yaml, which is how one environment silently ends up holding another's
|
||||||
|
# credentials.
|
||||||
#
|
#
|
||||||
# Writes deploy/secrets/<env>.yaml ENCRYPTED (exact copy of the live env, so the
|
# Writes deploy/secrets/<env>.yaml ENCRYPTED (exact copy of the live env, so the
|
||||||
# deploy render is faithful), then verifies the render round-trips to the same
|
# deploy render is faithful), then verifies the render round-trips to the same
|
||||||
|
|
@ -21,13 +29,17 @@ ENV_NAME="${1:?usage: seed-from-live.sh <env> <ssh-target> [ssh-key]}"
|
||||||
SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}"
|
SSH_TARGET="${2:?ssh target, e.g. agent@169.58.46.181}"
|
||||||
SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}"
|
SSH_KEY="${3:-$HOME/.ssh/thermograph_agent_ed25519}"
|
||||||
OUT="deploy/secrets/${ENV_NAME}.yaml"
|
OUT="deploy/secrets/${ENV_NAME}.yaml"
|
||||||
|
# shellcheck source=infra/deploy/env-topology.sh
|
||||||
|
. deploy/env-topology.sh
|
||||||
|
thermograph_topology "$ENV_NAME"
|
||||||
|
REMOTE_ENV_FILE="$TG_ENV_FILE"
|
||||||
command -v sops >/dev/null || { echo "!! sops not on PATH" >&2; exit 1; }
|
command -v sops >/dev/null || { echo "!! sops not on PATH" >&2; exit 1; }
|
||||||
|
|
||||||
tmp="$(mktemp)"; trap 'rm -f "$tmp" "$tmp.env"' EXIT
|
tmp="$(mktemp)"; trap 'rm -f "$tmp" "$tmp.env"' EXIT
|
||||||
umask 077
|
umask 077
|
||||||
|
|
||||||
echo "==> Pulling ${SSH_TARGET}:/etc/thermograph.env (sudo)"
|
echo "==> Pulling ${SSH_TARGET}:${REMOTE_ENV_FILE} (sudo)"
|
||||||
ssh -i "$SSH_KEY" "$SSH_TARGET" 'sudo cat /etc/thermograph.env' > "$tmp"
|
ssh -i "$SSH_KEY" "$SSH_TARGET" "sudo cat $REMOTE_ENV_FILE" > "$tmp"
|
||||||
n=$(grep -cE '^[A-Za-z_][A-Za-z0-9_]*=' "$tmp" || true)
|
n=$(grep -cE '^[A-Za-z_][A-Za-z0-9_]*=' "$tmp" || true)
|
||||||
[ "$n" -gt 0 ] || { echo "!! no KEY=VALUE lines read (permission? path?)" >&2; exit 1; }
|
[ "$n" -gt 0 ] || { echo "!! no KEY=VALUE lines read (permission? path?)" >&2; exit 1; }
|
||||||
echo " ${n} vars"
|
echo " ${n} vars"
|
||||||
|
|
@ -47,7 +59,7 @@ for line in open(src, encoding="utf-8"):
|
||||||
continue
|
continue
|
||||||
pairs[k] = v
|
pairs[k] = v
|
||||||
with open(dst, "w", encoding="utf-8") as fh:
|
with open(dst, "w", encoding="utf-8") as fh:
|
||||||
fh.write(f"# SOPS-encrypted — {env} host secrets, seeded from the live /etc/thermograph.env.\n")
|
fh.write(f"# SOPS-encrypted — {env} host secrets, seeded from that environment’s live env file.\n")
|
||||||
for k, v in pairs.items():
|
for k, v in pairs.items():
|
||||||
fh.write(f"{k}: {json.dumps(v)}\n") # JSON-escaped scalar is valid YAML
|
fh.write(f"{k}: {json.dumps(v)}\n") # JSON-escaped scalar is valid YAML
|
||||||
print(f" {len(pairs)} keys")
|
print(f" {len(pairs)} keys")
|
||||||
|
|
|
||||||
117
infra/deploy/secrets/verify-age-quorum.sh
Executable file
117
infra/deploy/secrets/verify-age-quorum.sh
Executable file
|
|
@ -0,0 +1,117 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Assert that the age recovery quorum is intact.
|
||||||
|
#
|
||||||
|
# infra/deploy/secrets/verify-age-quorum.sh
|
||||||
|
#
|
||||||
|
# The age private key is the single recovery root for all five SOPS vaults AND for
|
||||||
|
# every off-box backup (ops-cron.yml:142,197 pipe dumps through `age -r` to the same
|
||||||
|
# recipient). The copies at /etc/thermograph/age.key on vps1 and vps2 are a DELIBERATE
|
||||||
|
# two-host quorum, not provisioning residue — see the "age key" section of
|
||||||
|
# deploy/secrets/README.md. This script is what makes that policy checkable instead of
|
||||||
|
# merely stated.
|
||||||
|
#
|
||||||
|
# It fails if the key is missing on either host, if the two copies have diverged, or
|
||||||
|
# if either is more permissive than 0440. Divergence matters as much as absence: two
|
||||||
|
# hosts holding different keys means one of them renders secrets nobody else can
|
||||||
|
# decrypt, and a restore from backup silently picks the wrong one.
|
||||||
|
#
|
||||||
|
# OUTPUT DISCIPLINE: prints SHA-256 digests and file modes only. A digest of a
|
||||||
|
# 32-byte random key is not reversible, so it is safe in a CI log; the key itself is
|
||||||
|
# never read, echoed or transferred. Nothing here copies the key anywhere.
|
||||||
|
#
|
||||||
|
# Exit status: 0 = quorum intact; 1 = something needs a human. Suitable as a cron gate.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
VPS1="${TG_VPS1_SSH:-vps1}"
|
||||||
|
VPS2="${TG_VPS2_SSH:-vps2}"
|
||||||
|
KEY_PATH="${TG_AGE_KEY_PATH:-/etc/thermograph/age.key}"
|
||||||
|
SSH_OPTS="${TG_SSH_OPTS:--o BatchMode=yes -o ConnectTimeout=10}"
|
||||||
|
|
||||||
|
# Reads the digest and mode of the key on one host. Uses sudo when the key is not
|
||||||
|
# directly readable, mirroring how render-secrets.sh lifts it.
|
||||||
|
probe() {
|
||||||
|
local target="$1" out
|
||||||
|
# The remote script is a QUOTED heredoc and the key path is passed as $1 to
|
||||||
|
# `bash -s`, so nothing expands on this side. Interpolating the path into the
|
||||||
|
# command string would work too, but it is the shape that quietly breaks the day
|
||||||
|
# a path contains a space, and shellcheck is right to flag it (SC2029).
|
||||||
|
# shellcheck disable=SC2086 # SSH_OPTS is a deliberate word-split option list
|
||||||
|
out=$(ssh $SSH_OPTS "$target" bash -s -- "$KEY_PATH" 2>/dev/null <<'REMOTE'
|
||||||
|
key="$1"
|
||||||
|
if [ ! -e "$key" ]; then echo MISSING; exit 0; fi
|
||||||
|
mode=$(stat -c %a "$key")
|
||||||
|
owner=$(stat -c %U:%G "$key")
|
||||||
|
if [ -r "$key" ]; then
|
||||||
|
digest=$(sha256sum "$key" | cut -d' ' -f1)
|
||||||
|
else
|
||||||
|
digest=$(sudo sha256sum "$key" 2>/dev/null | cut -d' ' -f1)
|
||||||
|
fi
|
||||||
|
[ -n "$digest" ] || { echo UNREADABLE; exit 0; }
|
||||||
|
echo "$digest $mode $owner"
|
||||||
|
REMOTE
|
||||||
|
) || { echo UNREACHABLE; return 0; }
|
||||||
|
printf '%s\n' "$out"
|
||||||
|
}
|
||||||
|
|
||||||
|
rc=0
|
||||||
|
declare -A DIGEST
|
||||||
|
|
||||||
|
for pair in "vps1:$VPS1" "vps2:$VPS2"; do
|
||||||
|
name="${pair%%:*}"
|
||||||
|
target="${pair#*:}"
|
||||||
|
result="$(probe "$target")"
|
||||||
|
case "$result" in
|
||||||
|
MISSING)
|
||||||
|
echo "!! ${name}: no key at ${KEY_PATH} — the quorum is DOWN TO ONE COPY" >&2
|
||||||
|
rc=1
|
||||||
|
;;
|
||||||
|
UNREADABLE)
|
||||||
|
echo "!! ${name}: key present but unreadable even via sudo" >&2
|
||||||
|
rc=1
|
||||||
|
;;
|
||||||
|
UNREACHABLE)
|
||||||
|
echo "!! ${name}: host unreachable — quorum NOT verified (this is not a pass)" >&2
|
||||||
|
rc=1
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
digest="${result%% *}"
|
||||||
|
rest="${result#* }"
|
||||||
|
DIGEST["$name"]="$digest"
|
||||||
|
printf ' %-5s %s mode=%s\n' "$name" "${digest:0:16}…" "$rest"
|
||||||
|
mode="${rest%% *}"
|
||||||
|
# 0400 or 0440 only. Anything wider means a non-root account on that box can
|
||||||
|
# read the key that decrypts prod — and vps1 also runs Forgejo and its CI runner.
|
||||||
|
case "$mode" in
|
||||||
|
400|440) ;;
|
||||||
|
*) echo "!! ${name}: mode ${mode} is wider than 0440" >&2; rc=1 ;;
|
||||||
|
esac
|
||||||
|
# Advisory, not a failure: 0440 with a non-root group means that group can read
|
||||||
|
# the key that decrypts prod. On vps1 that is doubly worth knowing, because the
|
||||||
|
# same box runs Forgejo, its CI runner and dev's unreviewed branches. Left
|
||||||
|
# non-fatal because it is the current provisioned state — a check that fails on
|
||||||
|
# day one is a check that gets ignored by day three. See README.md.
|
||||||
|
group="${rest#* }"; group="${group#*:}"
|
||||||
|
if [ "$mode" = 440 ] && [ "$group" != root ]; then
|
||||||
|
printf ' note: group %s can read this key\n' "$group"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -n "${DIGEST[vps1]:-}" ] && [ -n "${DIGEST[vps2]:-}" ]; then
|
||||||
|
if [ "${DIGEST[vps1]}" = "${DIGEST[vps2]}" ]; then
|
||||||
|
echo " quorum: 2 copies, identical"
|
||||||
|
else
|
||||||
|
echo "!! the two copies have DIVERGED — they are different keys" >&2
|
||||||
|
echo "!! do not 'fix' this by overwriting one. Establish which decrypts the" >&2
|
||||||
|
echo "!! vaults (sops -d on any deploy/secrets/*.yaml) before touching either." >&2
|
||||||
|
rc=1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$rc" -eq 0 ]; then
|
||||||
|
echo " OK — recovery quorum intact"
|
||||||
|
else
|
||||||
|
echo "!! recovery quorum degraded; see deploy/secrets/README.md" >&2
|
||||||
|
fi
|
||||||
|
exit "$rc"
|
||||||
|
|
@ -18,8 +18,20 @@
|
||||||
# rehearsal on the same host that cannot touch live data or ports.
|
# rehearsal on the same host that cannot touch live data or ports.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
APP_DIR="${APP_DIR:-/opt/thermograph}"
|
|
||||||
SERVICE="${SERVICE:-all}"
|
SERVICE="${SERVICE:-all}"
|
||||||
|
|
||||||
|
# Two stacks now run on vps2 (prod and beta), so nothing here may assume "the"
|
||||||
|
# stack: the name, the stack file, the loopback ports, the env file, the
|
||||||
|
# database role and the service-name prefix all come from env-topology.sh,
|
||||||
|
# keyed by the environment deploy.sh already resolved and exported.
|
||||||
|
SELF_DIR=$(cd "$(dirname "$0")" && pwd)
|
||||||
|
# shellcheck source=infra/deploy/env-topology.sh
|
||||||
|
. "$SELF_DIR/../env-topology.sh"
|
||||||
|
ENV_NAME=$(thermograph_env_name)
|
||||||
|
[ -n "$ENV_NAME" ] || ENV_NAME=prod
|
||||||
|
thermograph_topology "$ENV_NAME"
|
||||||
|
|
||||||
|
APP_DIR="${APP_DIR:-$TG_APP_DIR}"
|
||||||
cd "$APP_DIR"
|
cd "$APP_DIR"
|
||||||
|
|
||||||
case "$SERVICE" in
|
case "$SERVICE" in
|
||||||
|
|
@ -27,10 +39,26 @@ case "$SERVICE" in
|
||||||
*) echo "!! SERVICE must be backend|frontend|all, got '$SERVICE'" >&2; exit 2 ;;
|
*) echo "!! SERVICE must be backend|frontend|all, got '$SERVICE'" >&2; exit 2 ;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
|
# Which stack-file services this environment's roll targets. Prod's are bare
|
||||||
|
# (web, worker, ...); beta's carry a `beta-` prefix so its short DNS names never
|
||||||
|
# collide with prod's on the shared overlay — see env-topology.sh.
|
||||||
|
SVC_WEB="${TG_SVC_PREFIX}web"
|
||||||
|
SVC_WORKER="${TG_SVC_PREFIX}worker"
|
||||||
|
SVC_FRONTEND="${TG_SVC_PREFIX}frontend"
|
||||||
|
SVC_LAKE="${TG_SVC_PREFIX}lake"
|
||||||
|
SVC_DAEMON="${TG_SVC_PREFIX}daemon"
|
||||||
|
|
||||||
|
STACK_FILE="$APP_DIR/infra/$TG_STACK_FILE"
|
||||||
|
ENV_FILE="$TG_ENV_FILE"
|
||||||
|
STACK_ENV_FILE="$TG_STACK_ENV_FILE"
|
||||||
|
|
||||||
if [ "${STACK_TEST:-0}" = "1" ]; then
|
if [ "${STACK_TEST:-0}" = "1" ]; then
|
||||||
|
# Test mode rehearses PROD's stack only; it is not a second-environment path.
|
||||||
STACK_NAME="thermograph-test"
|
STACK_NAME="thermograph-test"
|
||||||
LB_NAME="thermograph-test-lb"
|
LB_NAME="thermograph-test-lb"
|
||||||
LB_HTTP_PORT=18137; LB_FE_PORT=18080
|
LB_HTTP_PORT=18137; LB_FE_PORT=18080
|
||||||
|
DATA_NETWORK="thermograph-test_internal"
|
||||||
|
DB_SERVICE="thermograph-test_db"
|
||||||
export PGDATA_VOLUME="thermograph-test_pgdata"
|
export PGDATA_VOLUME="thermograph-test_pgdata"
|
||||||
export APPDATA_VOLUME="thermograph-test_appdata"
|
export APPDATA_VOLUME="thermograph-test_appdata"
|
||||||
export APPLOGS_VOLUME="thermograph-test_applogs"
|
export APPLOGS_VOLUME="thermograph-test_applogs"
|
||||||
|
|
@ -38,29 +66,42 @@ if [ "${STACK_TEST:-0}" = "1" ]; then
|
||||||
docker volume create "$APPDATA_VOLUME" >/dev/null
|
docker volume create "$APPDATA_VOLUME" >/dev/null
|
||||||
docker volume create "$APPLOGS_VOLUME" >/dev/null
|
docker volume create "$APPLOGS_VOLUME" >/dev/null
|
||||||
else
|
else
|
||||||
STACK_NAME="${STACK_NAME:-thermograph}"
|
STACK_NAME="${STACK_NAME:-$TG_STACK_NAME}"
|
||||||
LB_NAME="thermograph-lb"
|
LB_NAME="$TG_LB_NAME"
|
||||||
LB_HTTP_PORT=8137; LB_FE_PORT=8080
|
LB_HTTP_PORT="$TG_LB_HTTP_PORT"; LB_FE_PORT="$TG_LB_FE_PORT"
|
||||||
|
# Where the database is. Prod owns it inside its own overlay; beta reaches
|
||||||
|
# the same service across that overlay, which its stack declares external.
|
||||||
|
DATA_NETWORK="$TG_DATA_NETWORK"
|
||||||
|
DB_SERVICE="$TG_DB_SERVICE"
|
||||||
fi
|
fi
|
||||||
export STACK_NAME
|
export STACK_NAME
|
||||||
|
echo "==> Environment: $ENV_NAME (stack $STACK_NAME on $TG_HOST, checkout $APP_DIR)"
|
||||||
|
|
||||||
# --- secrets ------------------------------------------------------------------
|
# --- secrets ------------------------------------------------------------------
|
||||||
# Render (SOPS) + source /etc/thermograph.env exactly like deploy.sh, then
|
# Render (SOPS) + source /etc/thermograph.env exactly like deploy.sh, then
|
||||||
# install the uid-10001-readable copy the tasks' env-entrypoint shim sources.
|
# install the uid-10001-readable copy the tasks' env-entrypoint shim sources.
|
||||||
# 10001 = the app images' `thermograph` user; the file is 0400 to that uid.
|
# 10001 = the app images' `thermograph` user; the file is 0400 to that uid.
|
||||||
|
#
|
||||||
|
# Per-environment paths throughout: on vps2 prod uses /etc/thermograph.env +
|
||||||
|
# /etc/thermograph/stack.env while beta uses /etc/thermograph-beta.env +
|
||||||
|
# /etc/thermograph/beta-stack.env. Sharing either file would give beta prod's
|
||||||
|
# database credentials and vice versa.
|
||||||
if [ -f "$APP_DIR/infra/deploy/render-secrets.sh" ]; then
|
if [ -f "$APP_DIR/infra/deploy/render-secrets.sh" ]; then
|
||||||
# shellcheck source=infra/deploy/render-secrets.sh
|
# shellcheck source=infra/deploy/render-secrets.sh
|
||||||
. "$APP_DIR/infra/deploy/render-secrets.sh"
|
. "$APP_DIR/infra/deploy/render-secrets.sh"
|
||||||
render_thermograph_secrets "$APP_DIR/infra"
|
render_thermograph_secrets "$APP_DIR/infra" "$ENV_NAME" "$ENV_FILE"
|
||||||
fi
|
fi
|
||||||
# /etc/thermograph.env is rendered at deploy time from the SOPS vault — it
|
# $ENV_FILE is rendered at deploy time from the SOPS vault — it cannot exist at
|
||||||
# cannot exist at lint time, so don't ask shellcheck to follow it.
|
# lint time, so don't ask shellcheck to follow it.
|
||||||
set -a
|
set -a
|
||||||
# shellcheck source=/dev/null
|
# shellcheck source=/dev/null
|
||||||
. /etc/thermograph.env 2>/dev/null || true
|
. "$ENV_FILE" 2>/dev/null || true
|
||||||
set +a
|
set +a
|
||||||
sudo install -o 10001 -g 0 -m 0400 /etc/thermograph.env /etc/thermograph/stack.env \
|
sudo install -o 10001 -g 0 -m 0400 "$ENV_FILE" "$STACK_ENV_FILE" \
|
||||||
|| install -o 10001 -g 0 -m 0400 /etc/thermograph.env /etc/thermograph/stack.env
|
|| install -o 10001 -g 0 -m 0400 "$ENV_FILE" "$STACK_ENV_FILE"
|
||||||
|
# The stack file bind-mounts this path into every task; export it so the
|
||||||
|
# `volumes:` interpolation resolves to the right environment's copy.
|
||||||
|
export STACK_ENV_FILE
|
||||||
|
|
||||||
# --- image tags -----------------------------------------------------------------
|
# --- image tags -----------------------------------------------------------------
|
||||||
# Same persisted-tags contract as deploy.sh: incoming env wins, the file
|
# Same persisted-tags contract as deploy.sh: incoming env wins, the file
|
||||||
|
|
@ -90,7 +131,14 @@ FRONTEND_IMAGE="$REGISTRY_HOST/${FRONTEND_IMAGE_PATH:-emi/thermograph/frontend}:
|
||||||
# Hazard #7: the db image under an existing volume must never drift. Resolve
|
# Hazard #7: the db image under an existing volume must never drift. Resolve
|
||||||
# the digest-pinned ref from whatever is running (stack task or compose
|
# the digest-pinned ref from whatever is running (stack task or compose
|
||||||
# container), falling back to the local latest-pg18's digest on first bring-up.
|
# container), falling back to the local latest-pg18's digest on first bring-up.
|
||||||
if [ -z "${TIMESCALEDB_IMAGE:-}" ]; then
|
#
|
||||||
|
# Only the environment that OWNS the database needs this — i.e. the one whose
|
||||||
|
# own stack declares the db service. Beta's stack declares none (it uses prod's
|
||||||
|
# instance), so resolving an image pin there answers a question beta's stack
|
||||||
|
# file never asks.
|
||||||
|
OWNS_DB=0
|
||||||
|
[ "$DB_SERVICE" = "${STACK_NAME}_db" ] && OWNS_DB=1
|
||||||
|
if [ -z "${TIMESCALEDB_IMAGE:-}" ] && [ "$OWNS_DB" = 1 ]; then
|
||||||
cid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${STACK_NAME}_db" | head -1)
|
cid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${STACK_NAME}_db" | head -1)
|
||||||
[ -z "$cid" ] && cid=$(docker ps -q --filter "name=thermograph-db-1" | head -1)
|
[ -z "$cid" ] && cid=$(docker ps -q --filter "name=thermograph-db-1" | head -1)
|
||||||
if [ -n "$cid" ]; then
|
if [ -n "$cid" ]; then
|
||||||
|
|
@ -102,7 +150,11 @@ if [ -z "${TIMESCALEDB_IMAGE:-}" ]; then
|
||||||
[ -n "$TIMESCALEDB_IMAGE" ] || TIMESCALEDB_IMAGE="$img"
|
[ -n "$TIMESCALEDB_IMAGE" ] || TIMESCALEDB_IMAGE="$img"
|
||||||
fi
|
fi
|
||||||
export TIMESCALEDB_IMAGE
|
export TIMESCALEDB_IMAGE
|
||||||
echo "==> Images: web/worker=$BACKEND_IMAGE frontend=$FRONTEND_IMAGE db=$TIMESCALEDB_IMAGE"
|
if [ "$OWNS_DB" = 1 ]; then
|
||||||
|
echo "==> Images: web/worker=$BACKEND_IMAGE frontend=$FRONTEND_IMAGE db=$TIMESCALEDB_IMAGE"
|
||||||
|
else
|
||||||
|
echo "==> Images: web/worker=$BACKEND_IMAGE frontend=$FRONTEND_IMAGE (db: shared $DB_SERVICE)"
|
||||||
|
fi
|
||||||
|
|
||||||
# --- registry ------------------------------------------------------------------
|
# --- registry ------------------------------------------------------------------
|
||||||
if [ -n "${REGISTRY_TOKEN:-}" ]; then
|
if [ -n "${REGISTRY_TOKEN:-}" ]; then
|
||||||
|
|
@ -124,15 +176,22 @@ done
|
||||||
# in the stack). Runs on the stack's overlay so `db` resolves. First-ever
|
# in the stack). Runs on the stack's overlay so `db` resolves. First-ever
|
||||||
# deploy: the network doesn't exist yet — create it exactly as the stack will
|
# deploy: the network doesn't exist yet — create it exactly as the stack will
|
||||||
# (attachable overlay) so the name is adopted, then migrate, then deploy.
|
# (attachable overlay) so the name is adopted, then migrate, then deploy.
|
||||||
NET="${STACK_NAME}_internal"
|
#
|
||||||
# Never pre-create $NET: docker stack deploy must own it (a pre-existing
|
# The migrate task joins the network the DATABASE is on, which for beta is
|
||||||
# unlabeled network makes it fail with "already exists"). On first deploy the
|
# prod's overlay rather than beta's own — and it connects as this environment's
|
||||||
# migrate runs AFTER stack deploy instead (FIRST_DEPLOY_MIGRATE below).
|
# own role to this environment's own database (beta: thermograph_beta on both
|
||||||
|
# counts), so a beta migration can never touch prod's schema.
|
||||||
|
NET="$DATA_NETWORK"
|
||||||
|
MIGRATE_URL="postgresql+asyncpg://${TG_DB_USER}:${POSTGRES_PASSWORD}@db:5432/${TG_DB_NAME}"
|
||||||
|
# Never pre-create $NET when this stack owns it: docker stack deploy must own it
|
||||||
|
# (a pre-existing unlabeled network makes it fail with "already exists"). On
|
||||||
|
# first deploy the migrate runs AFTER stack deploy instead (FIRST_DEPLOY_MIGRATE
|
||||||
|
# below). Beta's data network is prod's and always already exists.
|
||||||
if [ "$SERVICE" = "backend" ] || [ "$SERVICE" = "all" ]; then
|
if [ "$SERVICE" = "backend" ] || [ "$SERVICE" = "all" ]; then
|
||||||
if docker service inspect "${STACK_NAME}_db" >/dev/null 2>&1; then
|
if docker service inspect "$DB_SERVICE" >/dev/null 2>&1; then
|
||||||
echo "==> One-shot migrate ($BACKEND_IMAGE)"
|
echo "==> One-shot migrate ($BACKEND_IMAGE -> ${TG_DB_NAME})"
|
||||||
docker run --rm --network "$NET" \
|
docker run --rm --network "$NET" \
|
||||||
-e THERMOGRAPH_DATABASE_URL="postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph" \
|
-e THERMOGRAPH_DATABASE_URL="$MIGRATE_URL" \
|
||||||
--entrypoint /app/deploy/entrypoint.sh "$BACKEND_IMAGE" migrate
|
--entrypoint /app/deploy/entrypoint.sh "$BACKEND_IMAGE" migrate
|
||||||
else
|
else
|
||||||
echo "==> First deploy: db not up yet; replicas will be rolled after stack deploy runs migrate below"
|
echo "==> First deploy: db not up yet; replicas will be rolled after stack deploy runs migrate below"
|
||||||
|
|
@ -141,38 +200,38 @@ fi
|
||||||
|
|
||||||
# --- deploy ----------------------------------------------------------------------
|
# --- deploy ----------------------------------------------------------------------
|
||||||
FIRST_DEPLOY_MIGRATE=0
|
FIRST_DEPLOY_MIGRATE=0
|
||||||
docker service inspect "${STACK_NAME}_db" >/dev/null 2>&1 || FIRST_DEPLOY_MIGRATE=1
|
docker service inspect "$DB_SERVICE" >/dev/null 2>&1 || FIRST_DEPLOY_MIGRATE=1
|
||||||
if [ "$SERVICE" = "all" ] || ! docker service inspect "${STACK_NAME}_web" >/dev/null 2>&1; then
|
if [ "$SERVICE" = "all" ] || ! docker service inspect "${STACK_NAME}_${SVC_WEB}" >/dev/null 2>&1; then
|
||||||
echo "==> docker stack deploy ($STACK_NAME)"
|
echo "==> docker stack deploy ($STACK_NAME)"
|
||||||
docker stack deploy --with-registry-auth -c "$APP_DIR/infra/deploy/stack/thermograph-stack.yml" "$STACK_NAME"
|
docker stack deploy --with-registry-auth -c "$STACK_FILE" "$STACK_NAME"
|
||||||
# First-ever deploy ran no migrate above (db didn't exist): wait for db,
|
# First-ever deploy ran no migrate above (db didn't exist): wait for db,
|
||||||
# migrate, then force web/worker to restart cleanly against the schema.
|
# migrate, then force web/worker to restart cleanly against the schema.
|
||||||
if [ "${FIRST_DEPLOY_MIGRATE:-0}" = "1" ]; then
|
if [ "${FIRST_DEPLOY_MIGRATE:-0}" = "1" ]; then
|
||||||
echo "==> Waiting for db, then first-boot migrate"
|
echo "==> Waiting for db, then first-boot migrate"
|
||||||
for i in $(seq 1 60); do
|
for i in $(seq 1 60); do
|
||||||
cid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${STACK_NAME}_db" | head -1)
|
cid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${DB_SERVICE}" | head -1)
|
||||||
[ -n "$cid" ] && docker exec "$cid" pg_isready -U thermograph -d thermograph >/dev/null 2>&1 && break
|
[ -n "$cid" ] && docker exec "$cid" pg_isready -U "$TG_DB_USER" -d "$TG_DB_NAME" >/dev/null 2>&1 && break
|
||||||
sleep 5
|
sleep 5
|
||||||
done
|
done
|
||||||
docker run --rm --network "$NET" \
|
docker run --rm --network "$NET" \
|
||||||
-e THERMOGRAPH_DATABASE_URL="postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph" \
|
-e THERMOGRAPH_DATABASE_URL="$MIGRATE_URL" \
|
||||||
--entrypoint /app/deploy/entrypoint.sh "$BACKEND_IMAGE" migrate
|
--entrypoint /app/deploy/entrypoint.sh "$BACKEND_IMAGE" migrate
|
||||||
docker service update --force --detach=false "${STACK_NAME}_web"
|
docker service update --force --detach=false "${STACK_NAME}_${SVC_WEB}"
|
||||||
docker service update --force --detach=false "${STACK_NAME}_worker"
|
docker service update --force --detach=false "${STACK_NAME}_${SVC_WORKER}"
|
||||||
fi
|
fi
|
||||||
else
|
else
|
||||||
case "$SERVICE" in
|
case "$SERVICE" in
|
||||||
backend)
|
backend)
|
||||||
echo "==> Rolling web + worker + lake to $BACKEND_IMAGE"
|
echo "==> Rolling $SVC_WEB + $SVC_WORKER + $SVC_LAKE to $BACKEND_IMAGE"
|
||||||
docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_web"
|
docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_${SVC_WEB}"
|
||||||
docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_worker"
|
docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_${SVC_WORKER}"
|
||||||
# lake and daemon ship in the same image; a stack file predating either
|
# lake and daemon ship in the same image; a stack file predating either
|
||||||
# has no service yet — the next SERVICE=all stack deploy creates it, so
|
# has no service yet — the next SERVICE=all stack deploy creates it, so
|
||||||
# don't fail here. The daemon especially must roll with web: they share
|
# don't fail here. The daemon especially must roll with web: they share
|
||||||
# the /internal/* contract, and a version skew between them is exactly
|
# the /internal/* contract, and a version skew between them is exactly
|
||||||
# what pinning one BACKEND_IMAGE_TAG exists to prevent (seen live: the
|
# what pinning one BACKEND_IMAGE_TAG exists to prevent (seen live: the
|
||||||
# first post-creation backend roll left the daemon a release behind).
|
# first post-creation backend roll left the daemon a release behind).
|
||||||
for extra in lake daemon; do
|
for extra in "$SVC_LAKE" "$SVC_DAEMON"; do
|
||||||
if docker service inspect "${STACK_NAME}_${extra}" >/dev/null 2>&1; then
|
if docker service inspect "${STACK_NAME}_${extra}" >/dev/null 2>&1; then
|
||||||
docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_${extra}"
|
docker service update --with-registry-auth --detach=false --image "$BACKEND_IMAGE" "${STACK_NAME}_${extra}"
|
||||||
else
|
else
|
||||||
|
|
@ -181,8 +240,8 @@ else
|
||||||
done
|
done
|
||||||
;;
|
;;
|
||||||
frontend)
|
frontend)
|
||||||
echo "==> Rolling frontend to $FRONTEND_IMAGE"
|
echo "==> Rolling $SVC_FRONTEND to $FRONTEND_IMAGE"
|
||||||
docker service update --with-registry-auth --detach=false --image "$FRONTEND_IMAGE" "${STACK_NAME}_frontend"
|
docker service update --with-registry-auth --detach=false --image "$FRONTEND_IMAGE" "${STACK_NAME}_${SVC_FRONTEND}"
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
fi
|
fi
|
||||||
|
|
@ -192,13 +251,20 @@ fi
|
||||||
# 0.0.0.0) on the attachable overlay, proxying to the service VIPs. Recreated
|
# 0.0.0.0) on the attachable overlay, proxying to the service VIPs. Recreated
|
||||||
# only when missing/dead — its config rarely changes; `docker rm -f $LB_NAME`
|
# only when missing/dead — its config rarely changes; `docker rm -f $LB_NAME`
|
||||||
# to force a refresh after editing lb/Caddyfile.
|
# to force a refresh after editing lb/Caddyfile.
|
||||||
|
#
|
||||||
|
# Note the network: the LB proxies to THIS stack's own service VIPs, so it joins
|
||||||
|
# this stack's own overlay — which for beta is beta's `internal`, not the shared
|
||||||
|
# data network its tasks also sit on. And its config is per-environment, because
|
||||||
|
# beta's services are named beta-web/beta-frontend.
|
||||||
|
LB_NETWORK="${STACK_NAME}_internal"
|
||||||
|
LB_CONFIG="$APP_DIR/infra/${TG_LB_CONFIG:-deploy/stack/lb/Caddyfile}"
|
||||||
if ! docker ps --format '{{.Names}}' | grep -qx "$LB_NAME"; then
|
if ! docker ps --format '{{.Names}}' | grep -qx "$LB_NAME"; then
|
||||||
docker rm -f "$LB_NAME" >/dev/null 2>&1 || true
|
docker rm -f "$LB_NAME" >/dev/null 2>&1 || true
|
||||||
echo "==> Starting loopback LB bridge $LB_NAME (127.0.0.1:$LB_HTTP_PORT, :$LB_FE_PORT)"
|
echo "==> Starting loopback LB bridge $LB_NAME (127.0.0.1:$LB_HTTP_PORT, :$LB_FE_PORT)"
|
||||||
docker run -d --name "$LB_NAME" --restart unless-stopped \
|
docker run -d --name "$LB_NAME" --restart unless-stopped \
|
||||||
--network "$NET" \
|
--network "$LB_NETWORK" \
|
||||||
-p "127.0.0.1:${LB_HTTP_PORT}:8137" -p "127.0.0.1:${LB_FE_PORT}:8080" \
|
-p "127.0.0.1:${LB_HTTP_PORT}:8137" -p "127.0.0.1:${LB_FE_PORT}:8080" \
|
||||||
-v "$APP_DIR/infra/deploy/stack/lb/Caddyfile:/etc/caddy/Caddyfile:ro" \
|
-v "$LB_CONFIG:/etc/caddy/Caddyfile:ro" \
|
||||||
caddy:2-alpine >/dev/null
|
caddy:2-alpine >/dev/null
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|
@ -233,8 +299,14 @@ docker images --format '{{.Repository}}:{{.Tag}}' \
|
||||||
|
|
||||||
# Post-deploy warm + IndexNow, via any web task (skip in test mode: no data,
|
# Post-deploy warm + IndexNow, via any web task (skip in test mode: no data,
|
||||||
# and the warmer would burn upstream quota against an empty cache).
|
# and the warmer would burn upstream quota against an empty cache).
|
||||||
if [ "${STACK_TEST:-0}" != "1" ] && { [ "$SERVICE" = backend ] || [ "$SERVICE" = all ]; }; then
|
#
|
||||||
wcid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${STACK_NAME}_web" | head -1)
|
# TG_POST_DEPLOY gates this per environment: prod does both, beta does neither
|
||||||
|
# (IndexNow from beta would ask search engines to index beta.thermograph.org,
|
||||||
|
# and the warm would spend the shared upstream archive quota on a rehearsal
|
||||||
|
# cache). See env-topology.sh.
|
||||||
|
if [ "${STACK_TEST:-0}" != "1" ] && [ "${TG_POST_DEPLOY:-1}" = 1 ] \
|
||||||
|
&& { [ "$SERVICE" = backend ] || [ "$SERVICE" = all ]; }; then
|
||||||
|
wcid=$(docker ps -q --filter "label=com.docker.swarm.service.name=${STACK_NAME}_${SVC_WEB}" | head -1)
|
||||||
if [ -n "$wcid" ]; then
|
if [ -n "$wcid" ]; then
|
||||||
echo "==> Warming city archives (detached) + IndexNow"
|
echo "==> Warming city archives (detached) + IndexNow"
|
||||||
docker exec -d "$wcid" sh -c 'python warm_cities.py --pace 2 >> /app/logs/warm-cities.log 2>&1' || true
|
docker exec -d "$wcid" sh -c 'python warm_cities.py --pace 2 >> /app/logs/warm-cities.log 2>&1' || true
|
||||||
|
|
|
||||||
34
infra/deploy/stack/lb/Caddyfile.beta
Normal file
34
infra/deploy/stack/lb/Caddyfile.beta
Normal file
|
|
@ -0,0 +1,34 @@
|
||||||
|
# Beta's loopback LB bridge config — the beta counterpart of ./Caddyfile.
|
||||||
|
#
|
||||||
|
# Two differences from prod's, both load-bearing:
|
||||||
|
#
|
||||||
|
# 1. It proxies to beta-web / beta-frontend, not web / frontend. Beta's Swarm
|
||||||
|
# services carry that prefix so their short DNS names cannot collide with
|
||||||
|
# prod's on the shared overlay (see thermograph-beta-stack.yml).
|
||||||
|
# 2. deploy-stack.sh publishes this container on 127.0.0.1:8237 and :8180,
|
||||||
|
# not 8137/8180 — prod's LB already owns 8137/8080 on this host. The
|
||||||
|
# LISTEN ports below stay 8137/8080: those are inside the container, and
|
||||||
|
# the host mapping is what differs. Keeping the container ports identical
|
||||||
|
# to prod's means the two configs differ only in the upstream names.
|
||||||
|
#
|
||||||
|
# The host Caddy on vps2 terminates TLS for beta.thermograph.org and proxies to
|
||||||
|
# 127.0.0.1:8237 / :8180 — see deploy/Caddyfile.
|
||||||
|
|
||||||
|
{
|
||||||
|
auto_https off
|
||||||
|
admin off
|
||||||
|
}
|
||||||
|
|
||||||
|
:8137 {
|
||||||
|
reverse_proxy beta-web:8137 {
|
||||||
|
# Fail fast to the client if the VIP has no healthy task; Swarm's own
|
||||||
|
# task healthchecks handle ejecting dead replicas from the VIP.
|
||||||
|
lb_try_duration 5s
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
:8080 {
|
||||||
|
reverse_proxy beta-frontend:8080 {
|
||||||
|
lb_try_duration 5s
|
||||||
|
}
|
||||||
|
}
|
||||||
278
infra/deploy/stack/thermograph-beta-stack.yml
Normal file
278
infra/deploy/stack/thermograph-beta-stack.yml
Normal file
|
|
@ -0,0 +1,278 @@
|
||||||
|
# Docker Swarm stack for BETA, co-resident with prod on vps2.
|
||||||
|
#
|
||||||
|
# Deployed by the same deploy/stack/deploy-stack.sh as prod, which picks this
|
||||||
|
# file (and beta's ports, env file, LB and DB role) out of deploy/env-topology.sh
|
||||||
|
# when THERMOGRAPH_ENV=beta. Beta moved here from its own box so that a beta
|
||||||
|
# green light is evidence about prod: same orchestrator, same host kernel, same
|
||||||
|
# Postgres build, same Caddy, same mail path, same mesh position.
|
||||||
|
#
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# WHY THIS IS A SEPARATE FILE AND NOT AN OVERLAY ON thermograph-stack.yml
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# `docker stack deploy` accepts multiple -c files and MERGES them. Merging can
|
||||||
|
# add and override, but it cannot REMOVE a service — and the single most
|
||||||
|
# important fact about beta is a removal: it has no `db`. It uses prod's. An
|
||||||
|
# overlay would therefore still create a second Postgres, which is the exact
|
||||||
|
# thing this design exists to avoid. The same goes for the two autoscalers,
|
||||||
|
# which beta deliberately does not run.
|
||||||
|
#
|
||||||
|
# The cost is a file that must be kept roughly in step with prod's by hand.
|
||||||
|
# Keep them in step for anything that affects whether the APP works (env vars,
|
||||||
|
# entrypoints, healthchecks, the migrate contract). Do NOT keep them in step on
|
||||||
|
# scale, replicas or resource limits — those differ on purpose (below).
|
||||||
|
#
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# THE THREE THINGS THAT MAKE CO-RESIDENCY SAFE
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. SERVICE NAMES ARE PREFIXED (beta-web, not web). Swarm registers a service's
|
||||||
|
# short name as a DNS alias on every network it joins. Beta's tasks share the
|
||||||
|
# `data` network with prod's, so two services both called `web` would make
|
||||||
|
# `web` ambiguous — prod's frontend could resolve a beta task, and vice
|
||||||
|
# versa. The prefix removes the collision without touching prod's stack file.
|
||||||
|
#
|
||||||
|
# 2. THE DATABASE IS SHARED, THE DATA IS NOT. Beta connects to prod's `db`
|
||||||
|
# service as the role `thermograph_beta`, to the database `thermograph_beta`.
|
||||||
|
# That role owns only its own database (see deploy/db/provision-env-db.sh),
|
||||||
|
# so a beta deploy running an unmerged branch — or a migration that goes
|
||||||
|
# wrong — cannot read or write production data. One server is a capacity
|
||||||
|
# decision, not a trust decision.
|
||||||
|
#
|
||||||
|
# 3. NOTHING ELSE IS SHARED BY ACCIDENT. Separate checkout (/opt/thermograph-beta),
|
||||||
|
# separate rendered env file (/etc/thermograph-beta.env), separate stack env
|
||||||
|
# (/etc/thermograph/beta-stack.env), separate volumes, separate loopback
|
||||||
|
# ports (8237/8180 — prod owns 8137/8080), separate LB container, separate
|
||||||
|
# deploy lock and image-tag file. Every one of those is derived in
|
||||||
|
# env-topology.sh rather than repeated here by hand.
|
||||||
|
#
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# WHAT BETA DELIBERATELY DOES NOT DO
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# - No autoscaling: fixed 1 replica per service. Beta exists to answer "does
|
||||||
|
# this code work", not "does it scale"; a second replica would only add a
|
||||||
|
# variable prod's rehearsal doesn't need, on a host prod is also using.
|
||||||
|
# - No IndexNow ping and no city-archive warm (deploy-stack.sh gates both on
|
||||||
|
# TG_POST_DEPLOY). Pinging IndexNow from beta asks Bing/DuckDuckGo/Yandex to
|
||||||
|
# index beta.thermograph.org; the warm spends the shared upstream archive
|
||||||
|
# quota to fill a cache only a rehearsal reads.
|
||||||
|
# - No real mail and no Discord gateway. Both are governed by beta's vault
|
||||||
|
# (THERMOGRAPH_MAIL_BACKEND=console, THERMOGRAPH_DISCORD_BOT=0) rather than
|
||||||
|
# pinned here, so the operator can opt in with `sops edit` if a release ever
|
||||||
|
# genuinely needs to rehearse them. Discord in particular allows ONE gateway
|
||||||
|
# connection per bot token — beta and prod must never both hold one.
|
||||||
|
|
||||||
|
services:
|
||||||
|
beta-web:
|
||||||
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
|
environment:
|
||||||
|
# Beta's OWN role and OWN database on the shared instance. `db` resolves
|
||||||
|
# across the external `data` network to prod's db service.
|
||||||
|
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph_beta:${POSTGRES_PASSWORD}@db:5432/thermograph_beta
|
||||||
|
THERMOGRAPH_BASE: /
|
||||||
|
PORT: 8137
|
||||||
|
THERMOGRAPH_SERVICE_ROLE: backend
|
||||||
|
THERMOGRAPH_FRONTEND_BASE_INTERNAL: http://beta-frontend:8080
|
||||||
|
WORKERS: ${BETA_WEB_WORKERS:-2}
|
||||||
|
THERMOGRAPH_DATA_DIR: /state
|
||||||
|
# Never the notifier/scheduler — that is beta-worker's job, exactly as in
|
||||||
|
# prod, so the two files stay honest about which process owns what.
|
||||||
|
THERMOGRAPH_ROLE: web
|
||||||
|
# Migrations run as the one-shot task in deploy-stack.sh.
|
||||||
|
RUN_MIGRATIONS: "0"
|
||||||
|
# Overlay tasks reach the HOST's Postfix via the docker_gwbridge gateway.
|
||||||
|
# Same host, same Postfix as prod — but see the mail note in the header:
|
||||||
|
# beta's vault selects the console backend, so nothing is actually sent.
|
||||||
|
THERMOGRAPH_SMTP_HOST: ${STACK_SMTP_HOST:-172.18.0.1}
|
||||||
|
THERMOGRAPH_LAKE_URL: http://beta-lake:8141
|
||||||
|
volumes:
|
||||||
|
- appdata:/state
|
||||||
|
- applogs:/app/logs
|
||||||
|
- /opt/thermograph-beta/infra/deploy/stack/env-entrypoint.sh:/host/env-entrypoint.sh:ro
|
||||||
|
- /etc/thermograph/beta-stack.env:/host/thermograph.env:ro
|
||||||
|
networks:
|
||||||
|
- internal
|
||||||
|
- data
|
||||||
|
deploy:
|
||||||
|
replicas: 1
|
||||||
|
# vps2 is the Swarm manager and every volume here is local to it. The
|
||||||
|
# desktop is a worker on this mesh and must never be scheduled the app.
|
||||||
|
placement:
|
||||||
|
constraints: ["node.role == manager"]
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
cpus: "${BETA_WEB_CPUS:-2}"
|
||||||
|
restart_policy:
|
||||||
|
condition: on-failure
|
||||||
|
update_config:
|
||||||
|
order: start-first
|
||||||
|
failure_action: rollback
|
||||||
|
|
||||||
|
beta-worker:
|
||||||
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
|
environment:
|
||||||
|
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph_beta:${POSTGRES_PASSWORD}@db:5432/thermograph_beta
|
||||||
|
THERMOGRAPH_BASE: /
|
||||||
|
PORT: 8137
|
||||||
|
THERMOGRAPH_SERVICE_ROLE: backend
|
||||||
|
THERMOGRAPH_FRONTEND_BASE_INTERNAL: http://beta-frontend:8080
|
||||||
|
WORKERS: "1"
|
||||||
|
THERMOGRAPH_DATA_DIR: /state
|
||||||
|
THERMOGRAPH_ROLE: worker
|
||||||
|
# The advisory lock is taken in beta's OWN database, so it can never
|
||||||
|
# contend with prod's worker despite the shared server.
|
||||||
|
THERMOGRAPH_SINGLETON_PG: "1"
|
||||||
|
RUN_MIGRATIONS: "0"
|
||||||
|
THERMOGRAPH_SMTP_HOST: ${STACK_SMTP_HOST:-172.18.0.1}
|
||||||
|
THERMOGRAPH_LAKE_URL: http://beta-lake:8141
|
||||||
|
volumes:
|
||||||
|
- appdata:/state
|
||||||
|
- applogs:/app/logs
|
||||||
|
- /opt/thermograph-beta/infra/deploy/stack/env-entrypoint.sh:/host/env-entrypoint.sh:ro
|
||||||
|
- /etc/thermograph/beta-stack.env:/host/thermograph.env:ro
|
||||||
|
networks:
|
||||||
|
- internal
|
||||||
|
- data
|
||||||
|
deploy:
|
||||||
|
replicas: 1
|
||||||
|
placement:
|
||||||
|
constraints: ["node.role == manager"]
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
cpus: "${BETA_WORKER_CPUS:-1}"
|
||||||
|
restart_policy:
|
||||||
|
condition: on-failure
|
||||||
|
|
||||||
|
beta-lake:
|
||||||
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
|
environment:
|
||||||
|
THERMOGRAPH_ROLE: lake
|
||||||
|
PORT: 8141
|
||||||
|
THERMOGRAPH_SERVICE_ROLE: backend
|
||||||
|
WORKERS: "1"
|
||||||
|
THERMOGRAPH_LAKE_CACHE: /state/lake-cache
|
||||||
|
volumes:
|
||||||
|
# Beta's own cache volume. Deliberately not prod's: they are read caches
|
||||||
|
# of the same bucket, but sharing a volume across two stacks would couple
|
||||||
|
# their lifecycles for no gain.
|
||||||
|
- lakecache:/state
|
||||||
|
- /opt/thermograph-beta/infra/deploy/stack/env-entrypoint.sh:/host/env-entrypoint.sh:ro
|
||||||
|
- /etc/thermograph/beta-stack.env:/host/thermograph.env:ro
|
||||||
|
# No `data` network: the lake reads object storage, never Postgres.
|
||||||
|
networks:
|
||||||
|
- internal
|
||||||
|
deploy:
|
||||||
|
replicas: 1
|
||||||
|
placement:
|
||||||
|
constraints: ["node.role == manager"]
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
cpus: "${BETA_LAKE_CPUS:-1}"
|
||||||
|
restart_policy:
|
||||||
|
condition: on-failure
|
||||||
|
update_config:
|
||||||
|
order: start-first
|
||||||
|
failure_action: rollback
|
||||||
|
|
||||||
|
beta-daemon:
|
||||||
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:?required}
|
||||||
|
# Same reasoning as prod's daemon: NOT env-entrypoint.sh, because that shim
|
||||||
|
# execs the image's own entrypoint (Alembic + uvicorn) and migrations belong
|
||||||
|
# to the one-shot task. Source the host-rendered env and exec the binary.
|
||||||
|
#
|
||||||
|
# Note the ordering consequence, which is load-bearing here: this sources
|
||||||
|
# the env file AFTER the `environment:` block is applied, so a key present
|
||||||
|
# in /etc/thermograph/beta-stack.env WINS over one set below. That is why
|
||||||
|
# THERMOGRAPH_DISCORD_BOT=0 lives in beta's vault and not in this file — a
|
||||||
|
# value set here would be silently overridden if the vault ever set one.
|
||||||
|
entrypoint:
|
||||||
|
- /bin/bash
|
||||||
|
- -c
|
||||||
|
- 'set -a; [ -f /host/thermograph.env ] && . /host/thermograph.env; set +a; exec /usr/local/bin/thermograph-daemon'
|
||||||
|
environment:
|
||||||
|
THERMOGRAPH_API_BASE_INTERNAL: http://beta-web:8137
|
||||||
|
volumes:
|
||||||
|
- /etc/thermograph/beta-stack.env:/host/thermograph.env:ro
|
||||||
|
# The image HEALTHCHECK curls /healthz on ${PORT}; the daemon serves
|
||||||
|
# nothing, so without this override Swarm restarts it forever.
|
||||||
|
healthcheck:
|
||||||
|
disable: true
|
||||||
|
networks:
|
||||||
|
- internal
|
||||||
|
deploy:
|
||||||
|
# EXACTLY 1, and in beta's case the Discord gateway is off entirely
|
||||||
|
# (vault: THERMOGRAPH_DISCORD_BOT=0) because prod's daemon holds the only
|
||||||
|
# permitted gateway connection for that bot token.
|
||||||
|
replicas: 1
|
||||||
|
placement:
|
||||||
|
constraints: ["node.role == manager"]
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
cpus: "0.5"
|
||||||
|
memory: 128m
|
||||||
|
restart_policy:
|
||||||
|
condition: on-failure
|
||||||
|
update_config:
|
||||||
|
order: stop-first
|
||||||
|
failure_action: rollback
|
||||||
|
|
||||||
|
beta-frontend:
|
||||||
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${FRONTEND_IMAGE_PATH:-emi/thermograph/frontend}:${FRONTEND_IMAGE_TAG:?required}
|
||||||
|
entrypoint: ["/host/env-entrypoint.sh"]
|
||||||
|
# REQUIRED: overriding `entrypoint:` with no `command:` drops the image's
|
||||||
|
# CMD entirely, and env-entrypoint.sh's fallback (`exec uvicorn app:app`)
|
||||||
|
# does not exist in this Go image — the task would exit 127 every deploy.
|
||||||
|
command: ["/usr/local/bin/thermograph-frontend"]
|
||||||
|
environment:
|
||||||
|
THERMOGRAPH_BASE: /
|
||||||
|
PORT: 8080
|
||||||
|
THERMOGRAPH_SERVICE_ROLE: frontend
|
||||||
|
THERMOGRAPH_API_BASE_INTERNAL: http://beta-web:8137
|
||||||
|
volumes:
|
||||||
|
- /opt/thermograph-beta/infra/deploy/stack/env-entrypoint.sh:/host/env-entrypoint.sh:ro
|
||||||
|
- /etc/thermograph/beta-stack.env:/host/thermograph.env:ro
|
||||||
|
networks:
|
||||||
|
- internal
|
||||||
|
deploy:
|
||||||
|
replicas: 1
|
||||||
|
placement:
|
||||||
|
constraints: ["node.role == manager"]
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
cpus: "${BETA_FRONTEND_CPUS:-1}"
|
||||||
|
restart_policy:
|
||||||
|
condition: on-failure
|
||||||
|
update_config:
|
||||||
|
order: start-first
|
||||||
|
failure_action: rollback
|
||||||
|
|
||||||
|
networks:
|
||||||
|
# Beta's own east-west network: beta-web <-> beta-frontend <-> beta-lake, and
|
||||||
|
# the loopback LB bridge joins it (hence attachable). Keeping this separate
|
||||||
|
# from prod's overlay means beta's ordinary traffic never touches it.
|
||||||
|
internal:
|
||||||
|
driver: overlay
|
||||||
|
attachable: true
|
||||||
|
|
||||||
|
# Prod's overlay, joined ONLY to reach the shared `db` service. Declared
|
||||||
|
# external because prod's stack owns it: `docker stack deploy` of THIS file
|
||||||
|
# must never create, modify or (on `docker stack rm thermograph-beta`) remove
|
||||||
|
# the network prod's database is on.
|
||||||
|
#
|
||||||
|
# Consequence worth knowing before you tear anything down: `docker stack rm
|
||||||
|
# thermograph` would take this network with it and beta would lose its
|
||||||
|
# database link until prod is redeployed.
|
||||||
|
data:
|
||||||
|
external: true
|
||||||
|
name: thermograph_internal
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
# Beta's own, created by this stack under the thermograph-beta_ prefix. Unlike
|
||||||
|
# prod's (which are `external` because they were inherited from the compose
|
||||||
|
# era and hold live data), these can be recreated: beta's appdata is a parquet
|
||||||
|
# cache plus derived files, and its DATABASE — the part that matters — lives
|
||||||
|
# on the shared instance, not here.
|
||||||
|
appdata: {}
|
||||||
|
applogs: {}
|
||||||
|
lakecache: {}
|
||||||
|
|
@ -1,36 +1,41 @@
|
||||||
# 3-node Docker Swarm (prod + beta + desktop), for hosting Forgejo
|
# 3-node Docker Swarm (vps2 + vps1 + desktop)
|
||||||
|
|
||||||
This Swarm cluster's only job is to run Forgejo (`deploy/forgejo/`) — it does
|
This Swarm mesh's only **Swarm-scheduled** workload is Forgejo
|
||||||
**not** orchestrate the Thermograph app itself, which stays on the
|
(`deploy/forgejo/`), pinned to vps1. It does **not** orchestrate the
|
||||||
Terraform-managed `docker compose` deploys on prod/beta independently (see
|
Thermograph app the same way — prod and beta each run as their own `docker
|
||||||
`terraform/README.md`). Keeping those separate means nothing here can strand
|
stack deploy` (`deploy/stack/thermograph-stack.yml` /
|
||||||
or interfere with the app's already-working, single-writer Postgres/TimescaleDB
|
`thermograph-beta-stack.yml`) that happens to land on this same manager node
|
||||||
deploys.
|
(vps2) because their volumes are local to it today. Keeping Forgejo's stack
|
||||||
|
and the app stacks conceptually separate means nothing here can strand or
|
||||||
|
interfere with the app's single-writer Postgres/TimescaleDB.
|
||||||
|
|
||||||
This is the canonical topology from
|
This is the canonical topology from
|
||||||
`thermograph-docs/runbooks/implementation-handoff.md` (Track B steps 2-3) — three nodes,
|
`thermograph-docs/runbooks/implementation-handoff.md` (Track B steps 2-3) — three nodes,
|
||||||
not two. An earlier revision of this doc/scripts covered just prod+beta;
|
not two. The desktop (formerly the LAN dev machine) joins as a worker for flex
|
||||||
the desktop (this LAN dev machine) joins too.
|
capacity and AI-model hosting; it hosts no Thermograph environment.
|
||||||
|
|
||||||
**Nodes:**
|
**Nodes:**
|
||||||
- **manager** — prod, the new 48 GB / 12-core box (more headroom).
|
- **manager** — vps2 (`169.58.46.181`), the box with headroom for prod's and
|
||||||
- **worker** — beta, the old VPS (`75.119.132.91`).
|
beta's Swarm stacks and their local volumes.
|
||||||
- **worker** — desktop, this LAN dev machine (also runs the Forgejo Actions
|
- **worker** — vps1 (`75.119.132.91`), pinned to run Forgejo
|
||||||
runner as a plain systemd service — see `deploy/forgejo/README.md` — not as
|
(`node.labels.role == forge`). Also runs Grafana/Loki/Alloy and the dev
|
||||||
a Swarm-scheduled container).
|
environment, both outside this Swarm cluster (plain Docker/compose on the
|
||||||
|
same host's daemon, not Swarm-scheduled).
|
||||||
|
- **worker** — desktop (this machine), flex capacity plus AI-model hosting. No
|
||||||
|
Thermograph environment runs here.
|
||||||
|
|
||||||
One manager, not more: Raft needs 3 nodes for real quorum-based HA, and this
|
One manager, not more: Raft needs 3 nodes for real quorum-based HA, and this
|
||||||
cluster only has 3 nodes total, so making even one more of them a manager
|
cluster only has 3 nodes total, so making even one more of them a manager
|
||||||
would still fall short of real HA while adding split-brain risk. If the
|
would still fall short of real HA while adding split-brain risk. If the
|
||||||
manager (prod) goes down, the workers keep running whatever was already
|
manager (vps2) goes down, the workers keep running whatever was already
|
||||||
scheduled on them (Forgejo, pinned to beta) but the cluster can't reschedule
|
scheduled on them (Forgejo, pinned to vps1) but the cluster can't reschedule
|
||||||
anything until prod's back — acceptable for a small cluster whose only job is
|
anything until vps2's back — acceptable for a small cluster whose only
|
||||||
CI/CD.
|
Swarm-scheduled job is CI/CD.
|
||||||
|
|
||||||
## Order of operations
|
## Order of operations
|
||||||
|
|
||||||
1. **Agent access first** (`deploy/provision-agent-access.sh`) on prod and
|
1. **Agent access first** (`deploy/provision-agent-access.sh`) on vps1 and
|
||||||
beta — everything below on those two boxes is run through that access. The
|
vps2 — everything below on those two boxes is run through that access. The
|
||||||
desktop is wherever you're already working from; no separate access step
|
desktop is wherever you're already working from; no separate access step
|
||||||
needed there.
|
needed there.
|
||||||
2. **WireGuard mesh** (`setup-wireguard.sh <my_wg_ip> <peers_file>`) — run on
|
2. **WireGuard mesh** (`setup-wireguard.sh <my_wg_ip> <peers_file>`) — run on
|
||||||
|
|
@ -38,19 +43,21 @@ CI/CD.
|
||||||
the two-pass key-exchange dance (pubkeys aren't known until every node has
|
the two-pass key-exchange dance (pubkeys aren't known until every node has
|
||||||
run it once). Verify with `ping <peer_wg_ip>` to each of the other two
|
run it once). Verify with `ping <peer_wg_ip>` to each of the other two
|
||||||
before continuing.
|
before continuing.
|
||||||
3. **Swarm init** (`init-swarm.sh <manager_wg_ip>`) on the manager (prod) only.
|
3. **Swarm init** (`init-swarm.sh <manager_wg_ip>`) on the manager (vps2) only.
|
||||||
4. **Swarm join** (`join-swarm.sh <manager_wg_ip> <token>`) on **each** of the
|
4. **Swarm join** (`join-swarm.sh <manager_wg_ip> <token>`) on **each** of the
|
||||||
two workers (beta, desktop) — same token for both.
|
two workers (vps1, desktop) — same token for both.
|
||||||
5. **Firewall lockdown** (`firewall-swarm.sh`) on **all three** nodes — closes
|
5. **Firewall lockdown** (`firewall-swarm.sh`) on **all three** nodes — closes
|
||||||
2377/7946/4789 to everything except the WireGuard interface. Do this
|
2377/7946/4789 to everything except the WireGuard interface. Do this
|
||||||
*after* joining is confirmed working on all three, not before (locking the
|
*after* joining is confirmed working on all three, not before (locking the
|
||||||
ports first would make the join itself fail).
|
ports first would make the join itself fail).
|
||||||
6. **Label beta** (`label-forge-node.sh <beta-node-name>`) on the manager —
|
6. **Label vps1** (`label-forge-node.sh <vps1-node-name>`) on the manager —
|
||||||
`docker node ls` shows each node's name/ID. Only beta gets `role=forge`;
|
`docker node ls` shows each node's name/ID. Only vps1 gets `role=forge`;
|
||||||
the desktop and prod don't need a Swarm label for anything in this setup.
|
the desktop and vps2 don't need a Swarm label for anything in this setup.
|
||||||
7. Deploy Forgejo: see `deploy/forgejo/README.md`.
|
7. Deploy Forgejo: see `deploy/forgejo/README.md`.
|
||||||
8. Register the Actions runner **on the desktop** (not through Swarm):
|
8. Register the Actions runner: `deploy/forgejo/register-lan-runner.sh`. See
|
||||||
`deploy/forgejo/register-lan-runner.sh`.
|
that document (and `DEPLOY-DEV.md`) for where it actually runs today — its
|
||||||
|
own header comment predates the vps1/vps2 rename and still describes "the
|
||||||
|
desktop" as the canonical placement.
|
||||||
|
|
||||||
## Why WireGuard instead of relying on Swarm's built-in TLS alone
|
## Why WireGuard instead of relying on Swarm's built-in TLS alone
|
||||||
|
|
||||||
|
|
@ -65,19 +72,19 @@ than trusting the public internet (or the desktop's home network) directly.
|
||||||
## Verifying
|
## Verifying
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# On the manager:
|
# On the manager (vps2):
|
||||||
docker node ls # all three nodes Ready
|
docker node ls # all three nodes Ready
|
||||||
docker node inspect <beta-node> --format '{{.Spec.Labels}}' # role:forge
|
docker node inspect <vps1-node> --format '{{.Spec.Labels}}' # role:forge
|
||||||
|
|
||||||
# From a FOURTH machine outside the mesh entirely, confirm the Swarm ports
|
# From a FOURTH machine outside the mesh entirely, confirm the Swarm ports
|
||||||
# are NOT reachable on either VPS's public IP (the desktop has no public IP
|
# are NOT reachable on either VPS's public IP (the desktop has no public IP
|
||||||
# to check this way):
|
# to check this way):
|
||||||
nc -zv -w2 <prod_or_beta_public_ip> 2377 # should fail/timeout
|
nc -zv -w2 <vps1_or_vps2_public_ip> 2377 # should fail/timeout
|
||||||
nc -zvu -w2 <prod_or_beta_public_ip> 4789 # should fail/timeout
|
nc -zvu -w2 <vps1_or_vps2_public_ip> 4789 # should fail/timeout
|
||||||
```
|
```
|
||||||
|
|
||||||
## Adding a node label back out (undo)
|
## Adding a node label back out (undo)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker node update --label-rm role <beta-node>
|
docker node update --label-rm role <vps1-node>
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,8 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Locks the Swarm ports (2377 control, 7946 gossip, 4789 overlay VXLAN) to the
|
# Locks the Swarm ports (2377 control, 7946 gossip, 4789 overlay VXLAN) to the
|
||||||
# WireGuard interface only — they must never be reachable from the public
|
# WireGuard interface only — they must never be reachable from the public
|
||||||
# internet. Run on BOTH boxes after joining the swarm. Existing app-facing
|
# internet. Run on ALL THREE nodes (vps2, vps1, desktop) after joining the
|
||||||
# rules (80/443, SSH, etc.) are untouched.
|
# swarm. Existing app-facing rules (80/443, SSH, etc.) are untouched.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
WG_IFACE="${WG_IFACE:-wg0}"
|
WG_IFACE="${WG_IFACE:-wg0}"
|
||||||
|
|
@ -34,8 +34,10 @@ if ! ufw status | grep -q "^Status: active"; then
|
||||||
echo "ufw is currently INACTIVE on this node — 'ufw enable' switches its"
|
echo "ufw is currently INACTIVE on this node — 'ufw enable' switches its"
|
||||||
echo "default policy to deny-incoming for EVERYTHING, not just the Swarm"
|
echo "default policy to deny-incoming for EVERYTHING, not just the Swarm"
|
||||||
echo "ports above. Before enabling, explicitly allow every port this node"
|
echo "ports above. Before enabling, explicitly allow every port this node"
|
||||||
echo "already serves publicly (SSH at minimum; on beta specifically, also"
|
echo "already serves publicly (SSH at minimum; on vps2 specifically, also"
|
||||||
echo "80/tcp and 443/tcp for the live thermograph.org Caddy) — check"
|
echo "80/tcp and 443/tcp for the live thermograph.org/beta.thermograph.org"
|
||||||
echo "'ss -tlnp' for what's actually listening first. Enabling ufw without"
|
echo "Caddy; on vps1, 80/tcp and 443/tcp for git.thermograph.org,"
|
||||||
echo "doing this WILL drop live traffic the moment it activates."
|
echo "dashboard.thermograph.org and emigriffith.dev) — check 'ss -tlnp' for"
|
||||||
|
echo "what's actually listening first. Enabling ufw without doing this WILL"
|
||||||
|
echo "drop live traffic the moment it activates."
|
||||||
fi
|
fi
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,9 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Run ONCE, on the manager node only (prod — the new 48 GB box). Initializes
|
# Run ONCE, on the manager node only (vps2 — the box with headroom for prod's
|
||||||
# the Swarm advertising the WireGuard address, so cluster traffic never
|
# and beta's Swarm stacks and their local volumes). Initializes the Swarm
|
||||||
# touches the public interface. Run setup-wireguard.sh on all THREE nodes
|
# advertising the WireGuard address, so cluster traffic never touches the
|
||||||
# first (prod, beta, and the desktop — see
|
# public interface. Run setup-wireguard.sh on all THREE nodes first (vps2,
|
||||||
|
# vps1, and the desktop — see
|
||||||
# docs/runbooks/implementation-handoff.md Track B steps 2-3).
|
# docs/runbooks/implementation-handoff.md Track B steps 2-3).
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
|
|
@ -21,7 +22,7 @@ echo "==> docker swarm init, advertising ${MY_WG_IP} (the WireGuard address, not
|
||||||
docker swarm init --advertise-addr "$MY_WG_IP" --listen-addr "${MY_WG_IP}:2377"
|
docker swarm init --advertise-addr "$MY_WG_IP" --listen-addr "${MY_WG_IP}:2377"
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "==> Worker join command (run this on EACH of the two workers — beta and"
|
echo "==> Worker join command (run this on EACH of the two workers — vps1 and"
|
||||||
echo " the desktop; the same token works for both):"
|
echo " the desktop; the same token works for both):"
|
||||||
TOKEN="$(docker swarm join-token -q worker)"
|
TOKEN="$(docker swarm join-token -q worker)"
|
||||||
echo " docker swarm join --token ${TOKEN} ${MY_WG_IP}:2377"
|
echo " docker swarm join --token ${TOKEN} ${MY_WG_IP}:2377"
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Run ONCE on EACH worker node (beta, and the desktop — not the manager,
|
# Run ONCE on EACH worker node (vps1, and the desktop — not the manager,
|
||||||
# prod). Joins the Swarm initialized by init-swarm.sh, over the WireGuard
|
# vps2). Joins the Swarm initialized by init-swarm.sh, over the WireGuard
|
||||||
# tunnel. The join token is the same for both workers.
|
# tunnel. The join token is the same for both workers.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Run ONCE, on the manager node, after both boxes have joined the swarm.
|
# Run ONCE, on the manager node (vps2), after both boxes have joined the
|
||||||
# Labels the worker (beta) so the Forgejo stack's placement constraint
|
# swarm. Labels the worker (vps1) so the Forgejo stack's placement constraint
|
||||||
# (node.labels.role == forge) schedules there and nowhere else.
|
# (node.labels.role == forge) schedules there and nowhere else.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Sets up this node's side of a full-mesh WireGuard tunnel across all of
|
# Sets up this node's side of a full-mesh WireGuard tunnel across all of
|
||||||
# prod, beta, and the desktop (three nodes, not two — see
|
# vps2, vps1, and the desktop (three nodes, not two — see
|
||||||
# docs/runbooks/implementation-handoff.md Track B step 2). Docker Swarm's
|
# docs/runbooks/implementation-handoff.md Track B step 2). Docker Swarm's
|
||||||
# control plane (2377/tcp) is TLS-encrypted by default, but the overlay data
|
# control plane (2377/tcp) is TLS-encrypted by default, but the overlay data
|
||||||
# plane (VXLAN, 4789/udp) is NOT — and it should never face the public
|
# plane (VXLAN, 4789/udp) is NOT — and it should never face the public
|
||||||
|
|
@ -14,8 +14,8 @@
|
||||||
# --- Peer list format --------------------------------------------------
|
# --- Peer list format --------------------------------------------------
|
||||||
# A text file, one line per OTHER node (not including the one you're running
|
# A text file, one line per OTHER node (not including the one you're running
|
||||||
# on), each line: <wg_ip> <public_ip> <pubkey-or-dash>
|
# on), each line: <wg_ip> <public_ip> <pubkey-or-dash>
|
||||||
# 10.10.0.1 169.58.46.181 <prod's pubkey, or - if not yet known>
|
# 10.10.0.1 169.58.46.181 <vps2's pubkey, or - if not yet known>
|
||||||
# 10.10.0.2 75.119.132.91 <beta's pubkey, or ->
|
# 10.10.0.2 75.119.132.91 <vps1's pubkey, or ->
|
||||||
# 10.10.0.3 <desktop's public/reachable IP or a DDNS name> <desktop's pubkey, or ->
|
# 10.10.0.3 <desktop's public/reachable IP or a DDNS name> <desktop's pubkey, or ->
|
||||||
#
|
#
|
||||||
# First pass on any node: peer pubkeys you don't have yet are "-". Run this
|
# First pass on any node: peer pubkeys you don't have yet are "-". Run this
|
||||||
|
|
|
||||||
|
|
@ -46,8 +46,9 @@ POSTGRES_PASSWORD=change-me
|
||||||
# an empty value is honored as-is and would break the fallback to the public API.
|
# an empty value is honored as-is and would break the fallback to the public API.
|
||||||
#THERMOGRAPH_ARCHIVE_URL=http://open-meteo-api:8080/v1/archive
|
#THERMOGRAPH_ARCHIVE_URL=http://open-meteo-api:8080/v1/archive
|
||||||
|
|
||||||
# Mark the session cookie Secure — required behind Caddy's HTTPS. Set to 1 in prod;
|
# Mark the session cookie Secure — required behind Caddy's HTTPS. Set to 1 for
|
||||||
# leave unset only for plain-HTTP LAN dev (a Secure cookie is never sent over HTTP).
|
# prod and beta (both TLS-fronted); leave unset only for dev's plain-HTTP
|
||||||
|
# mesh-only URL (a Secure cookie is never sent over HTTP).
|
||||||
THERMOGRAPH_COOKIE_SECURE=1
|
THERMOGRAPH_COOKIE_SECURE=1
|
||||||
|
|
||||||
# Pin these too (see their own sections below), so container restarts don't rotate
|
# Pin these too (see their own sections below), so container restarts don't rotate
|
||||||
|
|
@ -58,11 +59,14 @@ THERMOGRAPH_COOKIE_SECURE=1
|
||||||
|
|
||||||
# Number of uvicorn worker processes. More than 1 stops a single slow upstream fetch
|
# Number of uvicorn worker processes. More than 1 stops a single slow upstream fetch
|
||||||
# (e.g. a cache-miss weather lookup) from blocking every other request — the cause of
|
# (e.g. a cache-miss weather lookup) from blocking every other request — the cause of
|
||||||
# past brief outages. Prod runs 4 (the compose app service also defaults to WORKERS=4);
|
# past brief outages. Prod's Swarm `web` service defaults to 4 (WEB_WORKERS, also the
|
||||||
# leave unset (defaults to 1) on a small box. Workers elect one leader for the
|
# autoscaler's per-replica count); beta's defaults to 2 (BETA_WEB_WORKERS) — a
|
||||||
# subscription notifier via a lockfile (THERMOGRAPH_SINGLETON_LOCK, set by the compose
|
# rehearsal environment, not a decision that beta needs less headroom per se. The
|
||||||
# app service to /app/data/notifier.lock) so its timer-driven upstream sweep runs once,
|
# base compose file (dev) also defaults to 4. Leave unset (defaults to 1) on a small
|
||||||
# not once per worker. ~200 MB RAM per worker.
|
# box. Workers elect one leader for the subscription notifier via a lockfile
|
||||||
|
# (THERMOGRAPH_SINGLETON_LOCK, set by the compose/stack app service to
|
||||||
|
# /app/data/notifier.lock) so its timer-driven upstream sweep runs once, not once per
|
||||||
|
# worker. ~200 MB RAM per worker.
|
||||||
WORKERS=4
|
WORKERS=4
|
||||||
|
|
||||||
# THERMOGRAPH_SINGLETON_LOCK arbitrates workers on ONE host. Under multi-host Swarm,
|
# THERMOGRAPH_SINGLETON_LOCK arbitrates workers on ONE host. Under multi-host Swarm,
|
||||||
|
|
@ -206,11 +210,32 @@ THERMOGRAPH_BASE_URL=https://thermograph.org
|
||||||
# Discord allows ONE gateway connection per bot token, so enable this in exactly
|
# Discord allows ONE gateway connection per bot token, so enable this in exactly
|
||||||
# one environment — prod, the only vault with the token.
|
# one environment — prod, the only vault with the token.
|
||||||
#THERMOGRAPH_DISCORD_BOT=
|
#THERMOGRAPH_DISCORD_BOT=
|
||||||
# Account linking (OAuth2 "identify"): lets a signed-in user connect their Discord
|
# Discord as an identity (notifications/discord_link.py). One pair of credentials
|
||||||
# account, storing their Discord user id for DM alerts. CLIENT_ID is the same App
|
# gates BOTH flows, and setting them makes both appear in the UI at once:
|
||||||
# ID above. The CLIENT_SECRET is from OAuth2 -> Client Secret (a credential). In the
|
# - account linking (scope "identify") — a signed-in user connects their Discord
|
||||||
# portal, add the redirect: https://thermograph.org/api/v2/discord/link/callback
|
# account, storing their Discord user id for DM alerts;
|
||||||
|
# - sign in with Discord (scope "identify email") — an anonymous visitor
|
||||||
|
# authenticates, resolving or creating an account from the Discord identity.
|
||||||
|
# CLIENT_ID is the same App ID above. The CLIENT_SECRET is from OAuth2 -> Client
|
||||||
|
# Secret (a credential). In the portal, add the redirect:
|
||||||
|
# https://thermograph.org/api/v2/discord/link/callback
|
||||||
|
# Both flows come back to that one path (the purpose is carried in the signed
|
||||||
|
# state), so enabling sign-in needs no new redirect registered. Scopes are asked
|
||||||
|
# for per request, not registered, so the email scope needs no portal change
|
||||||
|
# either. An environment without these two vars simply shows no Discord UI.
|
||||||
#THERMOGRAPH_DISCORD_CLIENT_SECRET=
|
#THERMOGRAPH_DISCORD_CLIENT_SECRET=
|
||||||
|
# Sign in with Google / connect a Google account — the same engine as Discord
|
||||||
|
# above (accounts/oauth.py), configured independently, so an environment may offer
|
||||||
|
# either, both, or neither. From Google Cloud Console -> APIs & Services ->
|
||||||
|
# Credentials -> OAuth 2.0 Client ID, type "Web application". The CLIENT_SECRET is
|
||||||
|
# a credential. Register this exact redirect URI on the client:
|
||||||
|
# https://thermograph.org/api/v2/oauth/google/callback
|
||||||
|
# (Google matches redirect URIs exactly — scheme, host and path — so beta and dev
|
||||||
|
# each need their own entry on the same client, or their own client.) The consent
|
||||||
|
# screen needs only the `openid` and `email` scopes, both non-sensitive, so it
|
||||||
|
# requires no Google verification review.
|
||||||
|
#THERMOGRAPH_GOOGLE_CLIENT_ID=
|
||||||
|
#THERMOGRAPH_GOOGLE_CLIENT_SECRET=
|
||||||
# Gateway bot (opt-in): holds a live websocket so the bot replies to messages that
|
# Gateway bot (opt-in): holds a live websocket so the bot replies to messages that
|
||||||
# @mention it (or DM it) with a city grade — e.g. "@Thermograph Phoenix". Reuses
|
# @mention it (or DM it) with a city grade — e.g. "@Thermograph Phoenix". Reuses
|
||||||
# THERMOGRAPH_DISCORD_BOT_TOKEN above. Runs on the single notifier leader only, so
|
# THERMOGRAPH_DISCORD_BOT_TOKEN above. Runs on the single notifier leader only, so
|
||||||
|
|
|
||||||
|
|
@ -13,11 +13,21 @@
|
||||||
# by deploy-dev.sh. This overlay must NOT reintroduce `build:`; it only relaxes
|
# by deploy-dev.sh. This overlay must NOT reintroduce `build:`; it only relaxes
|
||||||
# resource caps and LAN-exposes a port, same as the monorepo overlay did.
|
# resource caps and LAN-exposes a port, same as the monorepo overlay did.
|
||||||
#
|
#
|
||||||
# Differences from the prod stack (unchanged intent from the monorepo overlay):
|
# Differences from the prod stack:
|
||||||
# 1. backend is published on ALL interfaces (0.0.0.0:8137), not loopback, so
|
# 1. backend is published on ${DEV_BIND_ADDR}, defaulting to LOOPBACK.
|
||||||
# phones and other devices on the Wi-Fi can reach the dev server directly --
|
#
|
||||||
# dev has no Caddy in front (prod does, which is why the base file binds
|
# This used to be a flat 0.0.0.0:8137, from when dev ran on the operator's
|
||||||
# 127.0.0.1 only).
|
# LAN box and the point was for phones on the Wi-Fi to reach it. Dev now
|
||||||
|
# runs on vps1, a public VPS, where 0.0.0.0 would publish whatever
|
||||||
|
# unreviewed branch is in flight to the entire internet — with no Caddy, no
|
||||||
|
# TLS and no auth in front of it.
|
||||||
|
#
|
||||||
|
# So the default is 127.0.0.1 — safe anywhere, including a laptop running
|
||||||
|
# `make dev-up` — and on vps1 deploy/env-topology.sh keeps it there. Caddy
|
||||||
|
# fronts the stack at dev.thermograph.org with TLS, HTTP basic auth and
|
||||||
|
# X-Robots-Tag: noindex (deploy/Caddyfile.vps1). Loopback is what makes that
|
||||||
|
# guard total: the site block is the only route in, so nothing reaches the
|
||||||
|
# app without the password — not even from the WireGuard mesh.
|
||||||
# 2. frontend's port publish is dropped entirely -- dev has no Caddy to reach it
|
# 2. frontend's port publish is dropped entirely -- dev has no Caddy to reach it
|
||||||
# directly, so it stays compose-internal-only, reached solely through
|
# directly, so it stays compose-internal-only, reached solely through
|
||||||
# backend's own reverse-proxy fallback (THERMOGRAPH_FRONTEND_BASE_INTERNAL,
|
# backend's own reverse-proxy fallback (THERMOGRAPH_FRONTEND_BASE_INTERNAL,
|
||||||
|
|
@ -27,18 +37,21 @@
|
||||||
# prod's backend=4 / frontend=2 / db=2 allocation. `!reset` drops the base
|
# prod's backend=4 / frontend=2 / db=2 allocation. `!reset` drops the base
|
||||||
# value (both the top-level `cpus:` and the Swarm-style `deploy.resources`
|
# value (both the top-level `cpus:` and the Swarm-style `deploy.resources`
|
||||||
# block the base file carries for parity).
|
# block the base file carries for parity).
|
||||||
# 4. backend/daemon/lake get a second env_file entry pointing at a copy of the
|
# 4. backend/daemon/lake get a second env_file entry pointing at an OPTIONAL
|
||||||
# render under $APP_DIR (deploy-dev.sh sets THERMOGRAPH_SECRETS_ENV_FILE_MIRROR
|
# copy of the render under $APP_DIR (`required: false`, so it is a no-op
|
||||||
# to produce it). This box's Docker is the snap package, confined to $HOME --
|
# when absent). It exists for snap-packaged Docker, which is confined to
|
||||||
# it can't see /etc/thermograph.env at all (not a permissions error, it just
|
# $HOME and cannot see /etc/thermograph.env at all — not a permissions
|
||||||
# doesn't exist as far as snap-confined Docker is concerned), so the base
|
# error; the file simply does not exist as far as snap-confined Docker is
|
||||||
# file's env_file entry silently loads nothing here. compose appends env_file
|
# concerned, so the base file's env_file entry silently loads nothing.
|
||||||
# lists across overlays (last-wins on duplicate keys), so this is additive:
|
# deploy-dev.sh now produces the mirror ONLY when it detects snap Docker,
|
||||||
# prod/beta never set the mirror var, so they only ever get the base entry.
|
# which vps1 (ordinary apt Docker) is not — there, dev reads
|
||||||
|
# /etc/thermograph.env like beta and prod do, and no plaintext copy of the
|
||||||
|
# render is written into the checkout. compose appends env_file lists across
|
||||||
|
# overlays (last-wins on duplicate keys), so this stays additive.
|
||||||
services:
|
services:
|
||||||
backend:
|
backend:
|
||||||
ports: !override
|
ports: !override
|
||||||
- "8137:8137"
|
- "${DEV_BIND_ADDR:-127.0.0.1}:8137:8137"
|
||||||
cpus: !reset null
|
cpus: !reset null
|
||||||
deploy: !reset null
|
deploy: !reset null
|
||||||
env_file:
|
env_file:
|
||||||
|
|
|
||||||
300
infra/openbao/README.md
Normal file
300
infra/openbao/README.md
Normal file
|
|
@ -0,0 +1,300 @@
|
||||||
|
# OpenBao — the secret store, phase 1
|
||||||
|
|
||||||
|
**Status: dormant.** Every file here is committed but nothing is cut over. SOPS is
|
||||||
|
still authoritative for dev, beta and prod, because `TG_SECRETS_BACKEND` defaults to
|
||||||
|
`sops` for all three in `infra/deploy/env-topology.sh`. Flipping an environment is a
|
||||||
|
one-line reviewed change, and it should not happen until `verify-parity.sh` passes.
|
||||||
|
|
||||||
|
Verified against **OpenBao v2.6.1** (2026-07-22). The binary is `bao`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Why do this at all
|
||||||
|
|
||||||
|
Not for encryption, and not for audit — though the audit log is a real gain. The
|
||||||
|
decisive reason is one specific failure class that a file-based vault cannot close.
|
||||||
|
|
||||||
|
`infra/CLAUDE.md` states the rule: *"vps1 must never hold prod credentials. It's the
|
||||||
|
box that runs Forgejo, its CI runner, and dev's unreviewed branch — the opposite of an
|
||||||
|
isolation boundary."*
|
||||||
|
|
||||||
|
Today that rule is enforced by a **shell flag** — `THERMOGRAPH_SECRETS_SKIP_COMMON=1`
|
||||||
|
— set by the *caller*, at three separate call sites (`env-topology.sh:196` →
|
||||||
|
`deploy.sh:67`, `deploy-dev.sh:86`, `infra-sync.yml:93-95`). The renderer itself does
|
||||||
|
not know that `env=dev` means never-common. A fourth call site, or one by-hand
|
||||||
|
`render_thermograph_secrets /opt/thermograph-dev/infra dev`, writes both S3 keypairs
|
||||||
|
(one read-write on the bucket holding prod's database backups), the VAPID private key
|
||||||
|
that signs Web Push to real subscribers, `REGISTRY_TOKEN`, and the IndexNow and metrics
|
||||||
|
tokens onto the CI-runner box — **and exits 0**.
|
||||||
|
|
||||||
|
Under `policies/tg-host-dev.hcl` that is not possible to get wrong. dev's identity
|
||||||
|
cannot read `thermograph/data/common`. A misconfigured caller gets a 403 instead of a
|
||||||
|
silent success. The failure class is *gone*, not guarded. That is what this migration
|
||||||
|
buys, and it is worth the cost of running a stateful service.
|
||||||
|
|
||||||
|
The estate has already hardened this once by hand (commit `7583258`, "infra-sync:
|
||||||
|
refuse to render dev's vault onto a host that is not dev"). That is the shape of a
|
||||||
|
problem that wants an ACL, not another guard.
|
||||||
|
|
||||||
|
### What it does *not* buy — stated plainly
|
||||||
|
|
||||||
|
- **Rotation of provider-issued credentials.** Roughly half the credential count is
|
||||||
|
Discord, Contabo S3, the Forgejo registry token, VAPID, IndexNow. OpenBao cannot
|
||||||
|
rotate any of them; they stay static KV entries. Contabo Object Storage is
|
||||||
|
S3-compatible but exposes no IAM API, so the AWS secrets engine cannot issue against
|
||||||
|
it either.
|
||||||
|
- **Dynamic database credentials.** Inapplicable here for two independent reasons: the
|
||||||
|
apps read `THERMOGRAPH_DATABASE_URL` once at import and hold a pool, so a TTL'd
|
||||||
|
credential kills the pool at expiry; and there is no reload path anywhere in the
|
||||||
|
codebase — no SIGHUP handler, no config re-read.
|
||||||
|
- **Better review than git.** This is a real regression. Today a secret change is a
|
||||||
|
PR with a diff, backstopped by `secrets-guard` CI. Under OpenBao it is an
|
||||||
|
out-of-band API call with no diff and no review. Mitigated by committing a key-name
|
||||||
|
manifest so CI can still assert no key vanished, plus the audit log and
|
||||||
|
`bao kv metadata` as the change record — but not fully.
|
||||||
|
- **Fewer moving parts.** SOPS needs no server. This adds a stateful service to keep
|
||||||
|
alive, upgrade, back up and TLS-rotate, for ~40 secrets that change rarely.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Shape of the deployment
|
||||||
|
|
||||||
|
| Decision | Choice | Why |
|
||||||
|
|---|---|---|
|
||||||
|
| Host | **vps2 only** | vps1 must never hold prod credentials, and with a static seal the box also holds the key that decrypts everything. Cost: vps1 needs vps2 up to deploy dev — acceptable, since when vps2 is down prod is down anyway. |
|
||||||
|
| Process | **native systemd**, not a container | A Swarm service is circular (`deploy-stack.sh` renders secrets to deploy the stack). A plain `docker run` makes the vault depend on the daemon deploys restart, and is one `prune -a` from gone. Native = one dependency, the disk. |
|
||||||
|
| Storage | **raft**, single node | 2.6.0 **deprecated the `file` backend** for removal in 2.7.0. Raft also has the only backup primitive (`operator raft snapshot save`). A *two*-node raft would be worse than one: quorum of two means losing either node loses writes. |
|
||||||
|
| Seal | **static auto-unseal** | See below. |
|
||||||
|
| TLS | self-signed, 10y, on disk | Must **not** come from Caddy/ACME — that would put DNS and the public internet in the boot chain of the secret store. |
|
||||||
|
| Auth | AppRole per environment, CIDR-bound | 5-minute tokens; secret-ids that only work from the right mesh address. |
|
||||||
|
| Consumption | keep rendering a dotenv file | Preserves every existing seam and keeps OpenBao a *deploy-time* dependency, never a runtime one. |
|
||||||
|
|
||||||
|
### The unseal decision, which is the crux
|
||||||
|
|
||||||
|
**Static seal**, key at `/etc/openbao/unseal.key` (0400), plus an off-box copy.
|
||||||
|
|
||||||
|
Shamir (`-key-shares=5 -key-threshold=3`) means the vault is sealed after every process
|
||||||
|
restart — reboot, package upgrade, OOM kill — and a human must type three shares.
|
||||||
|
`render-secrets.sh` is on the deploy path for all three environments, so a sealed vault
|
||||||
|
blocks every deploy and every hotfix. With one operator the N-of-M threshold is
|
||||||
|
theatre: one person holds all the shares, so it buys nothing against the real threat
|
||||||
|
while costing an availability property the estate has today for free. A Contabo reboot
|
||||||
|
currently needs no human at all.
|
||||||
|
|
||||||
|
OpenBao's docs hedge static seal — *"carefully evaluate"* — but that caveat is aimed at
|
||||||
|
multi-tenant enterprises. The argument here is that **static seal is not a downgrade
|
||||||
|
from the status quo**: `/etc/thermograph/age.key` is already one 0400 file per host
|
||||||
|
that decrypts everything forever with no audit trail. A static seal key is the
|
||||||
|
identical trust model. What changes is everything layered above it.
|
||||||
|
|
||||||
|
**The one new single point of failure:** lose that key and the raft snapshots are
|
||||||
|
unrestorable. It must exist in ≥2 places, one not on vps2. Hard operator obligation.
|
||||||
|
|
||||||
|
A `transit` seal against a second OpenBao on vps1 is genuinely better — vps2's disk
|
||||||
|
would no longer contain the unseal key — but the vps1 unsealer needs unsealing too, so
|
||||||
|
the circularity moves rather than dissolves, and it adds "vps1 must be up before vps2's
|
||||||
|
vault unseals" as a new failure mode. Revisit once the primary migration is boring.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
thermograph/data/common the 16 shared beta+prod values (= common.yaml)
|
||||||
|
thermograph/data/env/prod prod's 16 (= prod.yaml)
|
||||||
|
thermograph/data/env/beta beta's 8 (= beta.yaml)
|
||||||
|
thermograph/data/env/dev dev's 12 — inherits NOTHING (= dev.yaml)
|
||||||
|
thermograph/data/centralis/prod Centralis' 9 (= centralis.prod.yaml)
|
||||||
|
thermograph/data/ops/backup S3 creds for ops-cron + the backup age recipient
|
||||||
|
thermograph/data/legacy/age the age PRIVATE key — see "the age key survives"
|
||||||
|
```
|
||||||
|
|
||||||
|
Inheritance lives in the render order (`common` then `env/<name>`, last-wins);
|
||||||
|
isolation lives in the policy. Today both live in one shell variable.
|
||||||
|
|
||||||
|
**`centralis/<env>` renders differently from everything above it, and must.** The
|
||||||
|
app stack's `/etc/thermograph.env` is raw unquoted `KEY=value`; `/etc/centralis.env`
|
||||||
|
is consumed by `set -a && . /etc/centralis.env`, i.e. bash's full expansion
|
||||||
|
pipeline, so it is emitted as POSIX single-quoted assignments and then verified by
|
||||||
|
sourcing it in a clean shell and comparing every value back before the file is
|
||||||
|
written. `CENTRALIS_TOKENS` is JSON and cannot survive the unquoted form: bash
|
||||||
|
strips its quotes, Centralis fails closed on the malformed result and serves a
|
||||||
|
single `shared` identity, which is indistinguishable from the file never having
|
||||||
|
been written. That is the 2026-07-24 incident, and the round-trip check exists so a
|
||||||
|
render carrying it cannot reach `/etc`.
|
||||||
|
|
||||||
|
Consequence for seeding: `seed-from-sops.sh` applies the raw-dotenv rules (no shell
|
||||||
|
metacharacters, no newlines) only to `common` and `env/*`. Applying them to
|
||||||
|
`centralis/*` would reject data that renders perfectly well — the validator matches
|
||||||
|
the renderer that will consume the path, not a single global rule.
|
||||||
|
|
||||||
|
`common` is a top-level path rather than `env/common` on purpose: it makes the dev deny
|
||||||
|
rule expressible as exactly one path, with no wildcard over `env/*` able to
|
||||||
|
accidentally re-include it.
|
||||||
|
|
||||||
|
**~20 of the 61 keys are not secrets** (`PORT`, `WORKERS`, `APP_CPUS`, `TIMESCALEDB_TAG`,
|
||||||
|
`THERMOGRAPH_BASE_URL`, the Discord channel IDs, the VAPID *public* key, …). They stay
|
||||||
|
in the vault **for the cutover**, because byte-identical parity is the entire safety
|
||||||
|
property and splitting them would destroy it. Moving pure config back into the repo is
|
||||||
|
a clean follow-on that shrinks the vault-outage blast radius.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The age key survives this migration
|
||||||
|
|
||||||
|
**This is the trap that would cause silent data loss.**
|
||||||
|
|
||||||
|
`ops-cron.yml:142` and `:197` stream every off-box Postgres and Forgejo dump through
|
||||||
|
`pg_dump | age -r <recipient> | rclone rcat`, with 30-day S3 retention. Restore uses
|
||||||
|
`/etc/thermograph/age.key`. So the age keypair is not just the SOPS transport — **it is
|
||||||
|
the backup encryption key.**
|
||||||
|
|
||||||
|
Deleting it along with the SOPS vault files would destroy up to 30 days of database
|
||||||
|
recoverability, and nothing would notice until someone attempted a restore.
|
||||||
|
|
||||||
|
So: keep the age private key at `thermograph/data/legacy/age` *and* in the password
|
||||||
|
manager, and keep it on disk until either the newest age-encrypted object in S3 has
|
||||||
|
aged out, or the backup path is re-pointed at a dedicated backup recipient. Retiring
|
||||||
|
SOPS and retiring age are two different projects.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Runbook
|
||||||
|
|
||||||
|
### Stand it up (once, by the operator, on vps2)
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo bash infra/openbao/bootstrap.sh # binary, TLS, seal key, ufw, unit, init
|
||||||
|
# custody the recovery keys + /etc/openbao/unseal.key OFF the box, then:
|
||||||
|
export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=/etc/thermograph/openbao-ca.crt
|
||||||
|
export BAO_TOKEN=<root token>
|
||||||
|
sudo -E bash infra/openbao/bootstrap-policies.sh # mount, policies, approles
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Do not revoke the root token here.** OpenBao 2.6.0 replaced the unauthenticated
|
||||||
|
> `/sys/generate-root` with an authenticated `/sys/generate-root-token`
|
||||||
|
> (HCSEC-2026-08), and `bao operator generate-root` now targets the new one — so
|
||||||
|
> minting a root token requires a token you already have. **The recovery keys are not
|
||||||
|
> a way back in**; they rekey, they do not authenticate. Unless
|
||||||
|
> `disable_unauthed_generate_root_endpoints = false` is set in `config.hcl`, revoking
|
||||||
|
> the last root token locks you out of an otherwise healthy, unsealed vault, and the
|
||||||
|
> only remedy is re-initialising from scratch. Revoke only once a second admin
|
||||||
|
> identity exists.
|
||||||
|
|
||||||
|
Use `/etc/thermograph/openbao-ca.crt` (mode `0444`) rather than
|
||||||
|
`/etc/openbao/tls/bao.crt` — same certificate, but `/etc/openbao/tls/` is `0700
|
||||||
|
openbao`, so only root can read the latter. Without `BAO_CACERT` every command fails
|
||||||
|
with `certificate signed by unknown authority`.
|
||||||
|
|
||||||
|
### Seed and prove (on vps2 — the age key is already there)
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export SOPS_AGE_KEY_FILE=/etc/thermograph/age.key
|
||||||
|
infra/openbao/seed-from-sops.sh --dry-run # key names + counts, writes nothing
|
||||||
|
infra/openbao/seed-from-sops.sh --all
|
||||||
|
```
|
||||||
|
|
||||||
|
Then prove parity — **not `--all` from one box.** Each AppRole is `secret_id_bound_cidrs`
|
||||||
|
to the host that legitimately renders it, so an environment can only be verified from
|
||||||
|
its own host:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# on vps2
|
||||||
|
infra/openbao/verify-parity.sh --env prod # expect 32 keys
|
||||||
|
infra/openbao/verify-parity.sh --env beta # expect 24 keys
|
||||||
|
# on vps1
|
||||||
|
cd /opt/thermograph-dev && infra/openbao/verify-parity.sh --env dev # expect 12 keys
|
||||||
|
```
|
||||||
|
|
||||||
|
Beta needs no special handling: `env-topology.sh` derives `TG_BAO_APPROLE` per
|
||||||
|
environment, so beta authenticates as `tg-beta` rather than falling back to prod's
|
||||||
|
credentials and being denied its own path by `tg-host-prod.hcl`.
|
||||||
|
|
||||||
|
`seed-from-sops.sh` reads every production secret in plaintext. Per `infra/CLAUDE.md`
|
||||||
|
the equivalent `seed-from-live.sh` is explicitly *not for an agent to run*; this
|
||||||
|
inherits that rule.
|
||||||
|
|
||||||
|
### Cut an environment over
|
||||||
|
|
||||||
|
One PR per hop, following the estate's own promotion model.
|
||||||
|
|
||||||
|
```diff
|
||||||
|
dev)
|
||||||
|
- TG_SECRETS_BACKEND=sops
|
||||||
|
+ TG_SECRETS_BACKEND=openbao
|
||||||
|
```
|
||||||
|
|
||||||
|
Order: **dev → beta → prod**, with `verify-parity.sh` green throughout. Recommended
|
||||||
|
gate before prod: **7 consecutive green parity runs on all three environments**, run
|
||||||
|
nightly from `ops-cron.yml` over SSH (which gives continuous evidence without granting
|
||||||
|
CI any vault access).
|
||||||
|
|
||||||
|
`TG_SECRETS_BACKEND=sops` remains a working two-way door for the whole period. Keep the
|
||||||
|
SOPS path for a full release cycle after prod flips — it is the best mitigation
|
||||||
|
available for anything unforeseen.
|
||||||
|
|
||||||
|
### Rollback
|
||||||
|
|
||||||
|
Revert the one-line PR and redeploy. The SOPS path is untouched by this work — verified
|
||||||
|
by rendering dev (12 keys) and prod (32 keys) through it after the dispatch was added,
|
||||||
|
matching the live hosts exactly.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Failure modes
|
||||||
|
|
||||||
|
| Failure | Effect | Mitigation |
|
||||||
|
|---|---|---|
|
||||||
|
| Vault down/sealed at deploy | Render returns 1, deploy aborts, **running stack unaffected** — temp-then-install means `/etc/thermograph.env` is never truncated | `Restart=always` + static seal so a reboot self-heals; alert on `/v1/sys/health`; `TG_SECRETS_BACKEND=sops` is a working revert |
|
||||||
|
| **Audit device wedges** | ⚠️ OpenBao **stops answering requests entirely** when no enabled audit device can record them — a full `/var` on vps2 blocks every deploy | Two devices (file + syslog); logrotate signals **SIGHUP** or bao writes to an unlinked inode; disk alert on vps2 |
|
||||||
|
| Static seal key lost | Raft snapshots unrestorable | ≥2 copies, one off-box. The one new SPOF |
|
||||||
|
| Raft corruption / disk loss | Total vault loss | Nightly `operator raft snapshot save`, age-encrypted to S3 beside the DB dumps. **Verify a restore end-to-end before any consumer depends on it** |
|
||||||
|
| Cold boot of vps2 | Nothing needs the vault | Swarm restarts from persisted specs; `stack.env` / `thermograph.env` / `centralis.env` all persist. **This property exists because we render a file, and would be destroyed by app-native reads** |
|
||||||
|
| Secret-id leak | Useless off the mesh | `secret_id_bound_cidrs`; revoke by accessor without knowing the value |
|
||||||
|
| beta credential reaches prod | Cross-env compromise | Separate AppRoles, separate policies, secret-ids owned by the separate deploy users (`agent` vs `deploy`) — a boundary that does **not** exist today, since both currently reach the same root-owned age key. Honest limit: root on vps2 defeats it, exactly as it defeats the single age key now |
|
||||||
|
| Upgrade to 2.7.0 | Removes the `file` backend and built-in cloud KMS seals | Already avoided by choosing raft + static. Pin the version |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deliberately out of scope
|
||||||
|
|
||||||
|
For ~40 credentials and one operator the real risk is over-engineering. Not adopted:
|
||||||
|
HA/multi-node raft, namespaces, dynamic database credentials, `bao agent`, cert auth,
|
||||||
|
PKI, identity groups, transit seal (revisit later), OIDC for the operator.
|
||||||
|
|
||||||
|
**OIDC for CI is out of scope for a reason worth recording.** Forgejo Actions *does*
|
||||||
|
support OIDC — shipped in **Forgejo v15.0**, needing Runner > v12.5.0, enabled via
|
||||||
|
`enable-openid-connect: true` rather than GitHub's `permissions: id-token: write`. But
|
||||||
|
this estate runs `codeberg.org/forgejo/forgejo:9-rootless`
|
||||||
|
(`infra/deploy/forgejo/docker-stack.yml:59`). That is a six-major-version upgrade, and
|
||||||
|
coupling it to this migration would mean two large independent migrations at once with
|
||||||
|
no way to tell which one broke. Revisit after Forgejo reaches v15; then
|
||||||
|
`role_type=jwt` with `bound_claims` removes the last CI-side bearer credential.
|
||||||
|
|
||||||
|
CI keeps its SSH keys and `REGISTRY_TOKEN` as Forgejo secrets. **CI gets no vault
|
||||||
|
access at all** — the SSH-in architecture already means the *host* renders, never the
|
||||||
|
runner, which is the least-privilege topology and should be preserved rather than
|
||||||
|
"upgraded".
|
||||||
|
|
||||||
|
### Secret zero, honestly
|
||||||
|
|
||||||
|
Nothing eliminates it; you choose where it sits and how little of it there is. After
|
||||||
|
this phase the chain terminates at **9 Forgejo Actions secrets + 1 static seal key + 4
|
||||||
|
AppRole secret-ids**, versus today's **13 CI secrets + 1 omnipotent age key per host**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verification gaps to close on the box
|
||||||
|
|
||||||
|
Flagged rather than guessed, because these could not be checked from a workstation:
|
||||||
|
|
||||||
|
1. Whether a raft snapshot is restorable under a *different* seal key. Docs don't say.
|
||||||
|
**Test in phase 0, before anything depends on it.**
|
||||||
|
2. Exact `-wrap-ttl` flag spelling on `bao write` — documented in the response-wrapping
|
||||||
|
concept page but absent from the commands index. Check `bao write -h`.
|
||||||
|
3. Debian/Ubuntu package repo URL. The install docs claim `.deb` packages exist but
|
||||||
|
give no repository; `bootstrap.sh` uses the GitHub release tarball with checksum
|
||||||
|
verification, which is fine and also fixes a standing weakness — `provision-secrets.sh`
|
||||||
|
installs sops and age with a bare `sudo curl` and no verification at all.
|
||||||
|
4. Vault-era naming persists inside OpenBao config (`vault { }` stanza,
|
||||||
|
`X-Vault-Wrap-TTL` header). Expect it; don't "fix" it.
|
||||||
130
infra/openbao/bootstrap-policies.sh
Executable file
130
infra/openbao/bootstrap-policies.sh
Executable file
|
|
@ -0,0 +1,130 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Second half of bootstrap: KV mount, policies, AppRoles, host credentials.
|
||||||
|
# Run on vps2 with BAO_TOKEN set to a root token. Idempotent — safe to re-run.
|
||||||
|
#
|
||||||
|
# export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=/etc/openbao/tls/bao.crt
|
||||||
|
# export BAO_TOKEN=<root token>
|
||||||
|
# sudo -E bash infra/openbao/bootstrap-policies.sh
|
||||||
|
#
|
||||||
|
# Split from bootstrap.sh because everything here is safely automatable: none of it
|
||||||
|
# produces a credential a human has to custody. The one thing that does — the recovery
|
||||||
|
# material from `bao operator init` — is in bootstrap.sh and stops for the operator.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
MOUNT="${THERMOGRAPH_BAO_MOUNT:-thermograph}"
|
||||||
|
SELF_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
POLICY_DIR="$SELF_DIR/policies"
|
||||||
|
|
||||||
|
: "${BAO_TOKEN:?BAO_TOKEN must be set to a root token}"
|
||||||
|
: "${BAO_ADDR:=https://127.0.0.1:8200}"
|
||||||
|
export BAO_ADDR
|
||||||
|
command -v bao >/dev/null 2>&1 || { echo "!! bao not on PATH" >&2; exit 1; }
|
||||||
|
|
||||||
|
echo "==> KV v2 mount at ${MOUNT}/"
|
||||||
|
if bao secrets list -format=json | grep -q "\"${MOUNT}/\""; then
|
||||||
|
echo " already mounted"
|
||||||
|
else
|
||||||
|
bao secrets enable -path="$MOUNT" -version=2 kv
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> policies"
|
||||||
|
for p in "$POLICY_DIR"/*.hcl; do
|
||||||
|
name="$(basename "$p" .hcl)"
|
||||||
|
bao policy write "$name" "$p" >/dev/null
|
||||||
|
echo " wrote $name"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "==> approle auth"
|
||||||
|
bao auth list -format=json | grep -q '"approle/"' || bao auth enable approle
|
||||||
|
|
||||||
|
# One role per environment, CIDR-bound to the host that legitimately renders it.
|
||||||
|
#
|
||||||
|
# secret_id_ttl=0 (never expires) is deliberate: a secret-id that silently lapses
|
||||||
|
# mid-quarter is a self-inflicted outage. The short lifetime lives on the TOKEN
|
||||||
|
# instead — it only has to outlive one render.
|
||||||
|
#
|
||||||
|
# The CIDR binding is the part that makes a leaked secret-id close to useless: it is
|
||||||
|
# only accepted from the correct WireGuard address.
|
||||||
|
add_role() {
|
||||||
|
local role="$1" policy="$2" cidr="$3"
|
||||||
|
bao write "auth/approle/role/${role}" \
|
||||||
|
token_policies="$policy" \
|
||||||
|
secret_id_bound_cidrs="$cidr" \
|
||||||
|
token_bound_cidrs="$cidr" \
|
||||||
|
secret_id_ttl=0 secret_id_num_uses=0 \
|
||||||
|
token_ttl=5m token_max_ttl=15m token_num_uses=10 >/dev/null
|
||||||
|
echo " role ${role} -> ${policy} (bound ${cidr})"
|
||||||
|
}
|
||||||
|
|
||||||
|
add_role tg-prod tg-host-prod 10.10.0.1/32
|
||||||
|
add_role tg-beta tg-host-beta 10.10.0.1/32
|
||||||
|
add_role tg-dev tg-host-dev 10.10.0.2/32
|
||||||
|
add_role tg-centralis tg-centralis 10.10.0.1/32
|
||||||
|
|
||||||
|
# Install the local (vps2) credentials. prod and beta both render on this box.
|
||||||
|
#
|
||||||
|
# Both files are owned by `agent`, because that is the single identity both
|
||||||
|
# environments actually deploy as: env-topology.sh sets TG_SSH_TARGET=agent@... for
|
||||||
|
# beta and for prod, deploy.yml uses one VPS2_SSH_USER for both, and vps2 has no
|
||||||
|
# `deploy` user at all (only `ubuntu` and `agent`). An earlier revision chowned beta's
|
||||||
|
# file to `deploy:deploy` on the theory that beta and prod deploy as different users;
|
||||||
|
# they do not, and the chown simply failed.
|
||||||
|
#
|
||||||
|
# So be accurate about what separate credentials buy on this box: NOT deploy-user
|
||||||
|
# isolation — there is one deploy user, and it can read both files. What they buy is
|
||||||
|
# per-environment ATTRIBUTION in the audit log, and a policy boundary that stops a
|
||||||
|
# *mistake* crossing between environments: a beta render authenticating with beta's
|
||||||
|
# secret-id is refused thermograph/data/env/prod by tg-host-beta.hcl, so a wrong
|
||||||
|
# THERMOGRAPH_ENV fails closed instead of quietly rendering the other environment's
|
||||||
|
# database password. Real deploy-user isolation would need a `deploy` user on vps2 and
|
||||||
|
# a separate SSH credential for beta in deploy.yml — a larger change than this script.
|
||||||
|
install_creds() {
|
||||||
|
local role="$1" dest="$2" owner="$3"
|
||||||
|
local rid sid
|
||||||
|
rid=$(bao read -field=role_id "auth/approle/role/${role}/role-id")
|
||||||
|
sid=$(bao write -f -field=secret_id "auth/approle/role/${role}/secret-id")
|
||||||
|
# umask before creation, so the file is never briefly world-readable.
|
||||||
|
( umask 077; printf '%s\n%s\n' "$rid" "$sid" > "$dest" )
|
||||||
|
# Abort the function if chown fails, and remove the half-installed file. The call
|
||||||
|
# site wraps this in `|| echo`, which suppresses `set -e` for everything inside —
|
||||||
|
# so without this check a failed chown falls straight through to chmod and prints a
|
||||||
|
# line asserting an ownership that was never applied. A root-owned credential the
|
||||||
|
# renderer cannot read, reported as installed, is worse than no credential at all:
|
||||||
|
# it fails at deploy time instead of here.
|
||||||
|
if ! chown "$owner" "$dest"; then
|
||||||
|
rm -f "$dest"
|
||||||
|
echo " !! chown $owner failed for $dest — removed, nothing installed" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
chmod 0400 "$dest"
|
||||||
|
echo " installed $dest (0400 $owner)"
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ "$(id -u)" = 0 ]; then
|
||||||
|
install -d -o root -g root -m 0755 /etc/thermograph
|
||||||
|
install_creds tg-prod /etc/thermograph/openbao-approle "agent:agent"
|
||||||
|
install_creds tg-beta /etc/thermograph/openbao-approle-beta "agent:agent"
|
||||||
|
install_creds tg-centralis /etc/thermograph/openbao-approle-centralis "root:root"
|
||||||
|
else
|
||||||
|
echo "!! not root; skipping credential install. Re-run with sudo -E." >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat <<EOF
|
||||||
|
|
||||||
|
==> done. Policies and AppRoles are in place; the vault is still EMPTY.
|
||||||
|
|
||||||
|
dev's credentials must be installed on vps1, not here. From your own machine:
|
||||||
|
|
||||||
|
bao read -field=role_id auth/approle/role/tg-dev/role-id
|
||||||
|
bao write -f -field=secret_id auth/approle/role/tg-dev/secret-id
|
||||||
|
# then, on vps1, write both lines to /etc/thermograph/openbao-approle (0400 agent)
|
||||||
|
|
||||||
|
Prefer response wrapping so the secret-id never lands in shell history or a log:
|
||||||
|
bao write -f -wrap-ttl=5m auth/approle/role/tg-dev/secret-id # gives a wrapping token
|
||||||
|
# on vps1: bao unwrap <token> (single-use: a failed unwrap means interception)
|
||||||
|
|
||||||
|
Next: seed, then verify parity. Neither cuts anything over.
|
||||||
|
infra/openbao/seed-from-sops.sh --dry-run
|
||||||
|
infra/openbao/seed-from-sops.sh --all
|
||||||
|
infra/openbao/verify-parity.sh --all
|
||||||
|
EOF
|
||||||
228
infra/openbao/bootstrap.sh
Executable file
228
infra/openbao/bootstrap.sh
Executable file
|
|
@ -0,0 +1,228 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Bring up OpenBao on vps2 from nothing. Run ONCE, as root, ON vps2, BY THE OPERATOR.
|
||||||
|
#
|
||||||
|
# sudo bash infra/openbao/bootstrap.sh
|
||||||
|
#
|
||||||
|
# ============================================================================
|
||||||
|
# WHY THIS IS NOT AUTOMATED, AND MUST NOT BE
|
||||||
|
# ============================================================================
|
||||||
|
# `bao operator init` mints the credentials that become the new root of trust for the
|
||||||
|
# entire estate: the recovery keys and the initial root token. Those have to be
|
||||||
|
# CUSTODIED BY A HUMAN. Any place a script could put them — a file on the box, the
|
||||||
|
# repo, a CI log, an agent transcript — is worse custody than the age key this
|
||||||
|
# migration is meant to improve on.
|
||||||
|
#
|
||||||
|
# So this script does everything up to and including init, prints the recovery
|
||||||
|
# material ONCE to the operator's terminal, and then stops and tells you what to do
|
||||||
|
# with it. It deliberately does not store it, mail it, or push it anywhere.
|
||||||
|
#
|
||||||
|
# Everything AFTER custody (policies, AppRoles, seeding, parity checks) is automated,
|
||||||
|
# because none of it produces a credential a human needs to hold.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
BAO_VERSION="${BAO_VERSION:-2.6.1}"
|
||||||
|
MESH_ADDR="${MESH_ADDR:-10.10.0.1}"
|
||||||
|
SELF_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
[ "$(id -u)" = 0 ] || { echo "!! run as root (sudo bash $0)" >&2; exit 1; }
|
||||||
|
|
||||||
|
hostname_now="$(hostname)"
|
||||||
|
case "$hostname_now" in
|
||||||
|
vmi3453260) : ;; # vps2
|
||||||
|
*)
|
||||||
|
# Refuse to bootstrap the secret store on the wrong box. Putting it on vps1 would
|
||||||
|
# invert the estate's central security decision: infra/CLAUDE.md states "vps1 must
|
||||||
|
# never hold prod credentials. It's the box that runs Forgejo, its CI runner, and
|
||||||
|
# dev's unreviewed branch — the opposite of an isolation boundary." With a static
|
||||||
|
# seal, the box also holds the key that decrypts everything.
|
||||||
|
#
|
||||||
|
# This mirrors the guard infra-sync.yml already has (commit 7583258, "refuse to
|
||||||
|
# render dev's vault onto a host that is not dev").
|
||||||
|
echo "!! this host is '$hostname_now', not vps2 (vmi3453260)." >&2
|
||||||
|
echo "!! OpenBao belongs on vps2. Refusing." >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
echo "==> 1/7 install the bao binary (v${BAO_VERSION})"
|
||||||
|
if ! command -v bao >/dev/null 2>&1; then
|
||||||
|
# NOTE: checksum verification. provision-secrets.sh installs sops and age with a bare
|
||||||
|
# `sudo curl` and no verification at all, which is a standing weakness (a compromised
|
||||||
|
# release URL yields a root binary that sees every plaintext). Do not copy that here.
|
||||||
|
# The checksums file is signed by the release; verify before installing.
|
||||||
|
tmpd="$(mktemp -d)"
|
||||||
|
base="https://github.com/openbao/openbao/releases/download/v${BAO_VERSION}"
|
||||||
|
# Asset names are `openbao_<ver>_...`, and the checksums file is `checksums.txt`.
|
||||||
|
# The previous `bao_<ver>_...` / `bao_<ver>_SHA256SUMS` names both 404.
|
||||||
|
tarball="openbao_${BAO_VERSION}_linux_amd64.tar.gz"
|
||||||
|
curl -fsSL -o "$tmpd/$tarball" "${base}/${tarball}"
|
||||||
|
curl -fsSL -o "$tmpd/checksums" "${base}/checksums.txt"
|
||||||
|
# Anchor to end-of-line and save under the real asset name: checksums.txt also lists
|
||||||
|
# <tarball>.sbom.json, and `sha256sum -c` resolves each line by the filename IN it,
|
||||||
|
# so an unanchored match fails on a file that was never fetched.
|
||||||
|
( cd "$tmpd" && grep " ${tarball}\$" checksums | sha256sum -c - ) || {
|
||||||
|
echo "!! checksum mismatch on the bao tarball; refusing to install" >&2
|
||||||
|
rm -rf "$tmpd"; exit 1
|
||||||
|
}
|
||||||
|
tar -xzf "$tmpd/$tarball" -C "$tmpd" bao
|
||||||
|
install -m 0755 -o root -g root "$tmpd/bao" /usr/local/bin/bao
|
||||||
|
rm -rf "$tmpd"
|
||||||
|
fi
|
||||||
|
bao version
|
||||||
|
|
||||||
|
echo "==> 2/7 user, directories, TLS"
|
||||||
|
id -u openbao >/dev/null 2>&1 || useradd --system --home /var/lib/openbao --shell /usr/sbin/nologin openbao
|
||||||
|
install -d -o openbao -g openbao -m 0700 /var/lib/openbao
|
||||||
|
install -d -o openbao -g openbao -m 0750 /var/log/openbao
|
||||||
|
install -d -o root -g root -m 0755 /etc/openbao
|
||||||
|
install -d -o openbao -g openbao -m 0700 /etc/openbao/tls
|
||||||
|
|
||||||
|
if [ ! -f /etc/openbao/tls/bao.crt ]; then
|
||||||
|
# Self-signed, 10 years, SAN on the mesh address. Deliberately NOT from Caddy/ACME:
|
||||||
|
# that would put DNS and the public internet in the boot chain of the secret store.
|
||||||
|
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
|
||||||
|
-keyout /etc/openbao/tls/bao.key -out /etc/openbao/tls/bao.crt \
|
||||||
|
-subj "/CN=openbao.thermograph.internal" \
|
||||||
|
-addext "subjectAltName=IP:${MESH_ADDR},IP:127.0.0.1" >/dev/null 2>&1
|
||||||
|
chown openbao:openbao /etc/openbao/tls/bao.key /etc/openbao/tls/bao.crt
|
||||||
|
chmod 0400 /etc/openbao/tls/bao.key
|
||||||
|
chmod 0444 /etc/openbao/tls/bao.crt
|
||||||
|
fi
|
||||||
|
# Consumers need the cert to trust the listener.
|
||||||
|
install -m 0444 /etc/openbao/tls/bao.crt /etc/thermograph/openbao-ca.crt
|
||||||
|
|
||||||
|
echo "==> 3/7 static seal key"
|
||||||
|
if [ ! -f /etc/openbao/unseal.key ]; then
|
||||||
|
# 32 random bytes, base64. Same protection level as /etc/thermograph/age.key.
|
||||||
|
# tr -d '\n' is load-bearing: `openssl rand -base64` appends a newline, and the
|
||||||
|
# static seal reads this file RAW. That one trailing byte makes the key neither
|
||||||
|
# 32 raw bytes nor clean base64, so bao refuses to start with
|
||||||
|
# `Error configuring seal "static": unknown encoding for AES-256 key`.
|
||||||
|
( umask 077; openssl rand -base64 32 | tr -d '\n' > /etc/openbao/unseal.key )
|
||||||
|
chown openbao:openbao /etc/openbao/unseal.key
|
||||||
|
chmod 0400 /etc/openbao/unseal.key
|
||||||
|
echo " generated /etc/openbao/unseal.key (0400 openbao)"
|
||||||
|
echo " ⚠️ COPY THIS OFF THE BOX into your password manager before proceeding."
|
||||||
|
echo " Without an off-box copy, a raft snapshot is UNRESTORABLE."
|
||||||
|
else
|
||||||
|
echo " /etc/openbao/unseal.key already present, leaving alone"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> 4/7 config + systemd unit"
|
||||||
|
install -m 0640 -o root -g openbao "$SELF_DIR/config/openbao.hcl" /etc/openbao/config.hcl
|
||||||
|
install -m 0644 -o root -g root "$SELF_DIR/openbao.service" /etc/systemd/system/openbao.service
|
||||||
|
# logrotate MUST use SIGHUP: without it bao keeps writing to an unlinked inode and the
|
||||||
|
# audit log silently stops growing — and a wedged audit device makes OpenBao stop
|
||||||
|
# answering requests entirely, which would block every deploy on the estate.
|
||||||
|
cat > /etc/logrotate.d/openbao <<'ROTATE'
|
||||||
|
/var/log/openbao/audit.log {
|
||||||
|
daily
|
||||||
|
rotate 30
|
||||||
|
compress
|
||||||
|
missingok
|
||||||
|
notifempty
|
||||||
|
create 0600 openbao openbao
|
||||||
|
postrotate
|
||||||
|
systemctl kill --signal=SIGHUP openbao.service 2>/dev/null || true
|
||||||
|
endscript
|
||||||
|
}
|
||||||
|
ROTATE
|
||||||
|
systemctl daemon-reload
|
||||||
|
systemctl enable --now openbao.service
|
||||||
|
sleep 3
|
||||||
|
systemctl is-active --quiet openbao.service || {
|
||||||
|
echo "!! openbao.service failed to start; check: journalctl -u openbao -n 50" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Mesh reachability for vps1. dev renders on vps1 and must reach this listener, but
|
||||||
|
# ufw defaults to deny(incoming) and nothing else in this tree opens the port. Without
|
||||||
|
# it, dev's render fails at cutover with a connection timeout that reads as an OpenBao
|
||||||
|
# fault rather than a firewall one. Scoped to vps1's mesh address specifically: the
|
||||||
|
# AppRole CIDR bindings are the real control, and nothing else on the mesh — the
|
||||||
|
# desktop, the phone — has any business reaching the secret store.
|
||||||
|
if command -v ufw >/dev/null 2>&1; then
|
||||||
|
if ufw status | grep -q '8200.*10\.10\.0\.2'; then
|
||||||
|
echo " ufw: 8200/tcp from 10.10.0.2 already allowed"
|
||||||
|
else
|
||||||
|
ufw allow in on wg0 from 10.10.0.2 to any port 8200 proto tcp \
|
||||||
|
comment 'vps1/dev -> openbao' >/dev/null
|
||||||
|
echo " ufw: allowed 8200/tcp on wg0 from 10.10.0.2 (vps1)"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo " !! ufw absent -- confirm 8200/tcp is reachable from 10.10.0.2 (vps1)" >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
export BAO_ADDR="https://127.0.0.1:8200"
|
||||||
|
export BAO_CACERT=/etc/openbao/tls/bao.crt
|
||||||
|
|
||||||
|
echo "==> 5/7 initialise"
|
||||||
|
if bao status -format=json 2>/dev/null | grep -q '"initialized": *true'; then
|
||||||
|
echo " already initialised, skipping"
|
||||||
|
else
|
||||||
|
echo
|
||||||
|
echo " ############################################################"
|
||||||
|
echo " # RECOVERY MATERIAL FOLLOWS. IT IS SHOWN EXACTLY ONCE. #"
|
||||||
|
echo " # Store the 3 recovery keys in 3 DIFFERENT places. #"
|
||||||
|
echo " # 2 of 3 are needed to rekey or mint a new root token. #"
|
||||||
|
echo " ############################################################"
|
||||||
|
echo
|
||||||
|
# -recovery-shares/-threshold, not -key-shares: with an auto-unseal seal, init
|
||||||
|
# yields RECOVERY keys (which authorise rekey) rather than unseal keys. 2-of-3 here
|
||||||
|
# is redundancy against loss, not separation of duty — there is one operator.
|
||||||
|
#
|
||||||
|
# These keys are NOT a route back to a root token on 2.6.x. OpenBao 2.6.0 replaced
|
||||||
|
# the unauthenticated /sys/generate-root with an authenticated /sys/generate-root-token
|
||||||
|
# (HCSEC-2026-08), and `bao operator generate-root` now targets the new one — so
|
||||||
|
# minting a root token requires a token you already have. The deprecated endpoint is
|
||||||
|
# off unless `disable_unauthed_generate_root_endpoints = false` is set in config.hcl.
|
||||||
|
# Consequence: revoking the last root token before another admin identity exists is
|
||||||
|
# a one-way door. See the ordering note in the handoff below.
|
||||||
|
bao operator init -recovery-shares=3 -recovery-threshold=2
|
||||||
|
echo
|
||||||
|
echo " Press Enter once the recovery keys AND /etc/openbao/unseal.key are"
|
||||||
|
echo " stored off this machine."
|
||||||
|
read -r _
|
||||||
|
fi
|
||||||
|
|
||||||
|
bao status || true
|
||||||
|
|
||||||
|
cat <<EOF
|
||||||
|
|
||||||
|
==> 6/7 and 7/7 need a root token, which is NOT stored anywhere by design.
|
||||||
|
|
||||||
|
Paste the initial root token printed above and run:
|
||||||
|
|
||||||
|
export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=/etc/openbao/tls/bao.crt
|
||||||
|
export BAO_TOKEN=<root token>
|
||||||
|
bash ${SELF_DIR}/bootstrap-policies.sh
|
||||||
|
|
||||||
|
KEEP THAT TOKEN until the whole sequence below is done. On 2.6.x the recovery keys
|
||||||
|
cannot mint a replacement (see the note above ${0##*/}'s init step), so a premature
|
||||||
|
\`bao token revoke -self\` locks you out of an otherwise healthy vault.
|
||||||
|
|
||||||
|
That script enables the KV mount, writes the policies, creates the AppRoles and
|
||||||
|
installs each host's credentials. Then seed — on THIS box, where the age key already
|
||||||
|
lives at /etc/thermograph/age.key. Per infra/CLAUDE.md a script that reads production
|
||||||
|
secrets is not for an agent to run:
|
||||||
|
|
||||||
|
export SOPS_AGE_KEY_FILE=/etc/thermograph/age.key
|
||||||
|
infra/openbao/seed-from-sops.sh --dry-run # check first: 16/12/8/16 keys
|
||||||
|
infra/openbao/seed-from-sops.sh --all
|
||||||
|
|
||||||
|
Then prove parity. NOT \`--all\` from one box: each AppRole is CIDR-bound to the host
|
||||||
|
that legitimately renders it, so an environment can only be verified from its own host.
|
||||||
|
|
||||||
|
# here (vps2)
|
||||||
|
infra/openbao/verify-parity.sh --env prod # expect 32 keys
|
||||||
|
infra/openbao/verify-parity.sh --env beta # expect 24 keys
|
||||||
|
# on vps1
|
||||||
|
cd /opt/thermograph-dev && infra/openbao/verify-parity.sh --env dev # expect 12 keys
|
||||||
|
|
||||||
|
Only once all three PASS, and only after you have a second admin identity that is not
|
||||||
|
the root token, revoke it: bao token revoke -self
|
||||||
|
|
||||||
|
NOTHING is cut over by any of the above. TG_SECRETS_BACKEND defaults to sops for
|
||||||
|
every environment in infra/deploy/env-topology.sh, so SOPS stays authoritative
|
||||||
|
until you flip an environment in a reviewed PR — dev first.
|
||||||
|
EOF
|
||||||
134
infra/openbao/config/openbao.hcl
Normal file
134
infra/openbao/config/openbao.hcl
Normal file
|
|
@ -0,0 +1,134 @@
|
||||||
|
// OpenBao server config for the Thermograph estate. Lives at /etc/openbao/config.hcl
|
||||||
|
// on vps2. Verified against OpenBao v2.6.1 (2026-07-22).
|
||||||
|
//
|
||||||
|
// ============================================================================
|
||||||
|
// WHY THIS RUNS AS A NATIVE SYSTEMD SERVICE AND NOT A CONTAINER
|
||||||
|
// ============================================================================
|
||||||
|
// The estate is Docker-native and this is the one thing that must not be. The rule
|
||||||
|
// is that OpenBao must not depend on anything it secures.
|
||||||
|
//
|
||||||
|
// * A Swarm service is fatal: deploy-stack.sh renders secrets in order to deploy
|
||||||
|
// the stack, so the stack cannot contain the thing that holds them.
|
||||||
|
// * A standalone `docker run --restart=always` is merely bad: it makes the vault's
|
||||||
|
// availability a function of the same Docker daemon that deploys restart, and
|
||||||
|
// leaves it one careless `docker system prune -a` from gone. The root CLAUDE.md
|
||||||
|
// already lists `prune -a` as a thing that eats the rollback image; it would
|
||||||
|
// eat this too.
|
||||||
|
//
|
||||||
|
// A native binary with Restart=always has exactly one dependency: the local disk.
|
||||||
|
// See infra/openbao/openbao.service.
|
||||||
|
//
|
||||||
|
// Corollary that is easy to get wrong: the listener cert must NOT come from the
|
||||||
|
// Caddy/ACME path. That would put DNS and the public internet in the boot chain of
|
||||||
|
// the estate's secret store. It is self-signed, long-lived, and on disk.
|
||||||
|
// ============================================================================
|
||||||
|
|
||||||
|
ui = false
|
||||||
|
cluster_name = "thermograph"
|
||||||
|
log_level = "info"
|
||||||
|
// `disable_mlock` is deliberately absent: OpenBao 2.6.1 does not recognise it and
|
||||||
|
// logs `unknown or unsupported field disable_mlock` on every start. Carrying a
|
||||||
|
// setting the server ignores only trains the operator to skim startup warnings,
|
||||||
|
// which is where a real one will eventually hide. The corresponding note about
|
||||||
|
// AmbientCapabilities=CAP_IPC_LOCK in openbao.service is moot for the same reason.
|
||||||
|
|
||||||
|
// STORAGE: raft, and this is now forced rather than chosen. OpenBao 2.6.0
|
||||||
|
// DEPRECATED the `file` backend for removal in v2.7.0, so picking `file` today buys
|
||||||
|
// a migration within a release. Raft also has the only first-class backup primitive
|
||||||
|
// (`bao operator raft snapshot save`), which is the disaster-recovery story.
|
||||||
|
//
|
||||||
|
// A single-node raft is the honest shape for a one-operator estate. Note that a
|
||||||
|
// TWO-node raft would be strictly worse than one: quorum of two means losing either
|
||||||
|
// node loses write availability. Three voters would be needed, and the desktop
|
||||||
|
// (10.10.0.3) is intermittent, so it cannot be a reliable third.
|
||||||
|
storage "raft" {
|
||||||
|
path = "/var/lib/openbao"
|
||||||
|
node_id = "vps2"
|
||||||
|
}
|
||||||
|
|
||||||
|
// Mesh-only. 10.10.0.1 is vps2's WireGuard address. There is deliberately no
|
||||||
|
// 0.0.0.0 listener and no public DNS record — vps2 is a public VPS, and reaching
|
||||||
|
// OpenBao should require being on the mesh, exactly like the Swarm control ports
|
||||||
|
// and the Loki endpoint.
|
||||||
|
listener "tcp" {
|
||||||
|
address = "10.10.0.1:8200"
|
||||||
|
tls_cert_file = "/etc/openbao/tls/bao.crt"
|
||||||
|
tls_key_file = "/etc/openbao/tls/bao.key"
|
||||||
|
}
|
||||||
|
|
||||||
|
// Loopback, so a by-hand `bao` on the box works without the mesh address.
|
||||||
|
listener "tcp" {
|
||||||
|
address = "127.0.0.1:8200"
|
||||||
|
tls_cert_file = "/etc/openbao/tls/bao.crt"
|
||||||
|
tls_key_file = "/etc/openbao/tls/bao.key"
|
||||||
|
}
|
||||||
|
|
||||||
|
api_addr = "https://10.10.0.1:8200"
|
||||||
|
cluster_addr = "https://10.10.0.1:8201"
|
||||||
|
|
||||||
|
// ============================================================================
|
||||||
|
// SEAL: static auto-unseal. This is the crux of the whole design.
|
||||||
|
// ============================================================================
|
||||||
|
// The `static` seal (OpenBao v2.4.0+) auto-unseals from a local key with no cloud
|
||||||
|
// KMS and no HSM. OpenBao's own docs hedge it — "carefully evaluate use of Static
|
||||||
|
// Key Auto Unseal" — but that caveat is written for multi-tenant enterprises.
|
||||||
|
//
|
||||||
|
// The decisive argument here is that static seal is NOT A DOWNGRADE FROM THE STATUS
|
||||||
|
// QUO. Today /etc/thermograph/age.key is one file, 0400, root-owned, on each
|
||||||
|
// rendering host, which decrypts every secret in the estate forever with no audit
|
||||||
|
// trail. A static seal key at 0400 on vps2 is the IDENTICAL trust model: one file on
|
||||||
|
// one box whose compromise is total compromise. What changes is everything layered
|
||||||
|
// above it — per-consumer ACLs, an audit log, five-minute tokens, versioned rollback.
|
||||||
|
//
|
||||||
|
// WHY NOT SHAMIR: `bao operator init -key-shares=5 -key-threshold=3` means the vault
|
||||||
|
// is sealed after every process restart — reboot, package upgrade, OOM kill — and a
|
||||||
|
// human must type three shares. render-secrets.sh is on the deploy path for all three
|
||||||
|
// environments, so a sealed vault blocks every deploy and every hotfix. With one
|
||||||
|
// operator the N-of-M threshold is theatre: one person holds all the shares, so it
|
||||||
|
// buys nothing against the real threat while costing an availability property the
|
||||||
|
// estate currently has for free. Today a Contabo reboot needs no human at all.
|
||||||
|
// Introducing a human-in-the-loop requirement that does not exist today is a
|
||||||
|
// regression this estate cannot absorb.
|
||||||
|
//
|
||||||
|
// THE ONE NEW SINGLE POINT OF FAILURE this migration introduces: lose this key and
|
||||||
|
// the raft snapshots become unrestorable. It MUST exist in at least two places, one
|
||||||
|
// of them not on vps2. That is a hard operator obligation, not a suggestion.
|
||||||
|
//
|
||||||
|
// Rotation is supported n-1 via previous_key / previous_key_id.
|
||||||
|
seal "static" {
|
||||||
|
current_key_id = "20260801-1"
|
||||||
|
current_key = "file:///etc/openbao/unseal.key"
|
||||||
|
}
|
||||||
|
|
||||||
|
// ============================================================================
|
||||||
|
// AUDIT: two devices, deliberately.
|
||||||
|
// ============================================================================
|
||||||
|
// ⚠️ OpenBao STOPS ANSWERING REQUESTS ENTIRELY when no enabled audit device can
|
||||||
|
// record them. That is correct behaviour for an audit log and a catastrophic way to
|
||||||
|
// discover a full disk: a full /var on vps2 would block every deploy on the estate.
|
||||||
|
// Two independent devices mean one wedging is survivable.
|
||||||
|
//
|
||||||
|
// Values are HMAC-SHA256'd in the log, so it is safe to ship to Loki through the
|
||||||
|
// existing Alloy pipeline — which finally makes "who read which secret, when" a
|
||||||
|
// queryable question. Today reading age.key leaves no trace whatsoever.
|
||||||
|
//
|
||||||
|
// logrotate on this path MUST signal with SIGHUP, or bao keeps writing to an
|
||||||
|
// unlinked inode and the log silently stops growing.
|
||||||
|
// OpenBao 2.6.1 requires `type` and `path` explicitly — the block label is the
|
||||||
|
// device NAME, not its type — and device settings go in `options`. Written as
|
||||||
|
// `audit "file" { file_path = ... }` the server refuses to start ("audit type must
|
||||||
|
// be specified"); with type+path but a bare file_path it starts but silently
|
||||||
|
// IGNORES the path and mode, which is the worse failure of the two.
|
||||||
|
audit "file" {
|
||||||
|
type = "file"
|
||||||
|
path = "file/"
|
||||||
|
options = {
|
||||||
|
file_path = "/var/log/openbao/audit.log"
|
||||||
|
mode = "0600"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
audit "syslog" {
|
||||||
|
type = "syslog"
|
||||||
|
path = "syslog/"
|
||||||
|
}
|
||||||
50
infra/openbao/openbao.service
Normal file
50
infra/openbao/openbao.service
Normal file
|
|
@ -0,0 +1,50 @@
|
||||||
|
# OpenBao, as a native systemd service on vps2. Install to
|
||||||
|
# /etc/systemd/system/openbao.service.
|
||||||
|
#
|
||||||
|
# Native, not containerised, on purpose — see the header of config/openbao.hcl. The
|
||||||
|
# whole point is that this unit's only dependency is the local disk: not the Docker
|
||||||
|
# daemon that deploys restart, not the Swarm it would otherwise be deployed by, not
|
||||||
|
# Caddy/ACME for its certificate, and not TimescaleDB.
|
||||||
|
#
|
||||||
|
# Note there is currently NO systemd unit of any kind loaded on either host for
|
||||||
|
# Thermograph (infra/deploy/thermograph.service exists in the repo but is not
|
||||||
|
# installed), so this is the first. That is deliberate: it is the one component that
|
||||||
|
# must survive the container layer being broken.
|
||||||
|
|
||||||
|
[Unit]
|
||||||
|
Description=OpenBao secret store
|
||||||
|
Documentation=https://openbao.org/docs/
|
||||||
|
# network-online rather than plain network: the mesh listener binds 10.10.0.1, and
|
||||||
|
# binding a WireGuard address before the interface is up fails the unit outright.
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
# Explicitly NOT After=docker.service. Adding it would recreate the dependency this
|
||||||
|
# design exists to avoid.
|
||||||
|
ConditionFileNotEmpty=/etc/openbao/config.hcl
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
User=openbao
|
||||||
|
Group=openbao
|
||||||
|
ExecStart=/usr/local/bin/bao server -config=/etc/openbao/config.hcl
|
||||||
|
ExecReload=/bin/kill --signal HUP $MAINPID
|
||||||
|
KillSignal=SIGINT
|
||||||
|
|
||||||
|
# Restart always, which together with the static seal is what makes a host reboot
|
||||||
|
# self-healing. This pair is the reason deploys are not blocked by a reboot.
|
||||||
|
Restart=always
|
||||||
|
RestartSec=5
|
||||||
|
|
||||||
|
# mlock is disabled in the config (disable_mlock = true), so no IPC_LOCK capability
|
||||||
|
# is needed. If mlock is ever enabled, add: AmbientCapabilities=CAP_IPC_LOCK
|
||||||
|
NoNewPrivileges=yes
|
||||||
|
PrivateTmp=yes
|
||||||
|
ProtectSystem=full
|
||||||
|
ProtectHome=yes
|
||||||
|
# The three paths the service legitimately writes.
|
||||||
|
ReadWritePaths=/var/lib/openbao /var/log/openbao
|
||||||
|
# Raft + audit both do a lot of small writes; the default is too low for a store.
|
||||||
|
LimitNOFILE=65536
|
||||||
|
LimitCORE=0
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue