Compare commits
4 commits
c17a4c3dd7
...
81135c1be5
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
81135c1be5 | ||
|
|
1d4defefb9 | ||
|
|
890cb4a764 | ||
|
|
0617a3b067 |
6 changed files with 380 additions and 16 deletions
219
.claude/BRANCHING.md
Normal file
219
.claude/BRANCHING.md
Normal file
|
|
@ -0,0 +1,219 @@
|
||||||
|
# Orchestrator instructions — trunk-based flow
|
||||||
|
|
||||||
|
`feat/*` → `dev` → `main` → `release`. One PR per hop, merges serialised by the
|
||||||
|
orchestrator. Forgejo has no native merge queue, so you are the merge queue.
|
||||||
|
|
||||||
|
Use the Centralis `forge_*` tools for every remote operation — PRs, protection,
|
||||||
|
merges, tags. Use local git only for rebases you have to resolve by hand.
|
||||||
|
Never call the Forgejo API with curl and a pasted token; the server holds it.
|
||||||
|
|
||||||
|
The same file exists in `centralis` with the same rules. Where the two repos
|
||||||
|
differ it is noted inline.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The branches
|
||||||
|
|
||||||
|
Each branch has its own environment here, and `infra/deploy/env-topology.sh` is
|
||||||
|
the single source of truth for where each one lives:
|
||||||
|
|
||||||
|
| Branch | Deploys to | Receives |
|
||||||
|
|---|---|---|
|
||||||
|
| `feat/*`, `fix/*` | nothing | your work |
|
||||||
|
| `dev` | dev — vps1, mesh-only, own Postgres | feature PRs |
|
||||||
|
| `main` | beta — beta.thermograph.org, vps2 | promotions from `dev` |
|
||||||
|
| `release` | prod — thermograph.org, vps2 | promotions from `main`, and hotfixes |
|
||||||
|
|
||||||
|
`deploy.yml` handles all three: the branch selects the environment, a matrix
|
||||||
|
covers backend and frontend, and each leg checks whether the push actually
|
||||||
|
touched its domain before rolling. So **FE and BE ship out of lockstep** — a
|
||||||
|
backend change lands without a frontend deploy. That is the point of the split,
|
||||||
|
and it is why `/api/version` and `PAYLOAD_VER` discipline are load-bearing:
|
||||||
|
whatever you promote, the other half may already be running something older.
|
||||||
|
|
||||||
|
`dev` is the default branch, so PRs target it without anyone remembering to
|
||||||
|
retarget. All three are protected: **everything is a PR**, for humans and agents
|
||||||
|
alike.
|
||||||
|
|
||||||
|
## Who may merge what
|
||||||
|
|
||||||
|
This is not the same question as how a change travels, and it is deliberately
|
||||||
|
asymmetric. `MERGE_AUTHORITY` in centralis'
|
||||||
|
`src/lib/promotion.ts` is the whole policy; `forge_pr_merge` enforces it.
|
||||||
|
|
||||||
|
- **Into `dev` — free.** Merge feature PRs yourself, and do not leave them open.
|
||||||
|
A PR opened against `dev` and abandoned is an unfinished task wearing the
|
||||||
|
costume of a delivered one.
|
||||||
|
- **`dev` → `main` — yours, batched.** Beta is the test environment, so
|
||||||
|
withholding this merge withholds the testing. The judgement being asked of you
|
||||||
|
is *when a batch is coherent enough to be worth testing*, not whether you are
|
||||||
|
allowed to test it.
|
||||||
|
- **`main` → `release` — the owner's.** That merge deploys to prod. Call
|
||||||
|
`promote(from="main", to="release")`, which prepares the PR and returns what
|
||||||
|
would ship, the CI evidence and a recommendation. Then put it to the owner —
|
||||||
|
including "I would wait", if that is what you think. No flag overrides this,
|
||||||
|
and routing around it is not an option.
|
||||||
|
|
||||||
|
`confirm_protected_base` is unrelated to the above: it is the hotfix path for a
|
||||||
|
*non-promotion* PR aimed straight at `main` or `release`, and it is audited
|
||||||
|
loudly.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1 — repo setup
|
||||||
|
|
||||||
|
Idempotent. Re-run it whenever you are unsure; every step reports "already
|
||||||
|
correct" rather than erroring.
|
||||||
|
|
||||||
|
1. **The three branches exist.**
|
||||||
|
`forge_branches(create={branch:"main", from:"dev"})` and the same for
|
||||||
|
`release`. If a branch exists but points elsewhere, the tool says so — that
|
||||||
|
is a divergence to reconcile deliberately, not to overwrite.
|
||||||
|
2. **`dev` is the default branch.**
|
||||||
|
`forge_repo_settings(apply_flow_defaults=true)` — sets the default branch and
|
||||||
|
permits every merge style the chain uses, including `fast-forward-only`, plus
|
||||||
|
rebase-updates (without which the merge queue's core primitive 403s).
|
||||||
|
3. **Protection.** `forge_branch_protection(branch="dev", apply=true)`, then
|
||||||
|
`main`, then `release`.
|
||||||
|
|
||||||
|
`dev` and `main` get the required check; **`release` does not**, and that is
|
||||||
|
deliberate. `pr-build.yml` fires only for PRs into `dev` and `main`, so a PR
|
||||||
|
into `release` reports no status at all — requiring a context nothing
|
||||||
|
produces would make every production promotion permanently unmergeable, with
|
||||||
|
a 405 that names no cause. This estate has already lost an afternoon to
|
||||||
|
exactly that, from a single mistyped check name. If you want it closed
|
||||||
|
properly, do it in this order: add `release` to `pr-build.yml`'s
|
||||||
|
`on.pull_request.branches`, confirm on a real PR that the context appears,
|
||||||
|
*then* pass `require_checks_on_release: true`. The other order blocks prod.
|
||||||
|
4. **Stale branches.** `forge_branches()` lists every branch with its age, how
|
||||||
|
much unmerged work it carries, and a verdict:
|
||||||
|
- `merged` — nothing on it the trunk lacks. Delete freely.
|
||||||
|
- `stale` — unmerged work, untouched 14+ days. **File a `stale-branch: <name>`
|
||||||
|
issue summarising its diff first**, then delete. Never discard work
|
||||||
|
silently.
|
||||||
|
- `ageing` — 2+ days with work still on it. Report it; do not delete.
|
||||||
|
5. **Module map.** `.claude/ownership.md` — read it before partitioning work.
|
||||||
|
|
||||||
|
## Phase 2 — the ongoing loop
|
||||||
|
|
||||||
|
### Dispatching subagents
|
||||||
|
|
||||||
|
Partition by module using `.claude/ownership.md`. Never give two concurrent
|
||||||
|
agents tasks touching the same files; if that is unavoidable, sequence them.
|
||||||
|
|
||||||
|
Inject this into every subagent task:
|
||||||
|
|
||||||
|
```
|
||||||
|
Branch off latest `dev` as feat/<slug>. Touch only these files/modules: <scope>.
|
||||||
|
One logical change, small diff. Rebase onto dev before opening your PR.
|
||||||
|
PR targets `dev`. The description must list: what changed, how it was tested,
|
||||||
|
files touched. Do not merge your own PR — the orchestrator handles all merges.
|
||||||
|
```
|
||||||
|
|
||||||
|
Keep a branch alive **less than a day**. A task bigger than that is several
|
||||||
|
stacked tasks.
|
||||||
|
|
||||||
|
### The merge queue
|
||||||
|
|
||||||
|
You merge **one PR into `dev` at a time**. For each:
|
||||||
|
|
||||||
|
1. `forge_prs(base="dev")` — the queue, with each PR's head sha and triage flag.
|
||||||
|
2. `forge_pr_update(pr=N)` — replay it onto the current tip. If the head sha
|
||||||
|
does not move, it was already current and no CI will re-fire; do not sit
|
||||||
|
waiting for a run that was never scheduled.
|
||||||
|
3. `forge_pr_await(pr=N, until="checks_complete")` — checks on the **rebased**
|
||||||
|
head. The run you saw before the update was for a merge base that no longer
|
||||||
|
exists.
|
||||||
|
4. `forge_pr_merge(pr=N, method="rebase", delete_branch=true)`.
|
||||||
|
5. Only then move to the next PR.
|
||||||
|
|
||||||
|
`block_on_outdated_branch` is set on `dev`, so Forgejo enforces step 2 rather
|
||||||
|
than trusting you to remember it: after one merge, the next PR is refused until
|
||||||
|
it has been replayed. Treat that 405 as the queue working, not as a fault.
|
||||||
|
|
||||||
|
If a rebase conflicts: resolve it yourself if it is trivial and mechanical
|
||||||
|
(under ~20 lines), otherwise hand it back to the owning agent with the conflict
|
||||||
|
context. Do not resolve a large conflict on someone else's behalf.
|
||||||
|
|
||||||
|
### Promotion: `dev` → `main`
|
||||||
|
|
||||||
|
Promote when `dev` is green and a coherent batch is complete — not on a timer,
|
||||||
|
not mid-feature. `promote(from="dev", to="main")` opens the PR with the
|
||||||
|
divergence, the conflict prediction and the changelog written into its body.
|
||||||
|
Merge it with `method="merge"`.
|
||||||
|
|
||||||
|
Unready work belongs behind a feature flag, or not on `dev` yet. **Never promote
|
||||||
|
around it with cherry-picks.**
|
||||||
|
|
||||||
|
`promote` refuses a hop whose two tips have identical trees — the commit counts
|
||||||
|
look like real work while the content is already on the target under different
|
||||||
|
SHAs, and merging it would fire the target's deploys for nothing.
|
||||||
|
|
||||||
|
### Promotion: `main` → `release`
|
||||||
|
|
||||||
|
Prepare with `promote`, then ask. On the owner's yes, merge with `method="merge"`
|
||||||
|
and tag it: `forge_releases(create={tag:"v<semver>", target:"release", ...})`.
|
||||||
|
|
||||||
|
**On fast-forward.** The written policy is that `release` moves only by
|
||||||
|
fast-forward. It cannot today: every promotion so far has been a merge commit
|
||||||
|
*into* the target, which the source never receives, so the branches are mutually
|
||||||
|
divergent by construction and Forgejo will refuse a `fast-forward-only` merge —
|
||||||
|
correctly. The style is implemented and accepted by `forge_pr_merge`; adopting
|
||||||
|
it needs a one-time reconciliation of the protected branches, which is the
|
||||||
|
owner's decision. Until that happens, promotions are merge commits. Do not
|
||||||
|
attempt to "fix" this with a rebase or a force-push.
|
||||||
|
|
||||||
|
### Hotfix
|
||||||
|
|
||||||
|
1. `forge_branches(create={branch:"hotfix/<slug>", from:"release"})`.
|
||||||
|
2. Fix, PR into `release`, checks green, merge with
|
||||||
|
`confirm_protected_base: true`, tag `v<semver-patch>`.
|
||||||
|
3. **Immediately down-merge** `release` → `main` and `main` → `dev`, as merges,
|
||||||
|
not cherry-picks, in the same session. A hotfix that exists only on `release`
|
||||||
|
is a regression scheduled for the next promotion.
|
||||||
|
|
||||||
|
Note that CI does not gate PRs into `release` — `pr-build.yml` fires only for
|
||||||
|
`dev` and `main`. A hotfix's green evidence therefore has to come from somewhere
|
||||||
|
else; say plainly where it came from rather than implying a check passed.
|
||||||
|
|
||||||
|
A hotfix that touches only one domain deploys only that domain. Check the other
|
||||||
|
half's `/api/version` before assuming the fix is live end to end.
|
||||||
|
|
||||||
|
### When you genuinely need to push directly
|
||||||
|
|
||||||
|
`apply_to_admins` is on, which means these rules bind the owner too. That is
|
||||||
|
deliberate — protection an admin walks straight through is documentation, not a
|
||||||
|
control, and every account here is an admin — but it does remove the escape
|
||||||
|
hatch, so the way back in is worth writing down before it is needed at 3am.
|
||||||
|
|
||||||
|
There is no force flag. The supported route is to lift the rule, do the thing,
|
||||||
|
and put it back:
|
||||||
|
|
||||||
|
```
|
||||||
|
forge_branch_protection(branch="dev", delete=true) # rule gone; direct push allowed
|
||||||
|
git push origin dev # the emergency change
|
||||||
|
forge_branch_protection(branch="dev", apply=true) # restored to policy
|
||||||
|
```
|
||||||
|
|
||||||
|
Both the removal and the restore are audited, which is the point: the hatch is
|
||||||
|
open on the record rather than by a quiet exception. Restore it in the same
|
||||||
|
session — a protection rule "temporarily" removed is how a repo ends up
|
||||||
|
unprotected for a month.
|
||||||
|
|
||||||
|
The failure mode that actually justifies this: if the required status context
|
||||||
|
is ever renamed and no longer matches, **nothing can merge into `dev` or `main`
|
||||||
|
at all**, including by you, because `apply_to_admins` no longer exempts anyone.
|
||||||
|
`forge_pr_await` and `forge_pr_merge` both name that case explicitly when they
|
||||||
|
see it. The fix is to correct `pr-build.yml`'s workflow/job names and
|
||||||
|
`CENTRALIS_REQUIRED_CHECK` together — but if you need to land the fix itself,
|
||||||
|
lift the rule as above.
|
||||||
|
|
||||||
|
## Guardrails
|
||||||
|
|
||||||
|
- Never force-push `dev`, `main` or `release`.
|
||||||
|
- Never merge a red PR, hotfix included. Fix CI, or fix the tests in the same PR.
|
||||||
|
- Never merge two green PRs back to back without re-checking the second against
|
||||||
|
the new tip.
|
||||||
|
- If two down-merges or promotions conflict beyond the trivial, stop and report
|
||||||
|
rather than resolving autonomously.
|
||||||
|
- Weekly: `forge_branches()` and report anything `ageing` or `stale`.
|
||||||
60
.claude/ownership.md
Normal file
60
.claude/ownership.md
Normal file
|
|
@ -0,0 +1,60 @@
|
||||||
|
# Module map — what can be worked on concurrently
|
||||||
|
|
||||||
|
Used for partitioning work across concurrent subagents. The rule is one owner
|
||||||
|
per file at a time: **never** give two concurrent agents tasks that touch the
|
||||||
|
same module. If two tasks genuinely need the same file, sequence them instead of
|
||||||
|
racing them — a rebase conflict costs more than the parallelism saved.
|
||||||
|
|
||||||
|
Read this before dispatching. It answers one question: *can these two tasks run
|
||||||
|
at the same time?*
|
||||||
|
|
||||||
|
The monorepo's four domains are the natural seam, and they are a real one: FE
|
||||||
|
and BE build and deploy independently, and `deploy.yml` checks per-leg whether a
|
||||||
|
push actually touched that domain. Two agents in two different domains will not
|
||||||
|
collide in CI either.
|
||||||
|
|
||||||
|
## Safe to assign concurrently
|
||||||
|
|
||||||
|
| Module | Files | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| Backend — API | `backend/api/**`, `backend/app.py` | Route handlers and serialisation. |
|
||||||
|
| Backend — core | `backend/core/**` | Grading, percentiles, climatology. The domain logic. |
|
||||||
|
| Backend — accounts | `backend/accounts/**` | Auth, sessions, OAuth links. |
|
||||||
|
| Backend — daemon | `backend/daemon/**` | Scheduled work. Anything named "prefetch" must not spend the Open-Meteo quota. |
|
||||||
|
| Backend — data & lake | `backend/data/**`, `backend/gen_era5_lake.py`, `backend/gen_cities.py` | |
|
||||||
|
| Backend — migrations | `backend/alembic/**`, `backend/alembic.ini` | One owner, always. Two agents generating revisions produce two heads. |
|
||||||
|
| Frontend — server | `frontend/server/**`, `frontend/app.py`, `frontend/api_client.py` | Go SSR and the Python app. |
|
||||||
|
| Frontend — static | `frontend/static/**` | |
|
||||||
|
| Frontend — templates | `frontend/templates/**`, `frontend/content/**`, `frontend/content.py` | |
|
||||||
|
| Infra — deploy | `infra/deploy/**` (excluding `secrets/`), `infra/docker-compose*.yml` | |
|
||||||
|
| Infra — secrets | `infra/deploy/secrets/**` | SOPS vault. One owner, and never alongside `infra/deploy`. |
|
||||||
|
| Infra — ops | `infra/ops/**`, `infra/lake-iceberg/**` | |
|
||||||
|
| Infra — terraform | `infra/terraform/**` | |
|
||||||
|
| Observability | `observability/**` | Grafana dashboards are provisioned from this JSON; a UI edit is overwritten. |
|
||||||
|
| Docs | `docs/**` | |
|
||||||
|
| Agent tooling | `.claude/hooks/**` | |
|
||||||
|
|
||||||
|
## Serialise — shared spine
|
||||||
|
|
||||||
|
Touched by most changes. A change here is its own task with nothing else running
|
||||||
|
against it.
|
||||||
|
|
||||||
|
| File | Why |
|
||||||
|
|---|---|
|
||||||
|
| `.forgejo/workflows/**` | CI lives **only** here, path-filtered per domain. Every domain's changes route through the same files, so two agents editing workflows conflict even when their domains do not. |
|
||||||
|
| `CLAUDE.md` and every domain `CLAUDE.md` | Read before every change; a stale one is a correctness bug. Small edits, landed fast. |
|
||||||
|
| `CUTOVER-NOTES.md` | The source of truth for what is and is not live. |
|
||||||
|
| `infra/deploy/env-topology.sh` | The single source of truth for where each environment's checkout, branch, stack and ports live. |
|
||||||
|
| `infra/deploy/deploy.sh`, `infra/deploy/stack/deploy-stack.sh` | The deploy contract's entry points. |
|
||||||
|
| `backend/CLAUDE.md` + the `/api/version` contract | FE and BE ship out of lockstep, so `PAYLOAD_VER` discipline is load-bearing. A change to the payload shape is one task spanning both domains — not two concurrent ones. |
|
||||||
|
|
||||||
|
## The recurring collisions
|
||||||
|
|
||||||
|
- **A schema change is not a backend-only task.** It is `backend/alembic` plus
|
||||||
|
whatever reads the column, and if the payload shape moves it is `PAYLOAD_VER`
|
||||||
|
and the frontend too. Scope it as one task across both domains rather than
|
||||||
|
two agents discovering each other mid-flight.
|
||||||
|
- **Adding CI for a domain edits the shared workflows.** Sequence it against any
|
||||||
|
other workflow work.
|
||||||
|
- **`infra/deploy/secrets/` and `infra/deploy/` are one owner between them.**
|
||||||
|
The rendered artifact and the thing that renders it move together.
|
||||||
|
|
@ -22,6 +22,11 @@ every change, so a stale one is a correctness bug, not a documentation bug.
|
||||||
`dev`, `main` and `release` are protected: **everything is a PR**, for humans and
|
`dev`, `main` and `release` are protected: **everything is a PR**, for humans and
|
||||||
agents alike. Promotion is one PR per hop, `dev` → `main` → `release`.
|
agents alike. Promotion is one PR per hop, `dev` → `main` → `release`.
|
||||||
|
|
||||||
|
`.claude/BRANCHING.md` is the orchestrator's runbook for that flow — the
|
||||||
|
serialised merge queue, promotions, hotfix down-merges — and
|
||||||
|
`.claude/ownership.md` is the module map saying which tasks may run
|
||||||
|
concurrently. Read both before dispatching parallel work.
|
||||||
|
|
||||||
**`dev` is a first-class hosted environment, not just an integration branch.**
|
**`dev` is a first-class hosted environment, not just an integration branch.**
|
||||||
It runs on vps1 — the same box as Forgejo and the monitoring stack — with its
|
It runs on vps1 — the same box as Forgejo and the monitoring stack — with its
|
||||||
own Postgres container, reachable only on the WireGuard mesh
|
own Postgres container, reachable only on the WireGuard mesh
|
||||||
|
|
|
||||||
|
|
@ -62,11 +62,21 @@ def _apply_sqlite_pragmas(dbapi_conn, _rec):
|
||||||
#
|
#
|
||||||
# pool_size=1/max_overflow=1 (the original setting on both) meant a 3rd
|
# pool_size=1/max_overflow=1 (the original setting on both) meant a 3rd
|
||||||
# concurrent request per worker had to wait out pool_timeout (~30s default) for a
|
# concurrent request per worker had to wait out pool_timeout (~30s default) for a
|
||||||
# slot — easy to hit with 4 uvicorn workers x real traffic. 5+5 gives each async
|
# slot — easy to hit with 4 uvicorn workers x real traffic. pool_size=5 keeps that
|
||||||
# engine headroom for a burst of concurrent account requests without coming close
|
# headroom for the steady state.
|
||||||
# to Postgres's own connection ceiling (two async engines here plus the sync
|
#
|
||||||
# engine below, times N workers, all share that budget).
|
# max_overflow is deliberately *tighter* than pool_size (2, not 5), because the
|
||||||
_ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=5, pool_pre_ping=True,
|
# overflow is the part nobody budgets for and it is what saturated Postgres on
|
||||||
|
# 2026-07-26. The connection ceiling is shared far more widely than this process
|
||||||
|
# knows: two async engines here plus the sync engine below, times N uvicorn
|
||||||
|
# workers, times the replica count (web autoscales), and prod and beta are two
|
||||||
|
# databases on ONE Postgres instance. At 5+5 the configured worst case across all
|
||||||
|
# of that exceeded the server's max_connections; at 5+2 prod web's ceiling is
|
||||||
|
# 4x14 rather than 4x20. Steady-state behaviour is unchanged — only the burst
|
||||||
|
# ceiling narrows, and a burst that would have opened a 6th..10th connection now
|
||||||
|
# waits on pool_timeout instead of being refused outright by the server (which is
|
||||||
|
# an error, not a wait). See infra/deploy/db/init/20-tuning.sh for the server side.
|
||||||
|
_ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=2, pool_pre_ping=True,
|
||||||
pool_recycle=1800, future=True)
|
pool_recycle=1800, future=True)
|
||||||
# Smaller than the async engines: only the notifier leader (one process
|
# Smaller than the async engines: only the notifier leader (one process
|
||||||
# cluster-wide, see core/singleton.py) uses the sync engine, driving its periodic
|
# cluster-wide, see core/singleton.py) uses the sync engine, driving its periodic
|
||||||
|
|
@ -74,7 +84,8 @@ _ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=5, pool_pre_ping=True,
|
||||||
# statement_timeout (via psycopg's `options`) bound how long a wedged Postgres
|
# statement_timeout (via psycopg's `options`) bound how long a wedged Postgres
|
||||||
# can hang the notifier — it runs for the lifetime of that process's leader
|
# can hang the notifier — it runs for the lifetime of that process's leader
|
||||||
# flock, so an indefinite hang here would silently stop every subscription
|
# flock, so an indefinite hang here would silently stop every subscription
|
||||||
# notification until the process is restarted.
|
# notification until the process is restarted. Left at 2+3: one process holds it,
|
||||||
|
# so it is not multiplied by the worker or replica count.
|
||||||
_SYNC_POOL_KWARGS = dict(pool_size=2, max_overflow=3, pool_pre_ping=True,
|
_SYNC_POOL_KWARGS = dict(pool_size=2, max_overflow=3, pool_pre_ping=True,
|
||||||
pool_recycle=1800, future=True,
|
pool_recycle=1800, future=True,
|
||||||
connect_args={"connect_timeout": 5,
|
connect_args={"connect_timeout": 5,
|
||||||
|
|
|
||||||
|
|
@ -9,6 +9,28 @@ from sqlalchemy import select
|
||||||
|
|
||||||
from accounts import db
|
from accounts import db
|
||||||
|
|
||||||
|
# The server-side ceiling these pools spend from, and the deployment shape that
|
||||||
|
# spends it. Kept here so a pool change that stops fitting is a failing test
|
||||||
|
# rather than a 2am "sorry, too many clients already". Mirrors max_connections in
|
||||||
|
# infra/deploy/db/init/20-tuning.sh. ONE Postgres instance serves prod and beta.
|
||||||
|
MAX_CONNECTIONS = 200
|
||||||
|
PROD_WEB_WORKERS = 4 # uvicorn workers per prod web replica
|
||||||
|
BETA_WEB_WORKERS = 2
|
||||||
|
ASYNC_ENGINES_PER_PROCESS = 2 # RW + RO, both built from _ASYNC_POOL_KWARGS
|
||||||
|
|
||||||
|
|
||||||
|
def _async_ceiling_per_process() -> int:
|
||||||
|
"""Connections one backend process can hold from its async pools at once."""
|
||||||
|
k = db._ASYNC_POOL_KWARGS
|
||||||
|
return ASYNC_ENGINES_PER_PROCESS * (k["pool_size"] + k["max_overflow"])
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_ceiling() -> int:
|
||||||
|
"""The notifier's sync pool. One process holds it cluster-wide (leader flock),
|
||||||
|
so it is NOT multiplied by workers or replicas."""
|
||||||
|
k = db._SYNC_POOL_KWARGS
|
||||||
|
return k["pool_size"] + k["max_overflow"]
|
||||||
|
|
||||||
|
|
||||||
def test_defaults_to_sqlite_when_no_database_url():
|
def test_defaults_to_sqlite_when_no_database_url():
|
||||||
assert db.IS_POSTGRES is False
|
assert db.IS_POSTGRES is False
|
||||||
|
|
@ -32,13 +54,18 @@ def test_postgres_pool_sizing_and_sync_timeouts():
|
||||||
"""The Postgres engine-construction kwargs, defined at module scope so they're
|
"""The Postgres engine-construction kwargs, defined at module scope so they're
|
||||||
testable without a real Postgres connection (the engines built from them are
|
testable without a real Postgres connection (the engines built from them are
|
||||||
only actually constructed when IS_POSTGRES, exercised by the docker-compose
|
only actually constructed when IS_POSTGRES, exercised by the docker-compose
|
||||||
stack -- see the module docstring). pool_size=1/max_overflow=1 (the old
|
stack -- see the module docstring). pool_size=1/max_overflow=1 (the original
|
||||||
setting) let a 3rd concurrent request per worker wait out pool_timeout for a
|
setting) let a 3rd concurrent request per worker wait out pool_timeout for a
|
||||||
slot; both async engines need real headroom, and the sync engine (the
|
slot, so the steady-state pool needs real headroom; the sync engine (the
|
||||||
notifier's, driven off-event-loop) must bound connect + statement time so a
|
notifier's, driven off-event-loop) must bound connect + statement time so a
|
||||||
wedged Postgres can't hang it forever under its leader flock."""
|
wedged Postgres can't hang it forever under its leader flock."""
|
||||||
assert db._ASYNC_POOL_KWARGS["pool_size"] >= 5
|
assert db._ASYNC_POOL_KWARGS["pool_size"] >= 5
|
||||||
assert db._ASYNC_POOL_KWARGS["max_overflow"] >= 5
|
|
||||||
|
# The overflow is the part nobody budgets for -- it is multiplied by two async
|
||||||
|
# engines x uvicorn workers x replicas, against a ceiling shared with beta.
|
||||||
|
# Prod saturated max_connections on 2026-07-26 with this at 5. Keep it well
|
||||||
|
# under pool_size; the budget test below is the real constraint.
|
||||||
|
assert db._ASYNC_POOL_KWARGS["max_overflow"] <= 2
|
||||||
|
|
||||||
assert db._SYNC_POOL_KWARGS["pool_size"] < db._ASYNC_POOL_KWARGS["pool_size"]
|
assert db._SYNC_POOL_KWARGS["pool_size"] < db._ASYNC_POOL_KWARGS["pool_size"]
|
||||||
connect_args = db._SYNC_POOL_KWARGS["connect_args"]
|
connect_args = db._SYNC_POOL_KWARGS["connect_args"]
|
||||||
|
|
@ -53,6 +80,27 @@ def test_postgres_pool_sizing_and_sync_timeouts():
|
||||||
engine.dispose()
|
engine.dispose()
|
||||||
|
|
||||||
|
|
||||||
|
def test_pool_budget_fits_under_max_connections():
|
||||||
|
"""At today's replica counts the whole estate's configured worst case must fit
|
||||||
|
inside the server's max_connections, with margin -- prod web + prod worker +
|
||||||
|
beta web + beta worker all draw on ONE Postgres instance. This is the test
|
||||||
|
that would have failed before 2026-07-26, when the configured worst case was
|
||||||
|
170 against a ceiling of 100.
|
||||||
|
|
||||||
|
Note this counts *configured maxima*: every pool at full overflow at the same
|
||||||
|
instant. Prod web autoscaling to its max of 3 replicas is deliberately NOT in
|
||||||
|
this sum -- that peak still does not fit, and closing it needs a connection
|
||||||
|
pooler or a lower autoscale ceiling rather than a bigger number here."""
|
||||||
|
per_process = _async_ceiling_per_process()
|
||||||
|
prod = per_process * PROD_WEB_WORKERS + (per_process + _sync_ceiling())
|
||||||
|
beta = per_process * BETA_WEB_WORKERS + (per_process + _sync_ceiling())
|
||||||
|
|
||||||
|
assert prod + beta <= MAX_CONNECTIONS * 0.8, (
|
||||||
|
f"configured worst case {prod + beta} leaves too little room under "
|
||||||
|
f"max_connections={MAX_CONNECTIONS}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_read_session_yields_a_working_session():
|
def test_read_session_yields_a_working_session():
|
||||||
# On SQLite the read session is fully usable (no read-only enforcement); this
|
# On SQLite the read session is fully usable (no read-only enforcement); this
|
||||||
# just confirms the dependency wiring resolves to a live AsyncSession.
|
# just confirms the dependency wiring resolves to a live AsyncSession.
|
||||||
|
|
|
||||||
|
|
@ -8,10 +8,10 @@
|
||||||
# 10-timescaledb.sql enables timescaledb. Settings are written via ALTER SYSTEM
|
# 10-timescaledb.sql enables timescaledb. Settings are written via ALTER SYSTEM
|
||||||
# (persisted to postgresql.auto.conf, which the timescaledb image's own
|
# (persisted to postgresql.auto.conf, which the timescaledb image's own
|
||||||
# timescaledb-tune postgresql.conf defers to); the container's post-init restart brings
|
# timescaledb-tune postgresql.conf defers to); the container's post-init restart brings
|
||||||
# restart-only settings (shared_buffers, …) into effect. NB: never ALTER SYSTEM SET
|
# restart-only settings (shared_buffers, max_connections, …) into effect. NB: never
|
||||||
# shared_preload_libraries here — that would land in auto.conf and override the image's
|
# ALTER SYSTEM SET shared_preload_libraries here — that would land in auto.conf and
|
||||||
# `timescaledb` preload. Init scripts do NOT re-run on an existing volume — to
|
# override the image's `timescaledb` preload. Init scripts do NOT re-run on an existing
|
||||||
# re-tune later, set DB_MEMORY and run this by hand, then restart:
|
# volume — to re-tune later, set DB_MEMORY and run this by hand, then restart:
|
||||||
# docker compose exec -e DB_MEMORY=16g db bash /docker-entrypoint-initdb.d/20-tuning.sh
|
# docker compose exec -e DB_MEMORY=16g db bash /docker-entrypoint-initdb.d/20-tuning.sh
|
||||||
# docker compose restart db
|
# docker compose restart db
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
@ -33,13 +33,30 @@ if [ "$mb" -lt 1024 ]; then mb=8192; fi
|
||||||
# maintenance_work_mem 512 MB) and scale linearly on a bigger box.
|
# maintenance_work_mem 512 MB) and scale linearly on a bigger box.
|
||||||
shared_buffers=$((mb / 4)) # 25% — the shared page cache
|
shared_buffers=$((mb / 4)) # 25% — the shared page cache
|
||||||
effective_cache=$((mb * 3 / 4)) # 75% — planner's view of total cache (PG + OS)
|
effective_cache=$((mb * 3 / 4)) # 75% — planner's view of total cache (PG + OS)
|
||||||
work_mem=$((mb / 128)) # ~64 MB at 8 GB (per-operation; kept modest)
|
maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM
|
||||||
|
|
||||||
|
# work_mem is per *sort operation*, not per connection, so its real cost is
|
||||||
|
# work_mem x concurrent sorts x max_connections (200, below) — and the db container
|
||||||
|
# has a hard memory limit. Left uncapped the ratio gives 128 MB at prod's 16g
|
||||||
|
# budget, which is more than a 200-connection instance should carry: a burst of
|
||||||
|
# heavy sorts can walk into the container limit, and an OOM-killed Postgres is a
|
||||||
|
# far worse day than a refused connection. Cap at 64 MB (dev's 8g is unaffected).
|
||||||
|
work_mem=$((mb / 128))
|
||||||
|
if [ "$work_mem" -gt 64 ]; then work_mem=64; fi
|
||||||
if [ "$work_mem" -lt 16 ]; then work_mem=16; fi
|
if [ "$work_mem" -lt 16 ]; then work_mem=16; fi
|
||||||
maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM
|
|
||||||
|
# Connection ceiling. NOT derived from DB_MEMORY: it is bounded by how many pools
|
||||||
|
# the app opens, not by RAM. One instance serves prod and beta, and every backend
|
||||||
|
# process opens three pools (two async engines + the notifier's sync engine, see
|
||||||
|
# backend/accounts/db.py), so the image default of 100 was below the configured
|
||||||
|
# worst case — prod saturated it on 2026-07-26 and started refusing clients with
|
||||||
|
# "sorry, too many clients already". 200 covers today's replica counts including
|
||||||
|
# web's autoscale maximum. ~+1 GB of per-connection overhead at 200.
|
||||||
|
max_conn=200
|
||||||
|
|
||||||
echo "[tuning] DB_MEMORY=${budget} -> ${mb}MB: shared_buffers=${shared_buffers}MB" \
|
echo "[tuning] DB_MEMORY=${budget} -> ${mb}MB: shared_buffers=${shared_buffers}MB" \
|
||||||
"effective_cache_size=${effective_cache}MB work_mem=${work_mem}MB" \
|
"effective_cache_size=${effective_cache}MB work_mem=${work_mem}MB" \
|
||||||
"maintenance_work_mem=${maint_mem}MB"
|
"maintenance_work_mem=${maint_mem}MB max_connections=${max_conn}"
|
||||||
|
|
||||||
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<SQL
|
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<SQL
|
||||||
-- Caching: the shared page cache, and the planner's view of total cache (PG + OS).
|
-- Caching: the shared page cache, and the planner's view of total cache (PG + OS).
|
||||||
|
|
@ -50,6 +67,10 @@ ALTER SYSTEM SET effective_cache_size = '${effective_cache}MB';
|
||||||
ALTER SYSTEM SET work_mem = '${work_mem}MB';
|
ALTER SYSTEM SET work_mem = '${work_mem}MB';
|
||||||
ALTER SYSTEM SET maintenance_work_mem = '${maint_mem}MB';
|
ALTER SYSTEM SET maintenance_work_mem = '${maint_mem}MB';
|
||||||
|
|
||||||
|
-- Connections. Restart-only, like shared_buffers: changing this on a live instance
|
||||||
|
-- takes an ALTER SYSTEM plus a db restart, it does not reload.
|
||||||
|
ALTER SYSTEM SET max_connections = ${max_conn};
|
||||||
|
|
||||||
-- Write throughput: fewer, larger checkpoints.
|
-- Write throughput: fewer, larger checkpoints.
|
||||||
ALTER SYSTEM SET wal_buffers = '16MB';
|
ALTER SYSTEM SET wal_buffers = '16MB';
|
||||||
ALTER SYSTEM SET min_wal_size = '1GB';
|
ALTER SYSTEM SET min_wal_size = '1GB';
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue