Compare commits
4 commits
c17a4c3dd7
...
81135c1be5
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
81135c1be5 | ||
|
|
1d4defefb9 | ||
|
|
890cb4a764 | ||
|
|
0617a3b067 |
6 changed files with 380 additions and 16 deletions
219
.claude/BRANCHING.md
Normal file
219
.claude/BRANCHING.md
Normal file
|
|
@ -0,0 +1,219 @@
|
|||
# Orchestrator instructions — trunk-based flow
|
||||
|
||||
`feat/*` → `dev` → `main` → `release`. One PR per hop, merges serialised by the
|
||||
orchestrator. Forgejo has no native merge queue, so you are the merge queue.
|
||||
|
||||
Use the Centralis `forge_*` tools for every remote operation — PRs, protection,
|
||||
merges, tags. Use local git only for rebases you have to resolve by hand.
|
||||
Never call the Forgejo API with curl and a pasted token; the server holds it.
|
||||
|
||||
The same file exists in `centralis` with the same rules. Where the two repos
|
||||
differ it is noted inline.
|
||||
|
||||
---
|
||||
|
||||
## The branches
|
||||
|
||||
Each branch has its own environment here, and `infra/deploy/env-topology.sh` is
|
||||
the single source of truth for where each one lives:
|
||||
|
||||
| Branch | Deploys to | Receives |
|
||||
|---|---|---|
|
||||
| `feat/*`, `fix/*` | nothing | your work |
|
||||
| `dev` | dev — vps1, mesh-only, own Postgres | feature PRs |
|
||||
| `main` | beta — beta.thermograph.org, vps2 | promotions from `dev` |
|
||||
| `release` | prod — thermograph.org, vps2 | promotions from `main`, and hotfixes |
|
||||
|
||||
`deploy.yml` handles all three: the branch selects the environment, a matrix
|
||||
covers backend and frontend, and each leg checks whether the push actually
|
||||
touched its domain before rolling. So **FE and BE ship out of lockstep** — a
|
||||
backend change lands without a frontend deploy. That is the point of the split,
|
||||
and it is why `/api/version` and `PAYLOAD_VER` discipline are load-bearing:
|
||||
whatever you promote, the other half may already be running something older.
|
||||
|
||||
`dev` is the default branch, so PRs target it without anyone remembering to
|
||||
retarget. All three are protected: **everything is a PR**, for humans and agents
|
||||
alike.
|
||||
|
||||
## Who may merge what
|
||||
|
||||
This is not the same question as how a change travels, and it is deliberately
|
||||
asymmetric. `MERGE_AUTHORITY` in centralis'
|
||||
`src/lib/promotion.ts` is the whole policy; `forge_pr_merge` enforces it.
|
||||
|
||||
- **Into `dev` — free.** Merge feature PRs yourself, and do not leave them open.
|
||||
A PR opened against `dev` and abandoned is an unfinished task wearing the
|
||||
costume of a delivered one.
|
||||
- **`dev` → `main` — yours, batched.** Beta is the test environment, so
|
||||
withholding this merge withholds the testing. The judgement being asked of you
|
||||
is *when a batch is coherent enough to be worth testing*, not whether you are
|
||||
allowed to test it.
|
||||
- **`main` → `release` — the owner's.** That merge deploys to prod. Call
|
||||
`promote(from="main", to="release")`, which prepares the PR and returns what
|
||||
would ship, the CI evidence and a recommendation. Then put it to the owner —
|
||||
including "I would wait", if that is what you think. No flag overrides this,
|
||||
and routing around it is not an option.
|
||||
|
||||
`confirm_protected_base` is unrelated to the above: it is the hotfix path for a
|
||||
*non-promotion* PR aimed straight at `main` or `release`, and it is audited
|
||||
loudly.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — repo setup
|
||||
|
||||
Idempotent. Re-run it whenever you are unsure; every step reports "already
|
||||
correct" rather than erroring.
|
||||
|
||||
1. **The three branches exist.**
|
||||
`forge_branches(create={branch:"main", from:"dev"})` and the same for
|
||||
`release`. If a branch exists but points elsewhere, the tool says so — that
|
||||
is a divergence to reconcile deliberately, not to overwrite.
|
||||
2. **`dev` is the default branch.**
|
||||
`forge_repo_settings(apply_flow_defaults=true)` — sets the default branch and
|
||||
permits every merge style the chain uses, including `fast-forward-only`, plus
|
||||
rebase-updates (without which the merge queue's core primitive 403s).
|
||||
3. **Protection.** `forge_branch_protection(branch="dev", apply=true)`, then
|
||||
`main`, then `release`.
|
||||
|
||||
`dev` and `main` get the required check; **`release` does not**, and that is
|
||||
deliberate. `pr-build.yml` fires only for PRs into `dev` and `main`, so a PR
|
||||
into `release` reports no status at all — requiring a context nothing
|
||||
produces would make every production promotion permanently unmergeable, with
|
||||
a 405 that names no cause. This estate has already lost an afternoon to
|
||||
exactly that, from a single mistyped check name. If you want it closed
|
||||
properly, do it in this order: add `release` to `pr-build.yml`'s
|
||||
`on.pull_request.branches`, confirm on a real PR that the context appears,
|
||||
*then* pass `require_checks_on_release: true`. The other order blocks prod.
|
||||
4. **Stale branches.** `forge_branches()` lists every branch with its age, how
|
||||
much unmerged work it carries, and a verdict:
|
||||
- `merged` — nothing on it the trunk lacks. Delete freely.
|
||||
- `stale` — unmerged work, untouched 14+ days. **File a `stale-branch: <name>`
|
||||
issue summarising its diff first**, then delete. Never discard work
|
||||
silently.
|
||||
- `ageing` — 2+ days with work still on it. Report it; do not delete.
|
||||
5. **Module map.** `.claude/ownership.md` — read it before partitioning work.
|
||||
|
||||
## Phase 2 — the ongoing loop
|
||||
|
||||
### Dispatching subagents
|
||||
|
||||
Partition by module using `.claude/ownership.md`. Never give two concurrent
|
||||
agents tasks touching the same files; if that is unavoidable, sequence them.
|
||||
|
||||
Inject this into every subagent task:
|
||||
|
||||
```
|
||||
Branch off latest `dev` as feat/<slug>. Touch only these files/modules: <scope>.
|
||||
One logical change, small diff. Rebase onto dev before opening your PR.
|
||||
PR targets `dev`. The description must list: what changed, how it was tested,
|
||||
files touched. Do not merge your own PR — the orchestrator handles all merges.
|
||||
```
|
||||
|
||||
Keep a branch alive **less than a day**. A task bigger than that is several
|
||||
stacked tasks.
|
||||
|
||||
### The merge queue
|
||||
|
||||
You merge **one PR into `dev` at a time**. For each:
|
||||
|
||||
1. `forge_prs(base="dev")` — the queue, with each PR's head sha and triage flag.
|
||||
2. `forge_pr_update(pr=N)` — replay it onto the current tip. If the head sha
|
||||
does not move, it was already current and no CI will re-fire; do not sit
|
||||
waiting for a run that was never scheduled.
|
||||
3. `forge_pr_await(pr=N, until="checks_complete")` — checks on the **rebased**
|
||||
head. The run you saw before the update was for a merge base that no longer
|
||||
exists.
|
||||
4. `forge_pr_merge(pr=N, method="rebase", delete_branch=true)`.
|
||||
5. Only then move to the next PR.
|
||||
|
||||
`block_on_outdated_branch` is set on `dev`, so Forgejo enforces step 2 rather
|
||||
than trusting you to remember it: after one merge, the next PR is refused until
|
||||
it has been replayed. Treat that 405 as the queue working, not as a fault.
|
||||
|
||||
If a rebase conflicts: resolve it yourself if it is trivial and mechanical
|
||||
(under ~20 lines), otherwise hand it back to the owning agent with the conflict
|
||||
context. Do not resolve a large conflict on someone else's behalf.
|
||||
|
||||
### Promotion: `dev` → `main`
|
||||
|
||||
Promote when `dev` is green and a coherent batch is complete — not on a timer,
|
||||
not mid-feature. `promote(from="dev", to="main")` opens the PR with the
|
||||
divergence, the conflict prediction and the changelog written into its body.
|
||||
Merge it with `method="merge"`.
|
||||
|
||||
Unready work belongs behind a feature flag, or not on `dev` yet. **Never promote
|
||||
around it with cherry-picks.**
|
||||
|
||||
`promote` refuses a hop whose two tips have identical trees — the commit counts
|
||||
look like real work while the content is already on the target under different
|
||||
SHAs, and merging it would fire the target's deploys for nothing.
|
||||
|
||||
### Promotion: `main` → `release`
|
||||
|
||||
Prepare with `promote`, then ask. On the owner's yes, merge with `method="merge"`
|
||||
and tag it: `forge_releases(create={tag:"v<semver>", target:"release", ...})`.
|
||||
|
||||
**On fast-forward.** The written policy is that `release` moves only by
|
||||
fast-forward. It cannot today: every promotion so far has been a merge commit
|
||||
*into* the target, which the source never receives, so the branches are mutually
|
||||
divergent by construction and Forgejo will refuse a `fast-forward-only` merge —
|
||||
correctly. The style is implemented and accepted by `forge_pr_merge`; adopting
|
||||
it needs a one-time reconciliation of the protected branches, which is the
|
||||
owner's decision. Until that happens, promotions are merge commits. Do not
|
||||
attempt to "fix" this with a rebase or a force-push.
|
||||
|
||||
### Hotfix
|
||||
|
||||
1. `forge_branches(create={branch:"hotfix/<slug>", from:"release"})`.
|
||||
2. Fix, PR into `release`, checks green, merge with
|
||||
`confirm_protected_base: true`, tag `v<semver-patch>`.
|
||||
3. **Immediately down-merge** `release` → `main` and `main` → `dev`, as merges,
|
||||
not cherry-picks, in the same session. A hotfix that exists only on `release`
|
||||
is a regression scheduled for the next promotion.
|
||||
|
||||
Note that CI does not gate PRs into `release` — `pr-build.yml` fires only for
|
||||
`dev` and `main`. A hotfix's green evidence therefore has to come from somewhere
|
||||
else; say plainly where it came from rather than implying a check passed.
|
||||
|
||||
A hotfix that touches only one domain deploys only that domain. Check the other
|
||||
half's `/api/version` before assuming the fix is live end to end.
|
||||
|
||||
### When you genuinely need to push directly
|
||||
|
||||
`apply_to_admins` is on, which means these rules bind the owner too. That is
|
||||
deliberate — protection an admin walks straight through is documentation, not a
|
||||
control, and every account here is an admin — but it does remove the escape
|
||||
hatch, so the way back in is worth writing down before it is needed at 3am.
|
||||
|
||||
There is no force flag. The supported route is to lift the rule, do the thing,
|
||||
and put it back:
|
||||
|
||||
```
|
||||
forge_branch_protection(branch="dev", delete=true) # rule gone; direct push allowed
|
||||
git push origin dev # the emergency change
|
||||
forge_branch_protection(branch="dev", apply=true) # restored to policy
|
||||
```
|
||||
|
||||
Both the removal and the restore are audited, which is the point: the hatch is
|
||||
open on the record rather than by a quiet exception. Restore it in the same
|
||||
session — a protection rule "temporarily" removed is how a repo ends up
|
||||
unprotected for a month.
|
||||
|
||||
The failure mode that actually justifies this: if the required status context
|
||||
is ever renamed and no longer matches, **nothing can merge into `dev` or `main`
|
||||
at all**, including by you, because `apply_to_admins` no longer exempts anyone.
|
||||
`forge_pr_await` and `forge_pr_merge` both name that case explicitly when they
|
||||
see it. The fix is to correct `pr-build.yml`'s workflow/job names and
|
||||
`CENTRALIS_REQUIRED_CHECK` together — but if you need to land the fix itself,
|
||||
lift the rule as above.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Never force-push `dev`, `main` or `release`.
|
||||
- Never merge a red PR, hotfix included. Fix CI, or fix the tests in the same PR.
|
||||
- Never merge two green PRs back to back without re-checking the second against
|
||||
the new tip.
|
||||
- If two down-merges or promotions conflict beyond the trivial, stop and report
|
||||
rather than resolving autonomously.
|
||||
- Weekly: `forge_branches()` and report anything `ageing` or `stale`.
|
||||
60
.claude/ownership.md
Normal file
60
.claude/ownership.md
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
# Module map — what can be worked on concurrently
|
||||
|
||||
Used for partitioning work across concurrent subagents. The rule is one owner
|
||||
per file at a time: **never** give two concurrent agents tasks that touch the
|
||||
same module. If two tasks genuinely need the same file, sequence them instead of
|
||||
racing them — a rebase conflict costs more than the parallelism saved.
|
||||
|
||||
Read this before dispatching. It answers one question: *can these two tasks run
|
||||
at the same time?*
|
||||
|
||||
The monorepo's four domains are the natural seam, and they are a real one: FE
|
||||
and BE build and deploy independently, and `deploy.yml` checks per-leg whether a
|
||||
push actually touched that domain. Two agents in two different domains will not
|
||||
collide in CI either.
|
||||
|
||||
## Safe to assign concurrently
|
||||
|
||||
| Module | Files | Notes |
|
||||
|---|---|---|
|
||||
| Backend — API | `backend/api/**`, `backend/app.py` | Route handlers and serialisation. |
|
||||
| Backend — core | `backend/core/**` | Grading, percentiles, climatology. The domain logic. |
|
||||
| Backend — accounts | `backend/accounts/**` | Auth, sessions, OAuth links. |
|
||||
| Backend — daemon | `backend/daemon/**` | Scheduled work. Anything named "prefetch" must not spend the Open-Meteo quota. |
|
||||
| Backend — data & lake | `backend/data/**`, `backend/gen_era5_lake.py`, `backend/gen_cities.py` | |
|
||||
| Backend — migrations | `backend/alembic/**`, `backend/alembic.ini` | One owner, always. Two agents generating revisions produce two heads. |
|
||||
| Frontend — server | `frontend/server/**`, `frontend/app.py`, `frontend/api_client.py` | Go SSR and the Python app. |
|
||||
| Frontend — static | `frontend/static/**` | |
|
||||
| Frontend — templates | `frontend/templates/**`, `frontend/content/**`, `frontend/content.py` | |
|
||||
| Infra — deploy | `infra/deploy/**` (excluding `secrets/`), `infra/docker-compose*.yml` | |
|
||||
| Infra — secrets | `infra/deploy/secrets/**` | SOPS vault. One owner, and never alongside `infra/deploy`. |
|
||||
| Infra — ops | `infra/ops/**`, `infra/lake-iceberg/**` | |
|
||||
| Infra — terraform | `infra/terraform/**` | |
|
||||
| Observability | `observability/**` | Grafana dashboards are provisioned from this JSON; a UI edit is overwritten. |
|
||||
| Docs | `docs/**` | |
|
||||
| Agent tooling | `.claude/hooks/**` | |
|
||||
|
||||
## Serialise — shared spine
|
||||
|
||||
Touched by most changes. A change here is its own task with nothing else running
|
||||
against it.
|
||||
|
||||
| File | Why |
|
||||
|---|---|
|
||||
| `.forgejo/workflows/**` | CI lives **only** here, path-filtered per domain. Every domain's changes route through the same files, so two agents editing workflows conflict even when their domains do not. |
|
||||
| `CLAUDE.md` and every domain `CLAUDE.md` | Read before every change; a stale one is a correctness bug. Small edits, landed fast. |
|
||||
| `CUTOVER-NOTES.md` | The source of truth for what is and is not live. |
|
||||
| `infra/deploy/env-topology.sh` | The single source of truth for where each environment's checkout, branch, stack and ports live. |
|
||||
| `infra/deploy/deploy.sh`, `infra/deploy/stack/deploy-stack.sh` | The deploy contract's entry points. |
|
||||
| `backend/CLAUDE.md` + the `/api/version` contract | FE and BE ship out of lockstep, so `PAYLOAD_VER` discipline is load-bearing. A change to the payload shape is one task spanning both domains — not two concurrent ones. |
|
||||
|
||||
## The recurring collisions
|
||||
|
||||
- **A schema change is not a backend-only task.** It is `backend/alembic` plus
|
||||
whatever reads the column, and if the payload shape moves it is `PAYLOAD_VER`
|
||||
and the frontend too. Scope it as one task across both domains rather than
|
||||
two agents discovering each other mid-flight.
|
||||
- **Adding CI for a domain edits the shared workflows.** Sequence it against any
|
||||
other workflow work.
|
||||
- **`infra/deploy/secrets/` and `infra/deploy/` are one owner between them.**
|
||||
The rendered artifact and the thing that renders it move together.
|
||||
|
|
@ -22,6 +22,11 @@ every change, so a stale one is a correctness bug, not a documentation bug.
|
|||
`dev`, `main` and `release` are protected: **everything is a PR**, for humans and
|
||||
agents alike. Promotion is one PR per hop, `dev` → `main` → `release`.
|
||||
|
||||
`.claude/BRANCHING.md` is the orchestrator's runbook for that flow — the
|
||||
serialised merge queue, promotions, hotfix down-merges — and
|
||||
`.claude/ownership.md` is the module map saying which tasks may run
|
||||
concurrently. Read both before dispatching parallel work.
|
||||
|
||||
**`dev` is a first-class hosted environment, not just an integration branch.**
|
||||
It runs on vps1 — the same box as Forgejo and the monitoring stack — with its
|
||||
own Postgres container, reachable only on the WireGuard mesh
|
||||
|
|
|
|||
|
|
@ -62,11 +62,21 @@ def _apply_sqlite_pragmas(dbapi_conn, _rec):
|
|||
#
|
||||
# pool_size=1/max_overflow=1 (the original setting on both) meant a 3rd
|
||||
# concurrent request per worker had to wait out pool_timeout (~30s default) for a
|
||||
# slot — easy to hit with 4 uvicorn workers x real traffic. 5+5 gives each async
|
||||
# engine headroom for a burst of concurrent account requests without coming close
|
||||
# to Postgres's own connection ceiling (two async engines here plus the sync
|
||||
# engine below, times N workers, all share that budget).
|
||||
_ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=5, pool_pre_ping=True,
|
||||
# slot — easy to hit with 4 uvicorn workers x real traffic. pool_size=5 keeps that
|
||||
# headroom for the steady state.
|
||||
#
|
||||
# max_overflow is deliberately *tighter* than pool_size (2, not 5), because the
|
||||
# overflow is the part nobody budgets for and it is what saturated Postgres on
|
||||
# 2026-07-26. The connection ceiling is shared far more widely than this process
|
||||
# knows: two async engines here plus the sync engine below, times N uvicorn
|
||||
# workers, times the replica count (web autoscales), and prod and beta are two
|
||||
# databases on ONE Postgres instance. At 5+5 the configured worst case across all
|
||||
# of that exceeded the server's max_connections; at 5+2 prod web's ceiling is
|
||||
# 4x14 rather than 4x20. Steady-state behaviour is unchanged — only the burst
|
||||
# ceiling narrows, and a burst that would have opened a 6th..10th connection now
|
||||
# waits on pool_timeout instead of being refused outright by the server (which is
|
||||
# an error, not a wait). See infra/deploy/db/init/20-tuning.sh for the server side.
|
||||
_ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=2, pool_pre_ping=True,
|
||||
pool_recycle=1800, future=True)
|
||||
# Smaller than the async engines: only the notifier leader (one process
|
||||
# cluster-wide, see core/singleton.py) uses the sync engine, driving its periodic
|
||||
|
|
@ -74,7 +84,8 @@ _ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=5, pool_pre_ping=True,
|
|||
# statement_timeout (via psycopg's `options`) bound how long a wedged Postgres
|
||||
# can hang the notifier — it runs for the lifetime of that process's leader
|
||||
# flock, so an indefinite hang here would silently stop every subscription
|
||||
# notification until the process is restarted.
|
||||
# notification until the process is restarted. Left at 2+3: one process holds it,
|
||||
# so it is not multiplied by the worker or replica count.
|
||||
_SYNC_POOL_KWARGS = dict(pool_size=2, max_overflow=3, pool_pre_ping=True,
|
||||
pool_recycle=1800, future=True,
|
||||
connect_args={"connect_timeout": 5,
|
||||
|
|
|
|||
|
|
@ -9,6 +9,28 @@ from sqlalchemy import select
|
|||
|
||||
from accounts import db
|
||||
|
||||
# The server-side ceiling these pools spend from, and the deployment shape that
|
||||
# spends it. Kept here so a pool change that stops fitting is a failing test
|
||||
# rather than a 2am "sorry, too many clients already". Mirrors max_connections in
|
||||
# infra/deploy/db/init/20-tuning.sh. ONE Postgres instance serves prod and beta.
|
||||
MAX_CONNECTIONS = 200
|
||||
PROD_WEB_WORKERS = 4 # uvicorn workers per prod web replica
|
||||
BETA_WEB_WORKERS = 2
|
||||
ASYNC_ENGINES_PER_PROCESS = 2 # RW + RO, both built from _ASYNC_POOL_KWARGS
|
||||
|
||||
|
||||
def _async_ceiling_per_process() -> int:
|
||||
"""Connections one backend process can hold from its async pools at once."""
|
||||
k = db._ASYNC_POOL_KWARGS
|
||||
return ASYNC_ENGINES_PER_PROCESS * (k["pool_size"] + k["max_overflow"])
|
||||
|
||||
|
||||
def _sync_ceiling() -> int:
|
||||
"""The notifier's sync pool. One process holds it cluster-wide (leader flock),
|
||||
so it is NOT multiplied by workers or replicas."""
|
||||
k = db._SYNC_POOL_KWARGS
|
||||
return k["pool_size"] + k["max_overflow"]
|
||||
|
||||
|
||||
def test_defaults_to_sqlite_when_no_database_url():
|
||||
assert db.IS_POSTGRES is False
|
||||
|
|
@ -32,13 +54,18 @@ def test_postgres_pool_sizing_and_sync_timeouts():
|
|||
"""The Postgres engine-construction kwargs, defined at module scope so they're
|
||||
testable without a real Postgres connection (the engines built from them are
|
||||
only actually constructed when IS_POSTGRES, exercised by the docker-compose
|
||||
stack -- see the module docstring). pool_size=1/max_overflow=1 (the old
|
||||
stack -- see the module docstring). pool_size=1/max_overflow=1 (the original
|
||||
setting) let a 3rd concurrent request per worker wait out pool_timeout for a
|
||||
slot; both async engines need real headroom, and the sync engine (the
|
||||
slot, so the steady-state pool needs real headroom; the sync engine (the
|
||||
notifier's, driven off-event-loop) must bound connect + statement time so a
|
||||
wedged Postgres can't hang it forever under its leader flock."""
|
||||
assert db._ASYNC_POOL_KWARGS["pool_size"] >= 5
|
||||
assert db._ASYNC_POOL_KWARGS["max_overflow"] >= 5
|
||||
|
||||
# The overflow is the part nobody budgets for -- it is multiplied by two async
|
||||
# engines x uvicorn workers x replicas, against a ceiling shared with beta.
|
||||
# Prod saturated max_connections on 2026-07-26 with this at 5. Keep it well
|
||||
# under pool_size; the budget test below is the real constraint.
|
||||
assert db._ASYNC_POOL_KWARGS["max_overflow"] <= 2
|
||||
|
||||
assert db._SYNC_POOL_KWARGS["pool_size"] < db._ASYNC_POOL_KWARGS["pool_size"]
|
||||
connect_args = db._SYNC_POOL_KWARGS["connect_args"]
|
||||
|
|
@ -53,6 +80,27 @@ def test_postgres_pool_sizing_and_sync_timeouts():
|
|||
engine.dispose()
|
||||
|
||||
|
||||
def test_pool_budget_fits_under_max_connections():
|
||||
"""At today's replica counts the whole estate's configured worst case must fit
|
||||
inside the server's max_connections, with margin -- prod web + prod worker +
|
||||
beta web + beta worker all draw on ONE Postgres instance. This is the test
|
||||
that would have failed before 2026-07-26, when the configured worst case was
|
||||
170 against a ceiling of 100.
|
||||
|
||||
Note this counts *configured maxima*: every pool at full overflow at the same
|
||||
instant. Prod web autoscaling to its max of 3 replicas is deliberately NOT in
|
||||
this sum -- that peak still does not fit, and closing it needs a connection
|
||||
pooler or a lower autoscale ceiling rather than a bigger number here."""
|
||||
per_process = _async_ceiling_per_process()
|
||||
prod = per_process * PROD_WEB_WORKERS + (per_process + _sync_ceiling())
|
||||
beta = per_process * BETA_WEB_WORKERS + (per_process + _sync_ceiling())
|
||||
|
||||
assert prod + beta <= MAX_CONNECTIONS * 0.8, (
|
||||
f"configured worst case {prod + beta} leaves too little room under "
|
||||
f"max_connections={MAX_CONNECTIONS}"
|
||||
)
|
||||
|
||||
|
||||
def test_read_session_yields_a_working_session():
|
||||
# On SQLite the read session is fully usable (no read-only enforcement); this
|
||||
# just confirms the dependency wiring resolves to a live AsyncSession.
|
||||
|
|
|
|||
|
|
@ -8,10 +8,10 @@
|
|||
# 10-timescaledb.sql enables timescaledb. Settings are written via ALTER SYSTEM
|
||||
# (persisted to postgresql.auto.conf, which the timescaledb image's own
|
||||
# timescaledb-tune postgresql.conf defers to); the container's post-init restart brings
|
||||
# restart-only settings (shared_buffers, …) into effect. NB: never ALTER SYSTEM SET
|
||||
# shared_preload_libraries here — that would land in auto.conf and override the image's
|
||||
# `timescaledb` preload. Init scripts do NOT re-run on an existing volume — to
|
||||
# re-tune later, set DB_MEMORY and run this by hand, then restart:
|
||||
# restart-only settings (shared_buffers, max_connections, …) into effect. NB: never
|
||||
# ALTER SYSTEM SET shared_preload_libraries here — that would land in auto.conf and
|
||||
# override the image's `timescaledb` preload. Init scripts do NOT re-run on an existing
|
||||
# volume — to re-tune later, set DB_MEMORY and run this by hand, then restart:
|
||||
# docker compose exec -e DB_MEMORY=16g db bash /docker-entrypoint-initdb.d/20-tuning.sh
|
||||
# docker compose restart db
|
||||
set -euo pipefail
|
||||
|
|
@ -33,13 +33,30 @@ if [ "$mb" -lt 1024 ]; then mb=8192; fi
|
|||
# maintenance_work_mem 512 MB) and scale linearly on a bigger box.
|
||||
shared_buffers=$((mb / 4)) # 25% — the shared page cache
|
||||
effective_cache=$((mb * 3 / 4)) # 75% — planner's view of total cache (PG + OS)
|
||||
work_mem=$((mb / 128)) # ~64 MB at 8 GB (per-operation; kept modest)
|
||||
maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM
|
||||
|
||||
# work_mem is per *sort operation*, not per connection, so its real cost is
|
||||
# work_mem x concurrent sorts x max_connections (200, below) — and the db container
|
||||
# has a hard memory limit. Left uncapped the ratio gives 128 MB at prod's 16g
|
||||
# budget, which is more than a 200-connection instance should carry: a burst of
|
||||
# heavy sorts can walk into the container limit, and an OOM-killed Postgres is a
|
||||
# far worse day than a refused connection. Cap at 64 MB (dev's 8g is unaffected).
|
||||
work_mem=$((mb / 128))
|
||||
if [ "$work_mem" -gt 64 ]; then work_mem=64; fi
|
||||
if [ "$work_mem" -lt 16 ]; then work_mem=16; fi
|
||||
maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM
|
||||
|
||||
# Connection ceiling. NOT derived from DB_MEMORY: it is bounded by how many pools
|
||||
# the app opens, not by RAM. One instance serves prod and beta, and every backend
|
||||
# process opens three pools (two async engines + the notifier's sync engine, see
|
||||
# backend/accounts/db.py), so the image default of 100 was below the configured
|
||||
# worst case — prod saturated it on 2026-07-26 and started refusing clients with
|
||||
# "sorry, too many clients already". 200 covers today's replica counts including
|
||||
# web's autoscale maximum. ~+1 GB of per-connection overhead at 200.
|
||||
max_conn=200
|
||||
|
||||
echo "[tuning] DB_MEMORY=${budget} -> ${mb}MB: shared_buffers=${shared_buffers}MB" \
|
||||
"effective_cache_size=${effective_cache}MB work_mem=${work_mem}MB" \
|
||||
"maintenance_work_mem=${maint_mem}MB"
|
||||
"maintenance_work_mem=${maint_mem}MB max_connections=${max_conn}"
|
||||
|
||||
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<SQL
|
||||
-- Caching: the shared page cache, and the planner's view of total cache (PG + OS).
|
||||
|
|
@ -50,6 +67,10 @@ ALTER SYSTEM SET effective_cache_size = '${effective_cache}MB';
|
|||
ALTER SYSTEM SET work_mem = '${work_mem}MB';
|
||||
ALTER SYSTEM SET maintenance_work_mem = '${maint_mem}MB';
|
||||
|
||||
-- Connections. Restart-only, like shared_buffers: changing this on a live instance
|
||||
-- takes an ALTER SYSTEM plus a db restart, it does not reload.
|
||||
ALTER SYSTEM SET max_connections = ${max_conn};
|
||||
|
||||
-- Write throughput: fewer, larger checkpoints.
|
||||
ALTER SYSTEM SET wal_buffers = '16MB';
|
||||
ALTER SYSTEM SET min_wal_size = '1GB';
|
||||
|
|
|
|||
Loading…
Reference in a new issue