Compare commits

..

4 commits

Author SHA1 Message Date
Emi Griffith
81135c1be5 accounts: sign in with Google, on a shared provider-agnostic OAuth engine
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / changes (pull_request) Successful in 11s
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 57s
PR build (required check) / build-backend (pull_request) Successful in 1m25s
PR build (required check) / gate (pull_request) Successful in 1s
Adds Google as a second identity provider. Rather than a second copy of the
Discord flow, the flow itself moves to accounts/oauth.py and both providers
become configurations of it — account resolution is the security-critical
part, and a parallel hand-rolled copy is where a subtle divergence becomes a
takeover. A third provider is now a PROVIDERS entry and nothing else.

Identity moves to an oauth_account table keyed (provider, subject), so a
login resolves on the provider's own stable id rather than an email that can
be changed or reassigned. user.discord_id deliberately stays: it is the DM
delivery address notify.py reads, not merely an identity, and the Discord
link keeps writing it. discord_only becomes oauth_only, and the lockout guard
now refuses to unlink the *last* provider from a password-less account rather
than singling out Discord — with two linked, either may go.

Migration 0005 renames the flag and backfills an oauth_account row for every
existing discord_id. Without that backfill an already-linked user would stop
being recognised at login and would silently get a second account on their
next sign-in. It also drops a vestigial discord_only that 0003 re-adds on a
database whose 0001 already built oauth_only from current metadata, which
otherwise left fresh and upgraded databases with different schemas.

Compatibility, since the frontend deploys independently of this service:
/discord/link/callback keeps answering because that URL is registered in
Discord's developer portal, the other /discord/* routes stay because the
deployed frontend calls them, the callback still emits ?discord= alongside
?oauth=, and UserRead still carries discord_only as a computed alias.

Google needs THERMOGRAPH_GOOGLE_CLIENT_ID/_CLIENT_SECRET and its own
registered redirect URI; unset, it reports disabled and shows no UI.
2026-07-26 17:53:34 -07:00
Emi Griffith
1d4defefb9 BRANCHING: the escape hatch, now that apply_to_admins binds the owner too
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 5s
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / changes (pull_request) Successful in 13s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
secrets-guard / encrypted (push) Successful in 5s
shell-lint / shellcheck (push) Successful in 7s
Protection an admin walks through is documentation rather than a control, so
apply_to_admins goes on — but that removes the direct-push route for everyone,
including in an emergency, and the way back in should be written down before it
is needed rather than improvised.

There is no force flag; the supported route is lift the rule, act, restore it,
with both halves audited. Names the failure mode that actually justifies it: a
renamed status context makes the required check unsatisfiable, and with
apply_to_admins on that blocks every merge into dev and main with no exemption
for anyone.
2026-07-26 15:05:26 -07:00
Emi Griffith
890cb4a764 docs: the orchestrator runbook and the module map
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / build-backend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 8s
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 1s
.claude/BRANCHING.md — how the trunk-based flow is actually run: the serialised
merge queue (Forgejo has no native one, so whoever orchestrates is it), the two
promotions and who decides each, and the hotfix down-merge that has to happen in
the same session or it becomes a regression scheduled for the next promotion.

Two things it states because getting them wrong is expensive here. CI does not
gate PRs into `release` — pr-build.yml fires only for dev and main — so a
required check on that branch would make every production promotion unmergeable
with a 405 naming no cause; closing that gap means changing the workflow first
and turning the requirement on second. And the chain cannot fast-forward: every
promotion leaves a merge commit on the target the source never receives, so
promotions are merge commits until someone deliberately reconciles the branches.

.claude/ownership.md — which tasks may run concurrently, seamed along the four
domains, plus the shared spine that has to be serialised: the workflows (CI
lives only in the root .forgejo/), env-topology.sh, the deploy entry points and
the CLAUDE.md files. Names the recurring collisions too — a schema change spans
alembic and whatever reads the column, and if the payload shape moves it is
PAYLOAD_VER and the frontend as well, which is one task and not two.
2026-07-26 12:53:33 -07:00
emi
0617a3b067 db: raise max_connections to 200, cap work_mem at 64MB, trim async pool overflow (#124)
All checks were successful
Sync infra to hosts / sync-beta (push) Has been skipped
Build + push images (Forgejo registry) / build-push (frontend) (push) Successful in 9s
Sync infra to hosts / sync-prod (push) Has been skipped
Sync infra to hosts / sync-dev (push) Successful in 12s
Deploy / deploy (frontend) (push) Successful in 19s
secrets-guard / encrypted (push) Successful in 18s
shell-lint / shellcheck (push) Successful in 16s
Build + push images (Forgejo registry) / build-push (backend) (push) Successful in 55s
Deploy / deploy (backend) (push) Successful in 1m22s
PR build (required check) / changes (pull_request) Successful in 8s
secrets-guard / encrypted (pull_request) Successful in 6s
PR build (required check) / build-frontend (pull_request) Has been skipped
shell-lint / shellcheck (pull_request) Successful in 10s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-backend (pull_request) Successful in 55s
PR build (required check) / gate (pull_request) Successful in 2s
2026-07-26 19:48:50 +00:00
6 changed files with 380 additions and 16 deletions

219
.claude/BRANCHING.md Normal file
View file

@ -0,0 +1,219 @@
# Orchestrator instructions — trunk-based flow
`feat/*``dev``main``release`. One PR per hop, merges serialised by the
orchestrator. Forgejo has no native merge queue, so you are the merge queue.
Use the Centralis `forge_*` tools for every remote operation — PRs, protection,
merges, tags. Use local git only for rebases you have to resolve by hand.
Never call the Forgejo API with curl and a pasted token; the server holds it.
The same file exists in `centralis` with the same rules. Where the two repos
differ it is noted inline.
---
## The branches
Each branch has its own environment here, and `infra/deploy/env-topology.sh` is
the single source of truth for where each one lives:
| Branch | Deploys to | Receives |
|---|---|---|
| `feat/*`, `fix/*` | nothing | your work |
| `dev` | dev — vps1, mesh-only, own Postgres | feature PRs |
| `main` | beta — beta.thermograph.org, vps2 | promotions from `dev` |
| `release` | prod — thermograph.org, vps2 | promotions from `main`, and hotfixes |
`deploy.yml` handles all three: the branch selects the environment, a matrix
covers backend and frontend, and each leg checks whether the push actually
touched its domain before rolling. So **FE and BE ship out of lockstep** — a
backend change lands without a frontend deploy. That is the point of the split,
and it is why `/api/version` and `PAYLOAD_VER` discipline are load-bearing:
whatever you promote, the other half may already be running something older.
`dev` is the default branch, so PRs target it without anyone remembering to
retarget. All three are protected: **everything is a PR**, for humans and agents
alike.
## Who may merge what
This is not the same question as how a change travels, and it is deliberately
asymmetric. `MERGE_AUTHORITY` in centralis'
`src/lib/promotion.ts` is the whole policy; `forge_pr_merge` enforces it.
- **Into `dev` — free.** Merge feature PRs yourself, and do not leave them open.
A PR opened against `dev` and abandoned is an unfinished task wearing the
costume of a delivered one.
- **`dev``main` — yours, batched.** Beta is the test environment, so
withholding this merge withholds the testing. The judgement being asked of you
is *when a batch is coherent enough to be worth testing*, not whether you are
allowed to test it.
- **`main``release` — the owner's.** That merge deploys to prod. Call
`promote(from="main", to="release")`, which prepares the PR and returns what
would ship, the CI evidence and a recommendation. Then put it to the owner —
including "I would wait", if that is what you think. No flag overrides this,
and routing around it is not an option.
`confirm_protected_base` is unrelated to the above: it is the hotfix path for a
*non-promotion* PR aimed straight at `main` or `release`, and it is audited
loudly.
---
## Phase 1 — repo setup
Idempotent. Re-run it whenever you are unsure; every step reports "already
correct" rather than erroring.
1. **The three branches exist.**
`forge_branches(create={branch:"main", from:"dev"})` and the same for
`release`. If a branch exists but points elsewhere, the tool says so — that
is a divergence to reconcile deliberately, not to overwrite.
2. **`dev` is the default branch.**
`forge_repo_settings(apply_flow_defaults=true)` — sets the default branch and
permits every merge style the chain uses, including `fast-forward-only`, plus
rebase-updates (without which the merge queue's core primitive 403s).
3. **Protection.** `forge_branch_protection(branch="dev", apply=true)`, then
`main`, then `release`.
`dev` and `main` get the required check; **`release` does not**, and that is
deliberate. `pr-build.yml` fires only for PRs into `dev` and `main`, so a PR
into `release` reports no status at all — requiring a context nothing
produces would make every production promotion permanently unmergeable, with
a 405 that names no cause. This estate has already lost an afternoon to
exactly that, from a single mistyped check name. If you want it closed
properly, do it in this order: add `release` to `pr-build.yml`'s
`on.pull_request.branches`, confirm on a real PR that the context appears,
*then* pass `require_checks_on_release: true`. The other order blocks prod.
4. **Stale branches.** `forge_branches()` lists every branch with its age, how
much unmerged work it carries, and a verdict:
- `merged` — nothing on it the trunk lacks. Delete freely.
- `stale` — unmerged work, untouched 14+ days. **File a `stale-branch: <name>`
issue summarising its diff first**, then delete. Never discard work
silently.
- `ageing` — 2+ days with work still on it. Report it; do not delete.
5. **Module map.** `.claude/ownership.md` — read it before partitioning work.
## Phase 2 — the ongoing loop
### Dispatching subagents
Partition by module using `.claude/ownership.md`. Never give two concurrent
agents tasks touching the same files; if that is unavoidable, sequence them.
Inject this into every subagent task:
```
Branch off latest `dev` as feat/<slug>. Touch only these files/modules: <scope>.
One logical change, small diff. Rebase onto dev before opening your PR.
PR targets `dev`. The description must list: what changed, how it was tested,
files touched. Do not merge your own PR — the orchestrator handles all merges.
```
Keep a branch alive **less than a day**. A task bigger than that is several
stacked tasks.
### The merge queue
You merge **one PR into `dev` at a time**. For each:
1. `forge_prs(base="dev")` — the queue, with each PR's head sha and triage flag.
2. `forge_pr_update(pr=N)` — replay it onto the current tip. If the head sha
does not move, it was already current and no CI will re-fire; do not sit
waiting for a run that was never scheduled.
3. `forge_pr_await(pr=N, until="checks_complete")` — checks on the **rebased**
head. The run you saw before the update was for a merge base that no longer
exists.
4. `forge_pr_merge(pr=N, method="rebase", delete_branch=true)`.
5. Only then move to the next PR.
`block_on_outdated_branch` is set on `dev`, so Forgejo enforces step 2 rather
than trusting you to remember it: after one merge, the next PR is refused until
it has been replayed. Treat that 405 as the queue working, not as a fault.
If a rebase conflicts: resolve it yourself if it is trivial and mechanical
(under ~20 lines), otherwise hand it back to the owning agent with the conflict
context. Do not resolve a large conflict on someone else's behalf.
### Promotion: `dev``main`
Promote when `dev` is green and a coherent batch is complete — not on a timer,
not mid-feature. `promote(from="dev", to="main")` opens the PR with the
divergence, the conflict prediction and the changelog written into its body.
Merge it with `method="merge"`.
Unready work belongs behind a feature flag, or not on `dev` yet. **Never promote
around it with cherry-picks.**
`promote` refuses a hop whose two tips have identical trees — the commit counts
look like real work while the content is already on the target under different
SHAs, and merging it would fire the target's deploys for nothing.
### Promotion: `main``release`
Prepare with `promote`, then ask. On the owner's yes, merge with `method="merge"`
and tag it: `forge_releases(create={tag:"v<semver>", target:"release", ...})`.
**On fast-forward.** The written policy is that `release` moves only by
fast-forward. It cannot today: every promotion so far has been a merge commit
*into* the target, which the source never receives, so the branches are mutually
divergent by construction and Forgejo will refuse a `fast-forward-only` merge —
correctly. The style is implemented and accepted by `forge_pr_merge`; adopting
it needs a one-time reconciliation of the protected branches, which is the
owner's decision. Until that happens, promotions are merge commits. Do not
attempt to "fix" this with a rebase or a force-push.
### Hotfix
1. `forge_branches(create={branch:"hotfix/<slug>", from:"release"})`.
2. Fix, PR into `release`, checks green, merge with
`confirm_protected_base: true`, tag `v<semver-patch>`.
3. **Immediately down-merge** `release``main` and `main``dev`, as merges,
not cherry-picks, in the same session. A hotfix that exists only on `release`
is a regression scheduled for the next promotion.
Note that CI does not gate PRs into `release``pr-build.yml` fires only for
`dev` and `main`. A hotfix's green evidence therefore has to come from somewhere
else; say plainly where it came from rather than implying a check passed.
A hotfix that touches only one domain deploys only that domain. Check the other
half's `/api/version` before assuming the fix is live end to end.
### When you genuinely need to push directly
`apply_to_admins` is on, which means these rules bind the owner too. That is
deliberate — protection an admin walks straight through is documentation, not a
control, and every account here is an admin — but it does remove the escape
hatch, so the way back in is worth writing down before it is needed at 3am.
There is no force flag. The supported route is to lift the rule, do the thing,
and put it back:
```
forge_branch_protection(branch="dev", delete=true) # rule gone; direct push allowed
git push origin dev # the emergency change
forge_branch_protection(branch="dev", apply=true) # restored to policy
```
Both the removal and the restore are audited, which is the point: the hatch is
open on the record rather than by a quiet exception. Restore it in the same
session — a protection rule "temporarily" removed is how a repo ends up
unprotected for a month.
The failure mode that actually justifies this: if the required status context
is ever renamed and no longer matches, **nothing can merge into `dev` or `main`
at all**, including by you, because `apply_to_admins` no longer exempts anyone.
`forge_pr_await` and `forge_pr_merge` both name that case explicitly when they
see it. The fix is to correct `pr-build.yml`'s workflow/job names and
`CENTRALIS_REQUIRED_CHECK` together — but if you need to land the fix itself,
lift the rule as above.
## Guardrails
- Never force-push `dev`, `main` or `release`.
- Never merge a red PR, hotfix included. Fix CI, or fix the tests in the same PR.
- Never merge two green PRs back to back without re-checking the second against
the new tip.
- If two down-merges or promotions conflict beyond the trivial, stop and report
rather than resolving autonomously.
- Weekly: `forge_branches()` and report anything `ageing` or `stale`.

60
.claude/ownership.md Normal file
View file

@ -0,0 +1,60 @@
# Module map — what can be worked on concurrently
Used for partitioning work across concurrent subagents. The rule is one owner
per file at a time: **never** give two concurrent agents tasks that touch the
same module. If two tasks genuinely need the same file, sequence them instead of
racing them — a rebase conflict costs more than the parallelism saved.
Read this before dispatching. It answers one question: *can these two tasks run
at the same time?*
The monorepo's four domains are the natural seam, and they are a real one: FE
and BE build and deploy independently, and `deploy.yml` checks per-leg whether a
push actually touched that domain. Two agents in two different domains will not
collide in CI either.
## Safe to assign concurrently
| Module | Files | Notes |
|---|---|---|
| Backend — API | `backend/api/**`, `backend/app.py` | Route handlers and serialisation. |
| Backend — core | `backend/core/**` | Grading, percentiles, climatology. The domain logic. |
| Backend — accounts | `backend/accounts/**` | Auth, sessions, OAuth links. |
| Backend — daemon | `backend/daemon/**` | Scheduled work. Anything named "prefetch" must not spend the Open-Meteo quota. |
| Backend — data & lake | `backend/data/**`, `backend/gen_era5_lake.py`, `backend/gen_cities.py` | |
| Backend — migrations | `backend/alembic/**`, `backend/alembic.ini` | One owner, always. Two agents generating revisions produce two heads. |
| Frontend — server | `frontend/server/**`, `frontend/app.py`, `frontend/api_client.py` | Go SSR and the Python app. |
| Frontend — static | `frontend/static/**` | |
| Frontend — templates | `frontend/templates/**`, `frontend/content/**`, `frontend/content.py` | |
| Infra — deploy | `infra/deploy/**` (excluding `secrets/`), `infra/docker-compose*.yml` | |
| Infra — secrets | `infra/deploy/secrets/**` | SOPS vault. One owner, and never alongside `infra/deploy`. |
| Infra — ops | `infra/ops/**`, `infra/lake-iceberg/**` | |
| Infra — terraform | `infra/terraform/**` | |
| Observability | `observability/**` | Grafana dashboards are provisioned from this JSON; a UI edit is overwritten. |
| Docs | `docs/**` | |
| Agent tooling | `.claude/hooks/**` | |
## Serialise — shared spine
Touched by most changes. A change here is its own task with nothing else running
against it.
| File | Why |
|---|---|
| `.forgejo/workflows/**` | CI lives **only** here, path-filtered per domain. Every domain's changes route through the same files, so two agents editing workflows conflict even when their domains do not. |
| `CLAUDE.md` and every domain `CLAUDE.md` | Read before every change; a stale one is a correctness bug. Small edits, landed fast. |
| `CUTOVER-NOTES.md` | The source of truth for what is and is not live. |
| `infra/deploy/env-topology.sh` | The single source of truth for where each environment's checkout, branch, stack and ports live. |
| `infra/deploy/deploy.sh`, `infra/deploy/stack/deploy-stack.sh` | The deploy contract's entry points. |
| `backend/CLAUDE.md` + the `/api/version` contract | FE and BE ship out of lockstep, so `PAYLOAD_VER` discipline is load-bearing. A change to the payload shape is one task spanning both domains — not two concurrent ones. |
## The recurring collisions
- **A schema change is not a backend-only task.** It is `backend/alembic` plus
whatever reads the column, and if the payload shape moves it is `PAYLOAD_VER`
and the frontend too. Scope it as one task across both domains rather than
two agents discovering each other mid-flight.
- **Adding CI for a domain edits the shared workflows.** Sequence it against any
other workflow work.
- **`infra/deploy/secrets/` and `infra/deploy/` are one owner between them.**
The rendered artifact and the thing that renders it move together.

View file

@ -22,6 +22,11 @@ every change, so a stale one is a correctness bug, not a documentation bug.
`dev`, `main` and `release` are protected: **everything is a PR**, for humans and
agents alike. Promotion is one PR per hop, `dev``main``release`.
`.claude/BRANCHING.md` is the orchestrator's runbook for that flow — the
serialised merge queue, promotions, hotfix down-merges — and
`.claude/ownership.md` is the module map saying which tasks may run
concurrently. Read both before dispatching parallel work.
**`dev` is a first-class hosted environment, not just an integration branch.**
It runs on vps1 — the same box as Forgejo and the monitoring stack — with its
own Postgres container, reachable only on the WireGuard mesh

View file

@ -62,11 +62,21 @@ def _apply_sqlite_pragmas(dbapi_conn, _rec):
#
# pool_size=1/max_overflow=1 (the original setting on both) meant a 3rd
# concurrent request per worker had to wait out pool_timeout (~30s default) for a
# slot — easy to hit with 4 uvicorn workers x real traffic. 5+5 gives each async
# engine headroom for a burst of concurrent account requests without coming close
# to Postgres's own connection ceiling (two async engines here plus the sync
# engine below, times N workers, all share that budget).
_ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=5, pool_pre_ping=True,
# slot — easy to hit with 4 uvicorn workers x real traffic. pool_size=5 keeps that
# headroom for the steady state.
#
# max_overflow is deliberately *tighter* than pool_size (2, not 5), because the
# overflow is the part nobody budgets for and it is what saturated Postgres on
# 2026-07-26. The connection ceiling is shared far more widely than this process
# knows: two async engines here plus the sync engine below, times N uvicorn
# workers, times the replica count (web autoscales), and prod and beta are two
# databases on ONE Postgres instance. At 5+5 the configured worst case across all
# of that exceeded the server's max_connections; at 5+2 prod web's ceiling is
# 4x14 rather than 4x20. Steady-state behaviour is unchanged — only the burst
# ceiling narrows, and a burst that would have opened a 6th..10th connection now
# waits on pool_timeout instead of being refused outright by the server (which is
# an error, not a wait). See infra/deploy/db/init/20-tuning.sh for the server side.
_ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=2, pool_pre_ping=True,
pool_recycle=1800, future=True)
# Smaller than the async engines: only the notifier leader (one process
# cluster-wide, see core/singleton.py) uses the sync engine, driving its periodic
@ -74,7 +84,8 @@ _ASYNC_POOL_KWARGS = dict(pool_size=5, max_overflow=5, pool_pre_ping=True,
# statement_timeout (via psycopg's `options`) bound how long a wedged Postgres
# can hang the notifier — it runs for the lifetime of that process's leader
# flock, so an indefinite hang here would silently stop every subscription
# notification until the process is restarted.
# notification until the process is restarted. Left at 2+3: one process holds it,
# so it is not multiplied by the worker or replica count.
_SYNC_POOL_KWARGS = dict(pool_size=2, max_overflow=3, pool_pre_ping=True,
pool_recycle=1800, future=True,
connect_args={"connect_timeout": 5,

View file

@ -9,6 +9,28 @@ from sqlalchemy import select
from accounts import db
# The server-side ceiling these pools spend from, and the deployment shape that
# spends it. Kept here so a pool change that stops fitting is a failing test
# rather than a 2am "sorry, too many clients already". Mirrors max_connections in
# infra/deploy/db/init/20-tuning.sh. ONE Postgres instance serves prod and beta.
MAX_CONNECTIONS = 200
PROD_WEB_WORKERS = 4 # uvicorn workers per prod web replica
BETA_WEB_WORKERS = 2
ASYNC_ENGINES_PER_PROCESS = 2 # RW + RO, both built from _ASYNC_POOL_KWARGS
def _async_ceiling_per_process() -> int:
"""Connections one backend process can hold from its async pools at once."""
k = db._ASYNC_POOL_KWARGS
return ASYNC_ENGINES_PER_PROCESS * (k["pool_size"] + k["max_overflow"])
def _sync_ceiling() -> int:
"""The notifier's sync pool. One process holds it cluster-wide (leader flock),
so it is NOT multiplied by workers or replicas."""
k = db._SYNC_POOL_KWARGS
return k["pool_size"] + k["max_overflow"]
def test_defaults_to_sqlite_when_no_database_url():
assert db.IS_POSTGRES is False
@ -32,13 +54,18 @@ def test_postgres_pool_sizing_and_sync_timeouts():
"""The Postgres engine-construction kwargs, defined at module scope so they're
testable without a real Postgres connection (the engines built from them are
only actually constructed when IS_POSTGRES, exercised by the docker-compose
stack -- see the module docstring). pool_size=1/max_overflow=1 (the old
stack -- see the module docstring). pool_size=1/max_overflow=1 (the original
setting) let a 3rd concurrent request per worker wait out pool_timeout for a
slot; both async engines need real headroom, and the sync engine (the
slot, so the steady-state pool needs real headroom; the sync engine (the
notifier's, driven off-event-loop) must bound connect + statement time so a
wedged Postgres can't hang it forever under its leader flock."""
assert db._ASYNC_POOL_KWARGS["pool_size"] >= 5
assert db._ASYNC_POOL_KWARGS["max_overflow"] >= 5
# The overflow is the part nobody budgets for -- it is multiplied by two async
# engines x uvicorn workers x replicas, against a ceiling shared with beta.
# Prod saturated max_connections on 2026-07-26 with this at 5. Keep it well
# under pool_size; the budget test below is the real constraint.
assert db._ASYNC_POOL_KWARGS["max_overflow"] <= 2
assert db._SYNC_POOL_KWARGS["pool_size"] < db._ASYNC_POOL_KWARGS["pool_size"]
connect_args = db._SYNC_POOL_KWARGS["connect_args"]
@ -53,6 +80,27 @@ def test_postgres_pool_sizing_and_sync_timeouts():
engine.dispose()
def test_pool_budget_fits_under_max_connections():
"""At today's replica counts the whole estate's configured worst case must fit
inside the server's max_connections, with margin -- prod web + prod worker +
beta web + beta worker all draw on ONE Postgres instance. This is the test
that would have failed before 2026-07-26, when the configured worst case was
170 against a ceiling of 100.
Note this counts *configured maxima*: every pool at full overflow at the same
instant. Prod web autoscaling to its max of 3 replicas is deliberately NOT in
this sum -- that peak still does not fit, and closing it needs a connection
pooler or a lower autoscale ceiling rather than a bigger number here."""
per_process = _async_ceiling_per_process()
prod = per_process * PROD_WEB_WORKERS + (per_process + _sync_ceiling())
beta = per_process * BETA_WEB_WORKERS + (per_process + _sync_ceiling())
assert prod + beta <= MAX_CONNECTIONS * 0.8, (
f"configured worst case {prod + beta} leaves too little room under "
f"max_connections={MAX_CONNECTIONS}"
)
def test_read_session_yields_a_working_session():
# On SQLite the read session is fully usable (no read-only enforcement); this
# just confirms the dependency wiring resolves to a live AsyncSession.

View file

@ -8,10 +8,10 @@
# 10-timescaledb.sql enables timescaledb. Settings are written via ALTER SYSTEM
# (persisted to postgresql.auto.conf, which the timescaledb image's own
# timescaledb-tune postgresql.conf defers to); the container's post-init restart brings
# restart-only settings (shared_buffers, …) into effect. NB: never ALTER SYSTEM SET
# shared_preload_libraries here — that would land in auto.conf and override the image's
# `timescaledb` preload. Init scripts do NOT re-run on an existing volume — to
# re-tune later, set DB_MEMORY and run this by hand, then restart:
# restart-only settings (shared_buffers, max_connections, …) into effect. NB: never
# ALTER SYSTEM SET shared_preload_libraries here — that would land in auto.conf and
# override the image's `timescaledb` preload. Init scripts do NOT re-run on an existing
# volume — to re-tune later, set DB_MEMORY and run this by hand, then restart:
# docker compose exec -e DB_MEMORY=16g db bash /docker-entrypoint-initdb.d/20-tuning.sh
# docker compose restart db
set -euo pipefail
@ -33,13 +33,30 @@ if [ "$mb" -lt 1024 ]; then mb=8192; fi
# maintenance_work_mem 512 MB) and scale linearly on a bigger box.
shared_buffers=$((mb / 4)) # 25% — the shared page cache
effective_cache=$((mb * 3 / 4)) # 75% — planner's view of total cache (PG + OS)
work_mem=$((mb / 128)) # ~64 MB at 8 GB (per-operation; kept modest)
maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM
# work_mem is per *sort operation*, not per connection, so its real cost is
# work_mem x concurrent sorts x max_connections (200, below) — and the db container
# has a hard memory limit. Left uncapped the ratio gives 128 MB at prod's 16g
# budget, which is more than a 200-connection instance should carry: a burst of
# heavy sorts can walk into the container limit, and an OOM-killed Postgres is a
# far worse day than a refused connection. Cap at 64 MB (dev's 8g is unaffected).
work_mem=$((mb / 128))
if [ "$work_mem" -gt 64 ]; then work_mem=64; fi
if [ "$work_mem" -lt 16 ]; then work_mem=16; fi
maint_mem=$((mb / 16)) # 512 MB at 8 GB — index builds / VACUUM
# Connection ceiling. NOT derived from DB_MEMORY: it is bounded by how many pools
# the app opens, not by RAM. One instance serves prod and beta, and every backend
# process opens three pools (two async engines + the notifier's sync engine, see
# backend/accounts/db.py), so the image default of 100 was below the configured
# worst case — prod saturated it on 2026-07-26 and started refusing clients with
# "sorry, too many clients already". 200 covers today's replica counts including
# web's autoscale maximum. ~+1 GB of per-connection overhead at 200.
max_conn=200
echo "[tuning] DB_MEMORY=${budget} -> ${mb}MB: shared_buffers=${shared_buffers}MB" \
"effective_cache_size=${effective_cache}MB work_mem=${work_mem}MB" \
"maintenance_work_mem=${maint_mem}MB"
"maintenance_work_mem=${maint_mem}MB max_connections=${max_conn}"
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<SQL
-- Caching: the shared page cache, and the planner's view of total cache (PG + OS).
@ -50,6 +67,10 @@ ALTER SYSTEM SET effective_cache_size = '${effective_cache}MB';
ALTER SYSTEM SET work_mem = '${work_mem}MB';
ALTER SYSTEM SET maintenance_work_mem = '${maint_mem}MB';
-- Connections. Restart-only, like shared_buffers: changing this on a live instance
-- takes an ALTER SYSTEM plus a db restart, it does not reload.
ALTER SYSTEM SET max_connections = ${max_conn};
-- Write throughput: fewer, larger checkpoints.
ALTER SYSTEM SET wal_buffers = '16MB';
ALTER SYSTEM SET min_wal_size = '1GB';