Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Read each stream pass in one database session and turn off the pool pre-ping (CHOO-3667) (#1111)
Abel Dantas0ac8ec485d1e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4155b0a3e..bd1343ab9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -142,6 +142,20 @@ version of their own to them without also giving them a release of their own.compares it only when someone signs in.#### Changed+- **An agent's stream reads in one database session per pass, and pooled+ connections are no longer pinged before use.** A pass that delivered a+ message opened a session for each of its reads, six pool checkouts and six+ transactions, and the pre-ping added a transaction to every checkout. The+ reads of a pass now share one session, for an agent on its own socket and+ for each agent an agent controller serves. `DB_POOL_PRE_PING` now defaults+ to `false`: after a database restart the first query on each pod fails, along+ with any query running at the time, and the pool then replaces every+ connection it held. Set it to `true` behind+ anything that drops idle connections. In a load test of 300 agents at 10+ messages a second on 2 pods, CPU per message fell by a quarter and database+ transactions per message by more than two thirds. At most half of a pod's+ pool goes to streams' passes at once, so requests still get a connection+ when the pod's CPU is saturated.- **A cloud machine can move to another instance type and keep its disk.**The hosted controller now launches every new instance of a machine as theinstance type in its `gateway.json`, at the same moment it already moves thediff --git a/core/switch_core/bridges/agent/protocol/controller_stream.py b/core/switch_core/bridges/agent/protocol/controller_stream.pyindex a7dfee7b5..4562bc18c 100644--- a/core/switch_core/bridges/agent/protocol/controller_stream.py+++ b/core/switch_core/bridges/agent/protocol/controller_stream.py@@ -400,52 +400,57 @@ class _ControllerStream:self._tenant, agent_id, attached.reader_id, room_id)attached.counted.add(room_id)- # Take the number of the newest of the agent's own events before- # reading, and read no further than it. A read that finds nothing moves- # the cursor up to this number. An event added to `agent_events` while- # the read runs is numbered above it, so the cursor does not skip that- # event.- bound = await self._buffer.head(self._tenant, agent_id)- try:- pending = await self._buffer.read_from(+ # The reads below share one session (`EventBuffer.one_session`): one+ # pool checkout and one transaction for the agent's pass, not one per+ # read. The frames are built after it, and returned rather than+ # yielded, so the connection is back in the pool before any is sent.+ async with self._buffer.one_session(self._tenant):+ # Take the number of the newest of the agent's own events before+ # reading, and read no further than it. A read that finds nothing moves+ # the cursor up to this number. An event added to `agent_events` while+ # the read runs is numbered above it, so the cursor does not skip that+ # event.+ bound = await self._buffer.head(self._tenant, agent_id)+ try:+ pending = await self._buffer.read_from(+ self._tenant,+ agent_id,+ attached.cursor,+ limit=CATCH_UP_BATCH,+ through_seq=bound,+ )+ except CursorExpiredError as exc:+ resumed_at = exc.resume_after+ attached.cursor = resumed_at+ self._conn.sent_cursors[agent_id] = resumed_at+ return [+ frame(+ "agent.gap",+ {+ "agent_id": agent_id,+ "from_sequence": exc.requested,+ "resumed_at": resumed_at,+ "rooms": list(exc.rooms),+ "all_rooms": False,+ "reason": str(exc),+ },+ )+ ]+ if not pending:+ # Events above the cursor that are no longer held — a room the+ # agent left takes its events with it — are passed, or every pass+ # would find work it can never read and spin.+ attached.cursor = max(attached.cursor, bound)+ self._conn.sent_cursors[agent_id] = attached.cursor+ return []+ # The unread counts sent with the events below, counted for the whole+ # batch at once, so that catching up on a few hundred events costs a+ # few statements per room rather than several per event.+ missed = await self._buffer.unread_batch(self._tenant,agent_id,- attached.cursor,- limit=CATCH_UP_BATCH,- through_seq=bound,+ [(item.room_id, item.room_seq) for item in pending if item.notifiable],)- except CursorExpiredError as exc:- resumed_at = exc.resume_after- attached.cursor = resumed_at- self._conn.sent_cursors[agent_id] = resumed_at- return [- frame(- "agent.gap",- {- "agent_id": agent_id,- "from_sequence": exc.requested,- "resumed_at": resumed_at,- "rooms": list(exc.rooms),- "all_rooms": False,- "reason": str(exc),- },- )- ]- if not pending:- # Events above the cursor that are no longer held — a room the- # agent left takes its events with it — are passed, or every pass- # would find work it can never read and spin.- attached.cursor = max(attached.cursor, bound)- self._conn.sent_cursors[agent_id] = attached.cursor- return []- # The unread counts sent with the events below, counted for the whole- # batch at once, so that catching up on a few hundred events costs a- # few statements per room rather than several per event.- missed = await self._buffer.unread_batch(- self._tenant,- agent_id,- [(item.room_id, item.room_seq) for item in pending if item.notifiable],- )frames: list[ControllerFrame] = []for item in pending:data = item.event.model_dump(mode="json")diff --git a/core/switch_core/bridges/agent/protocol/event_buffer.py b/core/switch_core/bridges/agent/protocol/event_buffer.pyindex d5e9fcd84..b3340d088 100644--- a/core/switch_core/bridges/agent/protocol/event_buffer.py+++ b/core/switch_core/bridges/agent/protocol/event_buffer.py@@ -52,7 +52,8 @@ from __future__ import annotationsimport asyncioimport loggingimport time-from collections.abc import Iterable+from collections.abc import AsyncIterator, Iterable+from contextlib import asynccontextmanager, nullcontextfrom dataclasses import dataclass, fieldfrom datetime import UTC, datetime, timedeltafrom typing import TYPE_CHECKING, Any, Protocol, cast@@ -103,7 +104,11 @@ from switch_core.db.models import (PendingAttachmentGroup,Room,)-from switch_core.db.session_scope import tenant_session+from switch_core.db.session_scope import (+ holds_shared_session,+ shared_tenant_session,+ tenant_session,+)from switch_core.observability.catalogue import (AGENT_BUFFER_SCAN_DURATION,AGENT_BUFFER_SCANNED,@@ -373,6 +378,12 @@ _UNREAD_SQL = text().bindparams(bindparam("throughs", type_=ARRAY(BigInteger)))+def session_slots_for(pool_size: int, max_overflow: int) -> int:+ """How many `EventBuffer.one_session` blocks may hold a connection at once:+ half the pool, so requests still find one when the pod is saturated."""+ return max(1, (pool_size + max_overflow) // 2)++class EventBuffer:def __init__(self,@@ -381,8 +392,14 @@ class EventBuffer:sequence_base: int,rooms: RoomAnnouncements,retention_seconds: float = DEFAULT_RETENTION_SECONDS,+ session_slots: int | None,) -> None:self._session_factory = session_factory+ # How many `one_session` blocks may hold a connection at once in this+ # process; None for no limit. See `one_session`.+ self._session_slots = (+ asyncio.Semaphore(session_slots) if session_slots is not None else None+ )# `sequence_base` comes from a database sequence that advances on every# boot (`main.py`), so it is above every number an earlier boot handed# out. An agent's first own event is numbered from it. A client still holding@@ -402,6 +419,35 @@ class EventBuffer:# themselves are always read from `agent_events`.self._notify: dict[str, asyncio.Event] = {}+ @asynccontextmanager+ async def one_session(self, tenant_id: str) -> AsyncIterator[AsyncSession]:+ """Everything this buffer reads and writes for `tenant_id` inside the+ block, from the task that opened it, in one session.++ Each method here opens a session of its own, so a stream's pass that+ delivers, which asks for the agent's rooms, its head, its own events,+ what was dropped after its position, its rooms' conversation and its+ unread counts, cost six pool checkouts and as many transactions. In one+ session it costs one of each. `shared_tenant_session` says what the+ block may contain; a block must not open another.++ At most `session_slots` blocks hold a connection at once, and a block+ over that waits here holding nothing. When a pod's CPU is saturated,+ every await inside a block takes longer, so without the limit the+ streams' passes come to hold most of the pool, and a request (a send, a+ context read, authentication) waits out the pool timeout and fails.+ """+ # Checked before taking a slot: a nested block would otherwise wait on+ # its own slot once every slot is taken, a deadlock seen only under load.+ if holds_shared_session(self._session_factory, tenant_id):+ raise RuntimeError("one_session opened inside another in the same task")+ slots = self._session_slots or nullcontext()+ async with (+ slots,+ shared_tenant_session(self._session_factory, tenant_id) as session,+ ):+ yield session+# ------------------------------------------------------------------# Producing# ------------------------------------------------------------------diff --git a/core/switch_core/bridges/agent/protocol/stream.py b/core/switch_core/bridges/agent/protocol/stream.pyindex aeacaf312..5d86251a4 100644--- a/core/switch_core/bridges/agent/protocol/stream.py+++ b/core/switch_core/bridges/agent/protocol/stream.py@@ -348,24 +348,62 @@ async def _event_stream(await buffer.ensure_counting(tenant, agent_id, conn.id, room_id)counted.add(room_id)- conversation_rooms: set[str] = set()- if conn.reads_conversation:- conversation_rooms = await _place_in_conversation(- conn, registry, buffer, tenant- )- # Before the read, so a row written while it runs wakes the wait- # below rather than being left for the keepalive.- rooms.watch(conversation_rooms)-+ # The reads of a pass share one session, so they cost one pool+ # checkout and one transaction rather than one of each per read+ # (`EventBuffer.one_session`). Nothing is yielded inside it: a gap+ # goes out after the block has given its connection back.try:- pending, looked = await buffer.read(- tenant,- agent_id,- conn.position,- conversation_rooms=conversation_rooms,- notifiable_only=not conn.reads_conversation,- limit=CATCH_UP_BATCH,- )+ async with buffer.one_session(tenant):+ conversation_rooms: set[str] = set()+ if conn.reads_conversation:+ conversation_rooms = await _place_in_conversation(+ conn, registry, buffer, tenant+ )+ # Before the read, so a row written while it runs wakes the+ # wait below rather than being left for the keepalive.+ rooms.watch(conversation_rooms)++ pending, looked = await buffer.read(+ tenant,+ agent_id,+ conn.position,+ conversation_rooms=conversation_rooms,+ notifiable_only=not conn.reads_conversation,+ limit=CATCH_UP_BATCH,+ )++ # Room coverage is evaluated per event rather than up+ # front: a sibling claiming a room must take effect+ # immediately, without tearing the stream down. An event+ # for a room this connection does not cover is not sent,+ # but the position still moves past it, in order: only once+ # every event before it has gone out, so the position never+ # says the client was sent an event that is still to come.+ # `covered` is the plan when the batch is read, which the+ # tickets are minted for: room messages are in it because+ # `_conversation_rooms` only returned rooms this connection+ # covers. Each event is checked again just before it is+ # sent, because the loop suspends at every yield and a+ # sibling can claim the room while it does. The check reads+ # the registry in memory, so it costs no query.+ covered = [+ item.source != OWN_EVENTS or registry.covers(conn, item.room_id)+ for item in pending+ ]+ tickets = await _tickets_for(conn, buffer, tenant, pending, covered)+ # The unread counts sent with the events below, counted for+ # the whole batch at once, so that catching up on a few+ # hundred events costs a few statements per room rather+ # than several per event.+ missed = await buffer.unread_batch(+ tenant,+ agent_id,+ [+ (item.room_id, item.room_seq)+ for item, send in zip(pending, covered, strict=True)+ if send and item.notifiable+ ],+ )except CursorExpiredError as exc:conn.position[OWN_EVENTS] = exc.resume_afteryield _frame(@@ -382,36 +420,6 @@ async def _event_stream()continue- # Room coverage is evaluated per event rather than up front: a- # sibling claiming a room must take effect immediately, without- # tearing the stream down. An event for a room this connection does- # not cover is not sent, but the position still moves past it, in- # order: only once every event before it has gone out, so the- # position never says the client was sent an event that is still- # to come. `covered` is the plan when the batch is read, which the- # tickets are minted for: room messages are in it because- # `_conversation_rooms` only returned rooms this connection covers.- # Each event is checked again just before it is sent, because the- # loop suspends at every yield and a sibling can claim the room- # while it does. The check reads the registry in memory, so it- # costs no query.- covered = [- item.source != OWN_EVENTS or registry.covers(conn, item.room_id)- for item in pending- ]- tickets = await _tickets_for(conn, buffer, tenant, pending, covered)- # The unread counts sent with the events below, counted for the- # whole batch at once, so that catching up on a few hundred events- # costs a few statements per room rather than several per event.- missed = await buffer.unread_batch(- tenant,- agent_id,- [- (item.room_id, item.room_seq)- for item, send in zip(pending, covered, strict=True)- if send and item.notifiable- ],- )delivered = Falseindex = -1for item, send in zip(pending, covered, strict=True):diff --git a/core/switch_core/config.py b/core/switch_core/config.pyindex 4799b02ef..f12a55c33 100644--- a/core/switch_core/config.py+++ b/core/switch_core/config.py@@ -734,7 +734,19 @@ class SwitchConfig(BaseSettings):db_pool_size: int = 30db_max_overflow: int = 10db_pool_recycle: int = 1800- db_pool_pre_ping: bool = True+ # Off by default. On, every checkout first pings its connection, and+ # SQLAlchemy's asyncpg ping is a BEGIN, an empty statement and a ROLLBACK:+ # three round trips and a transaction before any work, which under agent+ # load was over a third of all the transactions core ran. What it catches+ # is a connection that died while it sat in the pool, as every pooled+ # connection does when the database restarts or fails over. Without it the+ # first query on such a connection fails, and SQLAlchemy then replaces+ # every connection the pool held. So on each pod the first query fails+ # (`test_pool_pre_ping.py`), along with any query that was running when the+ # database went, which fails with the ping too. How soon a connection left+ # open by a failover is known to be dead is up to `db_tcp_keepalive_*` and+ # `db_tcp_user_timeout` below, with the ping or without it.+ db_pool_pre_ping: bool = False# SQLAlchemy's default is 30s, long enough that pool starvation surfaces as# unexplained latency rather than an error. Fail fast and loudly instead.db_pool_timeout: float = 5.0diff --git a/core/switch_core/db/session_scope.py b/core/switch_core/db/session_scope.pyindex 56f4a1d49..f0ff7bca2 100644--- a/core/switch_core/db/session_scope.py+++ b/core/switch_core/db/session_scope.py@@ -54,14 +54,41 @@ session's first query.from __future__ import annotations+import asynciofrom collections.abc import AsyncIteratorfrom contextlib import asynccontextmanager+from contextvars import ContextVar+from dataclasses import dataclassfrom sqlalchemy.ext.asyncio import AsyncSession, async_sessionmakerfrom switch_core.tenant_context import tenant_scope+@dataclass(frozen=True)+class _Lent:+ """The session a `shared_tenant_session` block lends, and who to."""++ session_factory: async_sessionmaker[AsyncSession]+ tenant_id: str+ task: asyncio.Task[object] | None+ session: AsyncSession++ def serves(+ self, session_factory: async_sessionmaker[AsyncSession], tenant_id: str+ ) -> bool:+ # Only the task that opened the block: a task started inside it copies+ # this variable, and an `AsyncSession` must not be used by two tasks.+ return (+ self.session_factory is session_factory+ and self.tenant_id == tenant_id+ and self.task is asyncio.current_task()+ )+++_lent: ContextVar[_Lent | None] = ContextVar("switch_lent_session", default=None)++@asynccontextmanagerasync def tenant_session(session_factory: async_sessionmaker[AsyncSession], tenant_id: str@@ -77,7 +104,69 @@ async def tenant_session(already-open transaction changes nothing about that transaction — thestartup bootstrap seeding did exactly that and wrote a `tenant_members`row on a connection that had never been told which tenant it was for.++ Inside a `shared_tenant_session` block for the same factory and tenant,+ opened by the same task, the session is that block's rather than a new one."""+ lent = _lent.get()+ if lent is not None and lent.serves(session_factory, tenant_id):+ with tenant_scope(tenant_id):+ yield lent.session+ returnwith tenant_scope(tenant_id):async with session_factory() as session:yield session+++def holds_shared_session(+ session_factory: async_sessionmaker[AsyncSession], tenant_id: str+) -> bool:+ """Whether this task is inside a `shared_tenant_session` block for this+ factory and tenant."""+ lent = _lent.get()+ return lent is not None and lent.serves(session_factory, tenant_id)+++@asynccontextmanager+async def shared_tenant_session(+ session_factory: async_sessionmaker[AsyncSession], tenant_id: str+) -> AsyncIterator[AsyncSession]:+ """One session for a stretch of work that would otherwise open several.++ Every `tenant_session` opened inside the block for the same factory and+ tenant, by the task that opened the block, is this session. Work that+ opened a session per question then costs one pool checkout and one+ transaction instead of one of each per question. A `commit` inside the+ block ends the transaction, not the session: the next statement begins a+ new one, and `after_begin` stamps the tenant on it as on any other.++ Leaving an inner `tenant_session` closes and rolls back nothing. An+ uncommitted write in one inner block is committed by the next inner+ `commit`, and an inner `rollback` (as in an `except` around a write) rolls+ back the whole shared transaction. A write added to such a block has to+ commit or roll back with that in mind.++ The block holds a connection from its first statement to its end, so it+ must contain database work only. Nothing in it may wait on a client, a+ socket or a timer, and a generator must not yield inside it, or the+ connection is held for as long as its consumer takes to come back. A block+ must not open another for the same tenant: that raises.+ """+ if holds_shared_session(session_factory, tenant_id):+ raise RuntimeError(+ "a shared tenant session is already open in this task for this tenant"+ )+ async with tenant_session(session_factory, tenant_id) as session:+ token = _lent.set(+ _Lent(session_factory, tenant_id, asyncio.current_task(), session)+ )+ try:+ yield session+ finally:+ try:+ _lent.reset(token)+ except ValueError:+ # Unwound by GeneratorExit in another context, a frame being+ # garbage collected: nothing there to restore, for the reason+ # `tenant_context._held` gives.+ passdiff --git a/core/switch_core/main.py b/core/switch_core/main.pyindex 79e455632..69c7b2585 100644--- a/core/switch_core/main.py+++ b/core/switch_core/main.py@@ -36,7 +36,10 @@ from switch_core.bridges.agent.protocol.agent_core import AgentCorefrom switch_core.bridges.agent.protocol.agent_event_listener import AgentEventListenerfrom switch_core.bridges.agent.protocol.connection_rows import SharedConnectionsfrom switch_core.bridges.agent.protocol.controller_presence import ControllerPresence-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer+from switch_core.bridges.agent.protocol.event_buffer import (+ EventBuffer,+ session_slots_for,+)from switch_core.bridges.agent.registration_bootstrap import (BOOTSTRAP_KEY_LABEL,BOOTSTRAP_KEY_TYPE,@@ -761,6 +764,7 @@ async def run(config: SwitchConfig, process_id: str) -> None:sequence_base=event_boot << 32,rooms=message_listener,retention_seconds=config.agent_event_retention_seconds,+ session_slots=session_slots_for(config.db_pool_size, config.db_max_overflow),)# Listens for the `NOTIFY` that each insert into `agent_events` sends, and# wakes this process's streams for that agent, whichever process wrote it.diff --git a/core/tests/benchmarks/server.py b/core/tests/benchmarks/server.pyindex a3278a1c6..71a2d75f2 100644--- a/core/tests/benchmarks/server.py+++ b/core/tests/benchmarks/server.py@@ -36,7 +36,10 @@ from switch_core.bridges.agent.app import create_agent_bridge_appfrom switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistryfrom switch_core.bridges.agent.protocol.agent_core import AgentCorefrom switch_core.bridges.agent.protocol.agent_event_listener import AgentEventListener-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer+from switch_core.bridges.agent.protocol.event_buffer import (+ EventBuffer,+ session_slots_for,+)from switch_core.bridges.agent.protocol.types import (IntegrationProfile,)@@ -472,6 +475,7 @@ async def _serve(sequence_base=event_boot << 32,rooms=message_listener,retention_seconds=config.agent_event_retention_seconds,+ session_slots=session_slots_for(config.db_pool_size, config.db_max_overflow),)agent_event_listener = AgentEventListener(lambda: create_unpooled_engine(config), event_bufferdiff --git a/core/tests/db_buffer.py b/core/tests/db_buffer.pyindex 4cfc7e25e..2b9899484 100644--- a/core/tests/db_buffer.py+++ b/core/tests/db_buffer.py@@ -61,6 +61,7 @@ def db_buffer_on(sequence_base: int = 0,retention_seconds: float = 3600,rooms: RoomAnnouncements | None = None,+ session_slots: int | None = None,) -> EventBuffer:"""The db-buffer on a test's database."""return EventBuffer(@@ -68,6 +69,7 @@ def db_buffer_on(sequence_base=sequence_base,rooms=rooms if rooms is not None else QuietRooms(),retention_seconds=retention_seconds,+ session_slots=session_slots,)diff --git a/core/tests/integration/conftest.py b/core/tests/integration/conftest.pyindex 36b55c0e0..04756fab7 100644--- a/core/tests/integration/conftest.py+++ b/core/tests/integration/conftest.py@@ -44,7 +44,10 @@ from switch_core.bridges.agent.api.handlers import (from switch_core.bridges.agent.api_key_cache import ApiKeyCachefrom switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistryfrom switch_core.bridges.agent.protocol.agent_core import AgentCore-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer+from switch_core.bridges.agent.protocol.event_buffer import (+ EventBuffer,+ session_slots_for,+)from switch_core.bridges.agent.protocol.types import (IntegrationProfile,RegistrationResult,@@ -585,6 +588,9 @@ async def harness(sequence_base=0,rooms=message_listener,retention_seconds=config.agent_event_retention_seconds,+ session_slots=session_slots_for(+ config.db_pool_size, config.db_max_overflow+ ),)connections = AgentConnectionRegistry()collab_lifecycle: object = _NoBridges()diff --git a/core/tests/session_counting.py b/core/tests/session_counting.pynew file mode 100644index 000000000..2a812ff17--- /dev/null+++ b/core/tests/session_counting.py@@ -0,0 +1,20 @@+"""A session factory that counts the sessions opened through it, for tests+that pin how many pool checkouts a path costs. One session is one checkout."""++from __future__ import annotations++from typing import Any++from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker+++class CountingSessionFactory:+ """The real factory, counting the sessions opened through it."""++ def __init__(self, inner: async_sessionmaker[AsyncSession]) -> None:+ self._inner = inner+ self.opened = 0++ def __call__(self) -> Any:+ self.opened += 1+ return self._inner()diff --git a/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_session_slots.py b/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_session_slots.pynew file mode 100644index 000000000..24e2b82ad--- /dev/null+++ b/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_session_slots.py@@ -0,0 +1,109 @@+"""`EventBuffer.one_session` lets at most `session_slots` blocks hold a+connection at once, so the streams' passes cannot take the whole pool from+requests when a pod is saturated."""++from __future__ import annotations++import asyncio++import pytest+from sqlalchemy import text+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker++from switch_core.db.models import TENANT_ZERO_ID+from tests.db_buffer import db_buffer_on+++async def _hold(+ buffer, inside: list[int], most: list[int], release: asyncio.Event+) -> None:+ async with buffer.one_session(TENANT_ZERO_ID) as session:+ await session.execute(text("select 1"))+ inside[0] += 1+ most[0] = max(most[0], inside[0])+ await release.wait()+ inside[0] -= 1+++async def test_blocks_over_the_limit_wait_their_turn(+ session_factory: async_sessionmaker[AsyncSession],+) -> None:+ buffer = db_buffer_on(session_factory, session_slots=2)+ inside, most, release = [0], [0], asyncio.Event()+ tasks = [+ asyncio.create_task(_hold(buffer, inside, most, release)) for _ in range(5)+ ]+ for _ in range(50):+ if inside[0] == 2:+ break+ await asyncio.sleep(0.01)+ await asyncio.sleep(0.05)++ assert inside[0] == 2++ release.set()+ await asyncio.wait_for(asyncio.gather(*tasks), timeout=5)+ assert most[0] == 2+++async def test_a_block_that_fails_gives_its_slot_back(+ session_factory: async_sessionmaker[AsyncSession],+) -> None:+ buffer = db_buffer_on(session_factory, session_slots=1)+ with pytest.raises(RuntimeError, match="boom"):+ async with buffer.one_session(TENANT_ZERO_ID):+ raise RuntimeError("boom")++ async with asyncio.timeout(2):+ async with buffer.one_session(TENANT_ZERO_ID) as session:+ assert (await session.execute(text("select 1"))).scalar_one() == 1+++async def test_a_cancelled_waiter_takes_no_slot(+ session_factory: async_sessionmaker[AsyncSession],+) -> None:+ buffer = db_buffer_on(session_factory, session_slots=1)+ inside, most, release = [0], [0], asyncio.Event()+ holder = asyncio.create_task(_hold(buffer, inside, most, release))+ await asyncio.sleep(0.05)+ waiter = asyncio.create_task(_hold(buffer, inside, most, release))+ await asyncio.sleep(0.05)+ waiter.cancel()+ with pytest.raises(asyncio.CancelledError):+ await waiter++ release.set()+ await asyncio.wait_for(holder, timeout=5)+ async with asyncio.timeout(2):+ async with buffer.one_session(TENANT_ZERO_ID):+ pass+++async def test_without_a_limit_every_block_goes_in(+ session_factory: async_sessionmaker[AsyncSession],+) -> None:+ buffer = db_buffer_on(session_factory)+ inside, most, release = [0], [0], asyncio.Event()+ tasks = [+ asyncio.create_task(_hold(buffer, inside, most, release)) for _ in range(5)+ ]+ for _ in range(100):+ if inside[0] == 5:+ break+ await asyncio.sleep(0.01)++ assert inside[0] == 5++ release.set()+ await asyncio.wait_for(asyncio.gather(*tasks), timeout=5)+++async def test_a_block_inside_another_raises_rather_than_waiting_on_its_own_slot(+ session_factory: async_sessionmaker[AsyncSession],+) -> None:+ buffer = db_buffer_on(session_factory, session_slots=1)+ async with buffer.one_session(TENANT_ZERO_ID):+ with pytest.raises(RuntimeError, match="inside another"):+ async with asyncio.timeout(2):+ async with buffer.one_session(TENANT_ZERO_ID):+ passdiff --git a/core/tests/switch_core/bridges/agent/protocol/test_pool_checkouts.py b/core/tests/switch_core/bridges/agent/protocol/test_pool_checkouts.pyindex 2e2468a29..ea69d001f 100644--- a/core/tests/switch_core/bridges/agent/protocol/test_pool_checkouts.py+++ b/core/tests/switch_core/bridges/agent/protocol/test_pool_checkouts.py@@ -34,22 +34,7 @@ from switch_core.db.stores.agent_store import AgentStorefrom switch_core.db.stores.budget_store import BudgetStorefrom switch_core.db.stores.room_store import RoomStorefrom switch_core.db.stores.usage_store import UsageStore---class _CountingSessionFactory:- """The real factory, counting the sessions opened through it.-- One session is one pool checkout, which is the quantity this whole change- is about.- """-- def __init__(self, inner: async_sessionmaker[AsyncSession]) -> None:- self._inner = inner- self.opened = 0-- def __call__(self) -> Any:- self.opened += 1- return self._inner()+from tests.session_counting import CountingSessionFactoryclass _NoBridges:@@ -68,7 +53,7 @@ class _OneBridge:self.typing.append((room_id, agent_name, is_typing))-def _service(counting: _CountingSessionFactory) -> AgentCore:+def _service(counting: CountingSessionFactory) -> AgentCore:svc = object.__new__(AgentCore)svc.session_factory = counting # type: ignore[attr-defined]svc.agent_store = AgentStore() # type: ignore[attr-defined]@@ -136,7 +121,7 @@ class TestRequireRoomMember:self, session_factory: async_sessionmaker[AsyncSession]) -> None:room_id, member_id, _ = await _seed(session_factory)- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)svc = _service(counting)room = await svc.require_room_member(member_id, room_id)@@ -148,7 +133,7 @@ class TestRequireRoomMember:self, session_factory: async_sessionmaker[AsyncSession]) -> None:room_id, _, outsider_id = await _seed(session_factory)- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)svc = _service(counting)with pytest.raises(PermissionError):@@ -162,7 +147,7 @@ class TestRequireRoomMember:# "No such room" and "not your room" must stay distinguishable now that# the two reads have been folded into one statement._, member_id, _ = await _seed(session_factory)- svc = _service(_CountingSessionFactory(session_factory))+ svc = _service(CountingSessionFactory(session_factory))with pytest.raises(ValueError, match="Room not found"):await svc.require_room_member(member_id, "no-such-room")@@ -171,7 +156,7 @@ class TestRequireRoomMember:self, session_factory: async_sessionmaker[AsyncSession]) -> None:room_id, member_id, _ = await _seed(session_factory)- svc = _service(_CountingSessionFactory(session_factory))+ svc = _service(CountingSessionFactory(session_factory))room = await svc.require_room_member(member_id, room_id)@@ -193,7 +178,7 @@ class TestCanAddress:)await session.commit()- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)svc = _service(counting)async with session_factory() as session:target = await svc.agent_store.get(session, member_id)@@ -223,7 +208,7 @@ class TestCanAddress:)await session.commit()- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)svc = _service(counting)async with session_factory() as session:target = await svc.agent_store.get(session, member_id)@@ -245,7 +230,7 @@ class TestCanAddress:self, session_factory: async_sessionmaker[AsyncSession]) -> None:_, member_id, outsider_id = await _seed(session_factory)- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)svc = _service(counting)async with session_factory() as session:@@ -272,7 +257,7 @@ class TestTyping:# Was three checkouts: membership took two of its own, then a third# re-read the agent and the room the caller had already resolved.room_id, member_id, _ = await _seed(session_factory)- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)svc = _service(counting)await svc.set_typing(member_id, room_id, True)@@ -283,7 +268,7 @@ class TestTyping:self, session_factory: async_sessionmaker[AsyncSession]) -> None:_, member_id, _ = await _seed(session_factory)- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)svc = _service(counting)bridge = _OneBridge()svc.collab_lifecycle = bridge # type: ignore[assignment]@@ -310,7 +295,7 @@ class TestSendMessage:# agent for the typing indicator — and no more than that, because the# room the caller already resolved is passed along instead of re-read.room_id, member_id, _ = await _seed(session_factory)- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)svc = _service(counting)sent: list[tuple[str, str]] = []@@ -337,7 +322,7 @@ class TestSendMessage:self, session_factory: async_sessionmaker[AsyncSession]) -> None:room_id, _, outsider_id = await _seed(session_factory)- svc = _service(_CountingSessionFactory(session_factory))+ svc = _service(CountingSessionFactory(session_factory))with pytest.raises(PermissionError):await svc.send_message(outsider_id, room_id, "hello")@@ -367,7 +352,7 @@ class TestSendMessage:amount=2,)await session.commit()- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)svc = _service(counting)with pytest.raises(BudgetExceeded, match="member has reached its budget"):@@ -383,7 +368,7 @@ class TestAgentStatuses:# The gateway builds a room's detail inside its request transaction.# Resolving statuses used to open a second one from underneath it.room_id, member_id, _ = await _seed(session_factory)- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)svc = _service(counting)svc.agent_session_store = AgentSessionStore() # type: ignore[attr-defined]@@ -400,7 +385,7 @@ class TestAgentStatuses:self, session_factory: async_sessionmaker[AsyncSession]) -> None:room_id, member_id, _ = await _seed(session_factory)- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)svc = _service(counting)svc.agent_session_store = AgentSessionStore() # type: ignore[attr-defined]diff --git a/core/tests/switch_core/bridges/agent/protocol/test_stream.py b/core/tests/switch_core/bridges/agent/protocol/test_stream.pyindex 8f68dfbf9..4d001b881 100644--- a/core/tests/switch_core/bridges/agent/protocol/test_stream.py+++ b/core/tests/switch_core/bridges/agent/protocol/test_stream.py@@ -11,7 +11,7 @@ from __future__ import annotationsimport asyncioimport inspectfrom collections.abc import Callable-from typing import Any+from typing import Any, castimport pytestfrom sqlalchemy import select@@ -61,6 +61,7 @@ from tests.db_buffer import (seed_member,sender_of,)+from tests.session_counting import CountingSessionFactoryAGENT = "agent-1"ROOM_A = "room-a"@@ -1638,3 +1639,30 @@ class TestBothTransportsEncodeTheSameFrames:assert first.startswith(b"event: connection_state\n")assert not conn.stream_attachedassert registry.get(conn.id) is conn+++async def test_a_pass_reads_in_one_session(+ session_factory: async_sessionmaker[AsyncSession], room_bells: MessageListener+) -> None:+ """The reads of a pass that delivers (the agent's rooms, its head, its own+ events, what was dropped after its position, its rooms' conversation and+ the unread counts) cost one pool checkout and one transaction, not one of+ each per read. A contract like those in `test_pool_checkouts.py`: every+ message wakes a pass in each stream it is for."""+ counting = CountingSessionFactory(session_factory)+ buffer = db_buffer_on(cast(Any, counting), rooms=room_bells)+ registry = AgentConnectionRegistry()+ conn = _open(registry, scope="all", delivery_filter="all")+ stream = event_frames(conn=conn, registry=registry, buffer=buffer, approvals=None)+ await _addressed(session_factory, buffer, "first")+ opening = await _take(stream, 2)+ assert [name for name, _ in opening] == ["connection_state", "message"]++ await _addressed(session_factory, buffer, "second")+ counting.opened = 0+ ((name, data),) = await _take(stream, 1)+ await stream.aclose()++ assert name == "message"+ assert data["payload"]["body"] == "second"+ assert counting.opened == 1diff --git a/core/tests/switch_core/bridges/agent/test_api_key_cache.py b/core/tests/switch_core/bridges/agent/test_api_key_cache.pyindex cb42e560d..302abb14e 100644--- a/core/tests/switch_core/bridges/agent/test_api_key_cache.py+++ b/core/tests/switch_core/bridges/agent/test_api_key_cache.py@@ -30,6 +30,7 @@ from switch_core.db.models import Agent, ApiKey, Client, Userfrom switch_core.db.stores.agent_store import AgentStorefrom switch_core.db.stores.api_key_store import ApiKeyStorefrom switch_core.keys import Keyring+from tests.session_counting import CountingSessionFactory_PROFILE = IntegrationProfile(connection_model="session_passive",@@ -40,21 +41,6 @@ _PROFILE = IntegrationProfile()-class _CountingSessionFactory:- """Wraps the real factory and counts how many sessions are opened.-- A session is a pool checkout, so this is the number the incident is about.- """-- def __init__(self, inner: async_sessionmaker[AsyncSession]) -> None:- self._inner = inner- self.opened = 0-- def __call__(self) -> Any:- self.opened += 1- return self._inner()--class _FakeClientLifecycle:def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:self._session_factory = session_factory@@ -242,7 +228,7 @@ class TestResolutionHitsAndMisses:self, session_factory: async_sessionmaker[AsyncSession]) -> None:await _seed_agent(session_factory, "tok")- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)counting.opened = 0mw = _middleware(counting, ApiKeyCache(ttl_seconds=5, max_entries=8))@@ -270,7 +256,7 @@ class TestResolutionHitsAndMisses:# on a tenant_session to read the row itself. What the cache above# saves is these two round trips, not one.await _seed_agent(session_factory, "tok")- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)counting.opened = 0mw = _middleware(counting, ApiKeyCache(ttl_seconds=0, max_entries=8))@@ -285,7 +271,7 @@ class TestResolutionHitsAndMisses:# tenant to open the second, row-reading session under. This is the# load-shape that matters — an unauthenticated flood never reaches# the second query.- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)cache = ApiKeyCache(ttl_seconds=5, max_entries=8)mw = _middleware(counting, cache)@@ -298,7 +284,7 @@ class TestResolutionHitsAndMisses:# A token that does not resolve must reach Postgres every time, or a# key that starts existing would keep failing — and, far worse, the# memo would become a place to park an attacker-chosen string.- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)cache = ApiKeyCache(ttl_seconds=5, max_entries=8)mw = _middleware(counting, cache)@@ -318,7 +304,7 @@ class TestResolutionHitsAndMisses:key_type="registration",with_agent=False,)- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)counting.opened = 0cache = ApiKeyCache(ttl_seconds=5, max_entries=8)mw = _middleware(counting, cache)@@ -337,7 +323,7 @@ class TestResolutionHitsAndMisses:self, session_factory: async_sessionmaker[AsyncSession]) -> None:await _seed_agent(session_factory, "orphan", name="orphan", with_agent=False)- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)counting.opened = 0cache = ApiKeyCache(ttl_seconds=5, max_entries=8)mw = _middleware(counting, cache)@@ -354,7 +340,7 @@ class TestResolutionHitsAndMisses:self, session_factory: async_sessionmaker[AsyncSession]) -> None:await _seed_agent(session_factory, "tok")- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)counting.opened = 0mw = _middleware(counting, ApiKeyCache(ttl_seconds=0, max_entries=8))@@ -373,7 +359,7 @@ class TestResolutionHitsAndMisses:monkeypatch.setattr("switch_core.bridges.agent.api_key_cache.time.monotonic", lambda: now[0])- counting = _CountingSessionFactory(session_factory)+ counting = CountingSessionFactory(session_factory)counting.opened = 0mw = _middleware(counting, ApiKeyCache(ttl_seconds=5, max_entries=8))diff --git a/core/tests/switch_core/db/test_pool_pre_ping.py b/core/tests/switch_core/db/test_pool_pre_ping.pynew file mode 100644index 000000000..a7b695248--- /dev/null+++ b/core/tests/switch_core/db/test_pool_pre_ping.py@@ -0,0 +1,105 @@+"""What turning the pool's pre-ping off costs (`db_pool_pre_ping`).++A pooled connection the server closed while it sat in the pool is the case+the pre-ping exists for: a database restart closes every connection the pool+holds. With the pre-ping off, the first query on one of them fails, and+SQLAlchemy then treats every connection the pool held at that moment as+stale and replaces it on its next checkout. So a restart costs one failed+query, not one per pooled connection, which is the trade the default makes.+"""++from __future__ import annotations++import asyncio++import pytest+from sqlalchemy import text+from sqlalchemy.exc import DBAPIError+from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine++from switch_core.config import SwitchConfig+from switch_core.db.engine import create_engine_from_config, create_unpooled_engine+from tests.switch_core.db.test_dead_peer_detection import _config_for++POOLED = 3+++async def _pid(conn: AsyncConnection) -> int:+ return int((await conn.execute(text("select pg_backend_pid()"))).scalar_one())+++async def _fill_and_close_pool(+ postgres_url: str, *, pre_ping: bool+) -> tuple[AsyncEngine, set[int]]:+ """An engine whose pool holds `POOLED` connections, every one of which the+ server has since closed, as a restart does. Returns the engine and the+ closed connections' backend pids."""+ engine = create_engine_from_config(+ _config_for(+ postgres_url,+ db_pool_size=POOLED,+ db_max_overflow=0,+ db_pool_pre_ping=pre_ping,+ )+ )+ held = [await engine.connect() for _ in range(POOLED)]+ pids = {await _pid(conn) for conn in held}+ for conn in held:+ await conn.close()+ closer = create_unpooled_engine(_config_for(postgres_url))+ try:+ async with closer.connect() as conn:+ for pid in pids:+ await conn.execute(+ text("select pg_terminate_backend(:pid)"), {"pid": pid}+ )+ # Termination is a signal: wait until the backends have exited, or+ # one could still answer a ping and then die under the next query.+ for _ in range(100):+ left = await conn.scalar(+ text(+ "select count(*) from pg_stat_activity where pid = any(:pids)"+ ),+ {"pids": list(pids)},+ )+ if not left:+ break+ await asyncio.sleep(0.05)+ else:+ raise AssertionError("the closed backends did not exit")+ finally:+ await closer.dispose()+ return engine, pids+++def test_the_pre_ping_is_off_by_default() -> None:+ assert SwitchConfig.model_fields["db_pool_pre_ping"].default is False+++async def test_without_the_pre_ping_a_restart_costs_one_failed_query(+ postgres_url: str,+) -> None:+ engine, closed = await _fill_and_close_pool(postgres_url, pre_ping=False)+ try:+ with pytest.raises(DBAPIError) as failed:+ async with engine.connect() as conn:+ await _pid(conn)+ assert failed.value.connection_invalidated++ # Every connection after it is a new one, though two more of the+ # closed ones were still in the pool when the first query failed.+ for _ in range(POOLED):+ async with engine.connect() as conn:+ assert await _pid(conn) not in closed+ finally:+ await engine.dispose()+++async def test_with_the_pre_ping_no_query_fails(postgres_url: str) -> None:+ engine, closed = await _fill_and_close_pool(postgres_url, pre_ping=True)+ try:+ for _ in range(POOLED):+ async with engine.connect() as conn:+ assert await _pid(conn) not in closed+ finally:+ await engine.dispose()diff --git a/core/tests/switch_core/db/test_session_scope.py b/core/tests/switch_core/db/test_session_scope.pyindex 9b172a19e..3f8d7160e 100644--- a/core/tests/switch_core/db/test_session_scope.py+++ b/core/tests/switch_core/db/test_session_scope.py@@ -18,12 +18,16 @@ is pinned in `test_tenant_exemption_allowlist.py`.from __future__ import annotations+import asyncio+from typing import Any+import pytestfrom sqlalchemy import textfrom sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker-from switch_core.db.session_scope import tenant_session+from switch_core.db.session_scope import shared_tenant_session, tenant_sessionfrom switch_core.tenant_context import current_tenant_id, tenant_scope+from tests.session_counting import CountingSessionFactoryTENANT_A = "tenant-session-scope-a"TENANT_B = "tenant-session-scope-b"@@ -86,3 +90,108 @@ class TestTenantSession:async with tenant_session(session_factory, TENANT_B) as session:assert await _current_setting(session) == TENANT_Bawait session.commit()+++class TestSharedTenantSession:+ async def test_a_session_opened_inside_the_block_is_the_shared_session(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ counting: Any = CountingSessionFactory(session_factory)+ async with shared_tenant_session(counting, TENANT_A) as shared:+ async with tenant_session(counting, TENANT_A) as first:+ assert await _current_setting(first) == TENANT_A+ async with tenant_session(counting, TENANT_A) as second:+ assert await _current_setting(second) == TENANT_A+ assert first is shared+ assert second is shared+ assert counting.opened == 1++ async def test_another_tenant_opens_its_own(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ async with shared_tenant_session(session_factory, TENANT_A) as shared:+ assert await _current_setting(shared) == TENANT_A+ async with tenant_session(session_factory, TENANT_B) as other:+ assert other is not shared+ assert await _current_setting(other) == TENANT_B+ assert await _current_setting(shared) == TENANT_A++ async def test_another_task_opens_its_own(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """A task started inside the block inherits its context, but two tasks+ must never use one `AsyncSession` at once."""++ async def opened() -> AsyncSession:+ async with tenant_session(session_factory, TENANT_A) as session:+ await _current_setting(session)+ return session++ async with shared_tenant_session(session_factory, TENANT_A) as shared:+ assert await asyncio.create_task(opened()) is not shared++ async def test_nothing_is_lent_once_the_block_exits(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ async with shared_tenant_session(session_factory, TENANT_A) as shared:+ pass+ async with tenant_session(session_factory, TENANT_A) as after:+ assert after is not shared++ async def test_an_inner_block_leaves_the_session_open(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ async with shared_tenant_session(session_factory, TENANT_A) as shared:+ async with tenant_session(session_factory, TENANT_A) as inner:+ await _current_setting(inner)+ assert await _current_setting(shared) == TENANT_A++ async def test_a_commit_inside_the_block_keeps_the_tenant(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """A commit ends the transaction, and `set_config` is local to it; the+ next transaction is stamped again by `after_begin`."""+ async with shared_tenant_session(session_factory, TENANT_A):+ async with tenant_session(session_factory, TENANT_A) as session:+ assert await _current_setting(session) == TENANT_A+ await session.commit()+ async with tenant_session(session_factory, TENANT_A) as session:+ assert await _current_setting(session) == TENANT_A++ @pytest.mark.no_ambient_tenant+ async def test_the_tenant_is_unbound_once_the_block_exits(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ with pytest.raises(RuntimeError, match="boom"):+ async with shared_tenant_session(session_factory, TENANT_A):+ async with tenant_session(session_factory, TENANT_A):+ raise RuntimeError("boom")+ assert current_tenant_id() is None+ async with tenant_session(session_factory, TENANT_A) as session:+ assert await _current_setting(session) == TENANT_A++ async def test_a_block_inside_another_for_the_same_tenant_raises(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ async with shared_tenant_session(session_factory, TENANT_A):+ with pytest.raises(RuntimeError, match="already open"):+ async with shared_tenant_session(session_factory, TENANT_A):+ pass++ async def test_a_block_left_by_closing_its_generator_lends_nothing_after(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """Yielding inside a block is forbidden, but if a generator does and is+ then closed, later sessions in the same task are not handed the closed+ one."""++ async def holds() -> Any:+ async with shared_tenant_session(session_factory, TENANT_A) as session:+ yield session++ frames = holds()+ shared = await anext(frames)+ await frames.aclose()++ async with tenant_session(session_factory, TENANT_A) as after:+ assert after is not shareddiff --git a/core/tests/switch_core/gateway/test_gateway_ui.py b/core/tests/switch_core/gateway/test_gateway_ui.pyindex c069434c0..fc5c9c4c3 100644--- a/core/tests/switch_core/gateway/test_gateway_ui.py+++ b/core/tests/switch_core/gateway/test_gateway_ui.py@@ -168,7 +168,9 @@ def _agent_bridge_app(gateway_ui: GatewayUi | None) -> FastAPI:room_service=object(), # type: ignore[arg-type]client_lifecycle=object(), # type: ignore[arg-type]collab_lifecycle=object(), # type: ignore[arg-type]- event_buffer=EventBuffer(object(), sequence_base=0, rooms=QuietRooms()), # type: ignore[arg-type]+ event_buffer=EventBuffer(+ object(), sequence_base=0, rooms=QuietRooms(), session_slots=None+ ), # type: ignore[arg-type]resource_service=object(), # type: ignore[arg-type]api_key_store=object(), # type: ignore[arg-type]external_user_store=object(), # type: ignore[arg-type]diff --git a/core/tests/switch_core/management/test_middleware.py b/core/tests/switch_core/management/test_middleware.pyindex 14b5951bb..06f9626a9 100644--- a/core/tests/switch_core/management/test_middleware.py+++ b/core/tests/switch_core/management/test_middleware.py@@ -310,7 +310,9 @@ class TestTheFlagOff:room_service=object(), # type: ignore[arg-type]client_lifecycle=object(), # type: ignore[arg-type]collab_lifecycle=object(), # type: ignore[arg-type]- event_buffer=EventBuffer(object(), sequence_base=0, rooms=QuietRooms()), # type: ignore[arg-type]+ event_buffer=EventBuffer(+ object(), sequence_base=0, rooms=QuietRooms(), session_slots=None+ ), # type: ignore[arg-type]resource_service=object(), # type: ignore[arg-type]api_key_store=object(), # type: ignore[arg-type]external_user_store=object(), # type: ignore[arg-type]diff --git a/core/tests/switch_core/management/test_stream.py b/core/tests/switch_core/management/test_stream.pyindex 7c3471caa..39b3b3288 100644--- a/core/tests/switch_core/management/test_stream.py+++ b/core/tests/switch_core/management/test_stream.py@@ -38,6 +38,7 @@ from switch_core.management.notifier import (ControllerNotifier,)from tests.db_buffer import age_events, say+from tests.session_counting import CountingSessionFactoryfrom tests.switch_core.management.harness import (Harness,add_member,@@ -224,6 +225,32 @@ class TestOpening:assert data["event"]["payload"]["body"] == "@you"assert data["event"]["missed"] == {"count": 2, "reason": None}+ async def test_an_agents_pass_reads_in_one_session(+ self, harness: Harness, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ """The agent's head, its own events, what was dropped after its cursor+ and the unread counts cost one pool checkout and one transaction, not+ one of each per read."""+ owner = await add_member(harness.session_factory, "ada")+ async with harness.client() as client:+ controller = await enroll_console(harness, client, owner)+ agent_id = await place_agent(client, controller, name="reviewer")+ room_id = await add_room(harness.session_factory, agent_id)+ opened = await open_connection(client, controller)++ stream = await open_stream(harness, controller, opened)+ await take(stream, 2)+ await _ask(harness, agent_id, room_id, "@you")+ buffer = harness.protocol.event_buffer+ counting = CountingSessionFactory(buffer._session_factory)+ monkeypatch.setattr(buffer, "_session_factory", counting)+ [(name, data)] = await take(stream, 1)+ await stream.aclose()++ assert name == "agent.event"+ assert data["event"]["payload"]["body"] == "@you"+ assert counting.opened == 1+async def test_a_cursor_from_before_a_restart_is_a_gap(self, harness: Harness) -> None:diff --git a/core/tests/switch_core/observability/test_connection_metrics.py b/core/tests/switch_core/observability/test_connection_metrics.pyindex 5f11b48bb..094caaa21 100644--- a/core/tests/switch_core/observability/test_connection_metrics.py+++ b/core/tests/switch_core/observability/test_connection_metrics.py@@ -251,7 +251,12 @@ class _EmptyBuffer(EventBuffer):"""def __init__(self) -> None:- super().__init__(object(), sequence_base=0, rooms=QuietRooms()) # type: ignore[arg-type]+ super().__init__(+ object(), # type: ignore[arg-type]+ sequence_base=0,+ rooms=QuietRooms(),+ session_slots=None,+ )async def head(self, tenant_id: str, agent_id: str) -> int:return self.sequence_floor - 1diff --git a/core/tests/switch_core/telemetry/test_emitters_are_live.py b/core/tests/switch_core/telemetry/test_emitters_are_live.pyindex 59bb2afdf..f763415c6 100644--- a/core/tests/switch_core/telemetry/test_emitters_are_live.py+++ b/core/tests/switch_core/telemetry/test_emitters_are_live.py@@ -53,7 +53,9 @@ def _build(telemetry: TelemetryService) -> Any:room_service=object(), # type: ignore[arg-type]client_lifecycle=object(), # type: ignore[arg-type]collab_lifecycle=object(), # type: ignore[arg-type]- event_buffer=EventBuffer(object(), sequence_base=0, rooms=QuietRooms()), # type: ignore[arg-type]+ event_buffer=EventBuffer(+ object(), sequence_base=0, rooms=QuietRooms(), session_slots=None+ ), # type: ignore[arg-type]resource_service=object(), # type: ignore[arg-type]api_key_store=object(), # type: ignore[arg-type]external_user_store=object(), # type: ignore[arg-type]diff --git a/core/tests/switch_core/test_ingress_agent_api_paths.py b/core/tests/switch_core/test_ingress_agent_api_paths.pyindex c9a88450d..9c525acdb 100644--- a/core/tests/switch_core/test_ingress_agent_api_paths.py+++ b/core/tests/switch_core/test_ingress_agent_api_paths.py@@ -54,7 +54,9 @@ def _served_prefixes() -> set[str]:room_service=object(), # type: ignore[arg-type]client_lifecycle=object(), # type: ignore[arg-type]collab_lifecycle=object(), # type: ignore[arg-type]- event_buffer=EventBuffer(object(), sequence_base=0, rooms=QuietRooms()), # type: ignore[arg-type]+ event_buffer=EventBuffer(+ object(), sequence_base=0, rooms=QuietRooms(), session_slots=None+ ), # type: ignore[arg-type]resource_service=object(), # type: ignore[arg-type]api_key_store=object(), # type: ignore[arg-type]external_user_store=object(), # type: ignore[arg-type]diff --git a/docs/old/rds-migration.md b/docs/old/rds-migration.mdindex c07977655..632c40fd0 100644--- a/docs/old/rds-migration.md+++ b/docs/old/rds-migration.md@@ -157,8 +157,9 @@ investigated.Two related points for whoever watches this:-- `pool_pre_ping` is on, so pooled connections recover on their own after a- failover rather than serving one error each.+- `pool_pre_ping` is off by default since switch#1111. After a failover the+ first query on each pod fails, along with any query that was running, and+ SQLAlchemy then replaces every connection the pool held.- The notification queue is bounded (8 GB) and fills only if a listener sits ina long transaction. Ours does not, but a queue-full alarm is cheap.