Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Read each stream pass in one database session and turn off the pool pre-ping (CHOO-3667) (#1111)

Abel Dantas0ac8ec485d1e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4155b0a3e..bd1343ab9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -142,6 +142,20 @@ version of their own to them without also giving them a release of their own.
compares it only when someone signs in.
#### Changed
+- **An agent's stream reads in one database session per pass, and pooled
+ connections are no longer pinged before use.** A pass that delivered a
+ message opened a session for each of its reads, six pool checkouts and six
+ transactions, and the pre-ping added a transaction to every checkout. The
+ reads of a pass now share one session, for an agent on its own socket and
+ for each agent an agent controller serves. `DB_POOL_PRE_PING` now defaults
+ to `false`: after a database restart the first query on each pod fails, along
+ with any query running at the time, and the pool then replaces every
+ connection it held. Set it to `true` behind
+ anything that drops idle connections. In a load test of 300 agents at 10
+ messages a second on 2 pods, CPU per message fell by a quarter and database
+ transactions per message by more than two thirds. At most half of a pod's
+ pool goes to streams' passes at once, so requests still get a connection
+ when the pod's CPU is saturated.
- **A cloud machine can move to another instance type and keep its disk.**
The hosted controller now launches every new instance of a machine as the
instance type in its `gateway.json`, at the same moment it already moves the
diff --git a/core/switch_core/bridges/agent/protocol/controller_stream.py b/core/switch_core/bridges/agent/protocol/controller_stream.py
index a7dfee7b5..4562bc18c 100644
--- a/core/switch_core/bridges/agent/protocol/controller_stream.py
+++ b/core/switch_core/bridges/agent/protocol/controller_stream.py
@@ -400,52 +400,57 @@ class _ControllerStream:
self._tenant, agent_id, attached.reader_id, room_id
)
attached.counted.add(room_id)
- # Take the number of the newest of the agent's own events before
- # reading, and read no further than it. A read that finds nothing moves
- # the cursor up to this number. An event added to `agent_events` while
- # the read runs is numbered above it, so the cursor does not skip that
- # event.
- bound = await self._buffer.head(self._tenant, agent_id)
- try:
- pending = await self._buffer.read_from(
+ # The reads below share one session (`EventBuffer.one_session`): one
+ # pool checkout and one transaction for the agent's pass, not one per
+ # read. The frames are built after it, and returned rather than
+ # yielded, so the connection is back in the pool before any is sent.
+ async with self._buffer.one_session(self._tenant):
+ # Take the number of the newest of the agent's own events before
+ # reading, and read no further than it. A read that finds nothing moves
+ # the cursor up to this number. An event added to `agent_events` while
+ # the read runs is numbered above it, so the cursor does not skip that
+ # event.
+ bound = await self._buffer.head(self._tenant, agent_id)
+ try:
+ pending = await self._buffer.read_from(
+ self._tenant,
+ agent_id,
+ attached.cursor,
+ limit=CATCH_UP_BATCH,
+ through_seq=bound,
+ )
+ except CursorExpiredError as exc:
+ resumed_at = exc.resume_after
+ attached.cursor = resumed_at
+ self._conn.sent_cursors[agent_id] = resumed_at
+ return [
+ frame(
+ "agent.gap",
+ {
+ "agent_id": agent_id,
+ "from_sequence": exc.requested,
+ "resumed_at": resumed_at,
+ "rooms": list(exc.rooms),
+ "all_rooms": False,
+ "reason": str(exc),
+ },
+ )
+ ]
+ if not pending:
+ # Events above the cursor that are no longer held — a room the
+ # agent left takes its events with it — are passed, or every pass
+ # would find work it can never read and spin.
+ attached.cursor = max(attached.cursor, bound)
+ self._conn.sent_cursors[agent_id] = attached.cursor
+ return []
+ # The unread counts sent with the events below, counted for the whole
+ # batch at once, so that catching up on a few hundred events costs a
+ # few statements per room rather than several per event.
+ missed = await self._buffer.unread_batch(
self._tenant,
agent_id,
- attached.cursor,
- limit=CATCH_UP_BATCH,
- through_seq=bound,
+ [(item.room_id, item.room_seq) for item in pending if item.notifiable],
)
- except CursorExpiredError as exc:
- resumed_at = exc.resume_after
- attached.cursor = resumed_at
- self._conn.sent_cursors[agent_id] = resumed_at
- return [
- frame(
- "agent.gap",
- {
- "agent_id": agent_id,
- "from_sequence": exc.requested,
- "resumed_at": resumed_at,
- "rooms": list(exc.rooms),
- "all_rooms": False,
- "reason": str(exc),
- },
- )
- ]
- if not pending:
- # Events above the cursor that are no longer held — a room the
- # agent left takes its events with it — are passed, or every pass
- # would find work it can never read and spin.
- attached.cursor = max(attached.cursor, bound)
- self._conn.sent_cursors[agent_id] = attached.cursor
- return []
- # The unread counts sent with the events below, counted for the whole
- # batch at once, so that catching up on a few hundred events costs a
- # few statements per room rather than several per event.
- missed = await self._buffer.unread_batch(
- self._tenant,
- agent_id,
- [(item.room_id, item.room_seq) for item in pending if item.notifiable],
- )
frames: list[ControllerFrame] = []
for item in pending:
data = item.event.model_dump(mode="json")
diff --git a/core/switch_core/bridges/agent/protocol/event_buffer.py b/core/switch_core/bridges/agent/protocol/event_buffer.py
index d5e9fcd84..b3340d088 100644
--- a/core/switch_core/bridges/agent/protocol/event_buffer.py
+++ b/core/switch_core/bridges/agent/protocol/event_buffer.py
@@ -52,7 +52,8 @@ from __future__ import annotations
import asyncio
import logging
import time
-from collections.abc import Iterable
+from collections.abc import AsyncIterator, Iterable
+from contextlib import asynccontextmanager, nullcontext
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, Any, Protocol, cast
@@ -103,7 +104,11 @@ from switch_core.db.models import (
PendingAttachmentGroup,
Room,
)
-from switch_core.db.session_scope import tenant_session
+from switch_core.db.session_scope import (
+ holds_shared_session,
+ shared_tenant_session,
+ tenant_session,
+)
from switch_core.observability.catalogue import (
AGENT_BUFFER_SCAN_DURATION,
AGENT_BUFFER_SCANNED,
@@ -373,6 +378,12 @@ _UNREAD_SQL = text(
).bindparams(bindparam("throughs", type_=ARRAY(BigInteger)))
+def session_slots_for(pool_size: int, max_overflow: int) -> int:
+ """How many `EventBuffer.one_session` blocks may hold a connection at once:
+ half the pool, so requests still find one when the pod is saturated."""
+ return max(1, (pool_size + max_overflow) // 2)
+
+
class EventBuffer:
def __init__(
self,
@@ -381,8 +392,14 @@ class EventBuffer:
sequence_base: int,
rooms: RoomAnnouncements,
retention_seconds: float = DEFAULT_RETENTION_SECONDS,
+ session_slots: int | None,
) -> None:
self._session_factory = session_factory
+ # How many `one_session` blocks may hold a connection at once in this
+ # process; None for no limit. See `one_session`.
+ self._session_slots = (
+ asyncio.Semaphore(session_slots) if session_slots is not None else None
+ )
# `sequence_base` comes from a database sequence that advances on every
# boot (`main.py`), so it is above every number an earlier boot handed
# out. An agent's first own event is numbered from it. A client still holding
@@ -402,6 +419,35 @@ class EventBuffer:
# themselves are always read from `agent_events`.
self._notify: dict[str, asyncio.Event] = {}
+ @asynccontextmanager
+ async def one_session(self, tenant_id: str) -> AsyncIterator[AsyncSession]:
+ """Everything this buffer reads and writes for `tenant_id` inside the
+ block, from the task that opened it, in one session.
+
+ Each method here opens a session of its own, so a stream's pass that
+ delivers, which asks for the agent's rooms, its head, its own events,
+ what was dropped after its position, its rooms' conversation and its
+ unread counts, cost six pool checkouts and as many transactions. In one
+ session it costs one of each. `shared_tenant_session` says what the
+ block may contain; a block must not open another.
+
+ At most `session_slots` blocks hold a connection at once, and a block
+ over that waits here holding nothing. When a pod's CPU is saturated,
+ every await inside a block takes longer, so without the limit the
+ streams' passes come to hold most of the pool, and a request (a send, a
+ context read, authentication) waits out the pool timeout and fails.
+ """
+ # Checked before taking a slot: a nested block would otherwise wait on
+ # its own slot once every slot is taken, a deadlock seen only under load.
+ if holds_shared_session(self._session_factory, tenant_id):
+ raise RuntimeError("one_session opened inside another in the same task")
+ slots = self._session_slots or nullcontext()
+ async with (
+ slots,
+ shared_tenant_session(self._session_factory, tenant_id) as session,
+ ):
+ yield session
+
# ------------------------------------------------------------------
# Producing
# ------------------------------------------------------------------
diff --git a/core/switch_core/bridges/agent/protocol/stream.py b/core/switch_core/bridges/agent/protocol/stream.py
index aeacaf312..5d86251a4 100644
--- a/core/switch_core/bridges/agent/protocol/stream.py
+++ b/core/switch_core/bridges/agent/protocol/stream.py
@@ -348,24 +348,62 @@ async def _event_stream(
await buffer.ensure_counting(tenant, agent_id, conn.id, room_id)
counted.add(room_id)
- conversation_rooms: set[str] = set()
- if conn.reads_conversation:
- conversation_rooms = await _place_in_conversation(
- conn, registry, buffer, tenant
- )
- # Before the read, so a row written while it runs wakes the wait
- # below rather than being left for the keepalive.
- rooms.watch(conversation_rooms)
-
+ # The reads of a pass share one session, so they cost one pool
+ # checkout and one transaction rather than one of each per read
+ # (`EventBuffer.one_session`). Nothing is yielded inside it: a gap
+ # goes out after the block has given its connection back.
try:
- pending, looked = await buffer.read(
- tenant,
- agent_id,
- conn.position,
- conversation_rooms=conversation_rooms,
- notifiable_only=not conn.reads_conversation,
- limit=CATCH_UP_BATCH,
- )
+ async with buffer.one_session(tenant):
+ conversation_rooms: set[str] = set()
+ if conn.reads_conversation:
+ conversation_rooms = await _place_in_conversation(
+ conn, registry, buffer, tenant
+ )
+ # Before the read, so a row written while it runs wakes the
+ # wait below rather than being left for the keepalive.
+ rooms.watch(conversation_rooms)
+
+ pending, looked = await buffer.read(
+ tenant,
+ agent_id,
+ conn.position,
+ conversation_rooms=conversation_rooms,
+ notifiable_only=not conn.reads_conversation,
+ limit=CATCH_UP_BATCH,
+ )
+
+ # Room coverage is evaluated per event rather than up
+ # front: a sibling claiming a room must take effect
+ # immediately, without tearing the stream down. An event
+ # for a room this connection does not cover is not sent,
+ # but the position still moves past it, in order: only once
+ # every event before it has gone out, so the position never
+ # says the client was sent an event that is still to come.
+ # `covered` is the plan when the batch is read, which the
+ # tickets are minted for: room messages are in it because
+ # `_conversation_rooms` only returned rooms this connection
+ # covers. Each event is checked again just before it is
+ # sent, because the loop suspends at every yield and a
+ # sibling can claim the room while it does. The check reads
+ # the registry in memory, so it costs no query.
+ covered = [
+ item.source != OWN_EVENTS or registry.covers(conn, item.room_id)
+ for item in pending
+ ]
+ tickets = await _tickets_for(conn, buffer, tenant, pending, covered)
+ # The unread counts sent with the events below, counted for
+ # the whole batch at once, so that catching up on a few
+ # hundred events costs a few statements per room rather
+ # than several per event.
+ missed = await buffer.unread_batch(
+ tenant,
+ agent_id,
+ [
+ (item.room_id, item.room_seq)
+ for item, send in zip(pending, covered, strict=True)
+ if send and item.notifiable
+ ],
+ )
except CursorExpiredError as exc:
conn.position[OWN_EVENTS] = exc.resume_after
yield _frame(
@@ -382,36 +420,6 @@ async def _event_stream(
)
continue
- # Room coverage is evaluated per event rather than up front: a
- # sibling claiming a room must take effect immediately, without
- # tearing the stream down. An event for a room this connection does
- # not cover is not sent, but the position still moves past it, in
- # order: only once every event before it has gone out, so the
- # position never says the client was sent an event that is still
- # to come. `covered` is the plan when the batch is read, which the
- # tickets are minted for: room messages are in it because
- # `_conversation_rooms` only returned rooms this connection covers.
- # Each event is checked again just before it is sent, because the
- # loop suspends at every yield and a sibling can claim the room
- # while it does. The check reads the registry in memory, so it
- # costs no query.
- covered = [
- item.source != OWN_EVENTS or registry.covers(conn, item.room_id)
- for item in pending
- ]
- tickets = await _tickets_for(conn, buffer, tenant, pending, covered)
- # The unread counts sent with the events below, counted for the
- # whole batch at once, so that catching up on a few hundred events
- # costs a few statements per room rather than several per event.
- missed = await buffer.unread_batch(
- tenant,
- agent_id,
- [
- (item.room_id, item.room_seq)
- for item, send in zip(pending, covered, strict=True)
- if send and item.notifiable
- ],
- )
delivered = False
index = -1
for item, send in zip(pending, covered, strict=True):
diff --git a/core/switch_core/config.py b/core/switch_core/config.py
index 4799b02ef..f12a55c33 100644
--- a/core/switch_core/config.py
+++ b/core/switch_core/config.py
@@ -734,7 +734,19 @@ class SwitchConfig(BaseSettings):
db_pool_size: int = 30
db_max_overflow: int = 10
db_pool_recycle: int = 1800
- db_pool_pre_ping: bool = True
+ # Off by default. On, every checkout first pings its connection, and
+ # SQLAlchemy's asyncpg ping is a BEGIN, an empty statement and a ROLLBACK:
+ # three round trips and a transaction before any work, which under agent
+ # load was over a third of all the transactions core ran. What it catches
+ # is a connection that died while it sat in the pool, as every pooled
+ # connection does when the database restarts or fails over. Without it the
+ # first query on such a connection fails, and SQLAlchemy then replaces
+ # every connection the pool held. So on each pod the first query fails
+ # (`test_pool_pre_ping.py`), along with any query that was running when the
+ # database went, which fails with the ping too. How soon a connection left
+ # open by a failover is known to be dead is up to `db_tcp_keepalive_*` and
+ # `db_tcp_user_timeout` below, with the ping or without it.
+ db_pool_pre_ping: bool = False
# SQLAlchemy's default is 30s, long enough that pool starvation surfaces as
# unexplained latency rather than an error. Fail fast and loudly instead.
db_pool_timeout: float = 5.0
diff --git a/core/switch_core/db/session_scope.py b/core/switch_core/db/session_scope.py
index 56f4a1d49..f0ff7bca2 100644
--- a/core/switch_core/db/session_scope.py
+++ b/core/switch_core/db/session_scope.py
@@ -54,14 +54,41 @@ session's first query.
from __future__ import annotations
+import asyncio
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.tenant_context import tenant_scope
+@dataclass(frozen=True)
+class _Lent:
+ """The session a `shared_tenant_session` block lends, and who to."""
+
+ session_factory: async_sessionmaker[AsyncSession]
+ tenant_id: str
+ task: asyncio.Task[object] | None
+ session: AsyncSession
+
+ def serves(
+ self, session_factory: async_sessionmaker[AsyncSession], tenant_id: str
+ ) -> bool:
+ # Only the task that opened the block: a task started inside it copies
+ # this variable, and an `AsyncSession` must not be used by two tasks.
+ return (
+ self.session_factory is session_factory
+ and self.tenant_id == tenant_id
+ and self.task is asyncio.current_task()
+ )
+
+
+_lent: ContextVar[_Lent | None] = ContextVar("switch_lent_session", default=None)
+
+
@asynccontextmanager
async def tenant_session(
session_factory: async_sessionmaker[AsyncSession], tenant_id: str
@@ -77,7 +104,69 @@ async def tenant_session(
already-open transaction changes nothing about that transaction — the
startup bootstrap seeding did exactly that and wrote a `tenant_members`
row on a connection that had never been told which tenant it was for.
+
+ Inside a `shared_tenant_session` block for the same factory and tenant,
+ opened by the same task, the session is that block's rather than a new one.
"""
+ lent = _lent.get()
+ if lent is not None and lent.serves(session_factory, tenant_id):
+ with tenant_scope(tenant_id):
+ yield lent.session
+ return
with tenant_scope(tenant_id):
async with session_factory() as session:
yield session
+
+
+def holds_shared_session(
+ session_factory: async_sessionmaker[AsyncSession], tenant_id: str
+) -> bool:
+ """Whether this task is inside a `shared_tenant_session` block for this
+ factory and tenant."""
+ lent = _lent.get()
+ return lent is not None and lent.serves(session_factory, tenant_id)
+
+
+@asynccontextmanager
+async def shared_tenant_session(
+ session_factory: async_sessionmaker[AsyncSession], tenant_id: str
+) -> AsyncIterator[AsyncSession]:
+ """One session for a stretch of work that would otherwise open several.
+
+ Every `tenant_session` opened inside the block for the same factory and
+ tenant, by the task that opened the block, is this session. Work that
+ opened a session per question then costs one pool checkout and one
+ transaction instead of one of each per question. A `commit` inside the
+ block ends the transaction, not the session: the next statement begins a
+ new one, and `after_begin` stamps the tenant on it as on any other.
+
+ Leaving an inner `tenant_session` closes and rolls back nothing. An
+ uncommitted write in one inner block is committed by the next inner
+ `commit`, and an inner `rollback` (as in an `except` around a write) rolls
+ back the whole shared transaction. A write added to such a block has to
+ commit or roll back with that in mind.
+
+ The block holds a connection from its first statement to its end, so it
+ must contain database work only. Nothing in it may wait on a client, a
+ socket or a timer, and a generator must not yield inside it, or the
+ connection is held for as long as its consumer takes to come back. A block
+ must not open another for the same tenant: that raises.
+ """
+ if holds_shared_session(session_factory, tenant_id):
+ raise RuntimeError(
+ "a shared tenant session is already open in this task for this tenant"
+ )
+ async with tenant_session(session_factory, tenant_id) as session:
+ token = _lent.set(
+ _Lent(session_factory, tenant_id, asyncio.current_task(), session)
+ )
+ try:
+ yield session
+ finally:
+ try:
+ _lent.reset(token)
+ except ValueError:
+ # Unwound by GeneratorExit in another context, a frame being
+ # garbage collected: nothing there to restore, for the reason
+ # `tenant_context._held` gives.
+ pass
diff --git a/core/switch_core/main.py b/core/switch_core/main.py
index 79e455632..69c7b2585 100644
--- a/core/switch_core/main.py
+++ b/core/switch_core/main.py
@@ -36,7 +36,10 @@ from switch_core.bridges.agent.protocol.agent_core import AgentCore
from switch_core.bridges.agent.protocol.agent_event_listener import AgentEventListener
from switch_core.bridges.agent.protocol.connection_rows import SharedConnections
from switch_core.bridges.agent.protocol.controller_presence import ControllerPresence
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
+from switch_core.bridges.agent.protocol.event_buffer import (
+ EventBuffer,
+ session_slots_for,
+)
from switch_core.bridges.agent.registration_bootstrap import (
BOOTSTRAP_KEY_LABEL,
BOOTSTRAP_KEY_TYPE,
@@ -761,6 +764,7 @@ async def run(config: SwitchConfig, process_id: str) -> None:
sequence_base=event_boot << 32,
rooms=message_listener,
retention_seconds=config.agent_event_retention_seconds,
+ session_slots=session_slots_for(config.db_pool_size, config.db_max_overflow),
)
# Listens for the `NOTIFY` that each insert into `agent_events` sends, and
# wakes this process's streams for that agent, whichever process wrote it.
diff --git a/core/tests/benchmarks/server.py b/core/tests/benchmarks/server.py
index a3278a1c6..71a2d75f2 100644
--- a/core/tests/benchmarks/server.py
+++ b/core/tests/benchmarks/server.py
@@ -36,7 +36,10 @@ from switch_core.bridges.agent.app import create_agent_bridge_app
from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
from switch_core.bridges.agent.protocol.agent_core import AgentCore
from switch_core.bridges.agent.protocol.agent_event_listener import AgentEventListener
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
+from switch_core.bridges.agent.protocol.event_buffer import (
+ EventBuffer,
+ session_slots_for,
+)
from switch_core.bridges.agent.protocol.types import (
IntegrationProfile,
)
@@ -472,6 +475,7 @@ async def _serve(
sequence_base=event_boot << 32,
rooms=message_listener,
retention_seconds=config.agent_event_retention_seconds,
+ session_slots=session_slots_for(config.db_pool_size, config.db_max_overflow),
)
agent_event_listener = AgentEventListener(
lambda: create_unpooled_engine(config), event_buffer
diff --git a/core/tests/db_buffer.py b/core/tests/db_buffer.py
index 4cfc7e25e..2b9899484 100644
--- a/core/tests/db_buffer.py
+++ b/core/tests/db_buffer.py
@@ -61,6 +61,7 @@ def db_buffer_on(
sequence_base: int = 0,
retention_seconds: float = 3600,
rooms: RoomAnnouncements | None = None,
+ session_slots: int | None = None,
) -> EventBuffer:
"""The db-buffer on a test's database."""
return EventBuffer(
@@ -68,6 +69,7 @@ def db_buffer_on(
sequence_base=sequence_base,
rooms=rooms if rooms is not None else QuietRooms(),
retention_seconds=retention_seconds,
+ session_slots=session_slots,
)
diff --git a/core/tests/integration/conftest.py b/core/tests/integration/conftest.py
index 36b55c0e0..04756fab7 100644
--- a/core/tests/integration/conftest.py
+++ b/core/tests/integration/conftest.py
@@ -44,7 +44,10 @@ from switch_core.bridges.agent.api.handlers import (
from switch_core.bridges.agent.api_key_cache import ApiKeyCache
from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
from switch_core.bridges.agent.protocol.agent_core import AgentCore
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
+from switch_core.bridges.agent.protocol.event_buffer import (
+ EventBuffer,
+ session_slots_for,
+)
from switch_core.bridges.agent.protocol.types import (
IntegrationProfile,
RegistrationResult,
@@ -585,6 +588,9 @@ async def harness(
sequence_base=0,
rooms=message_listener,
retention_seconds=config.agent_event_retention_seconds,
+ session_slots=session_slots_for(
+ config.db_pool_size, config.db_max_overflow
+ ),
)
connections = AgentConnectionRegistry()
collab_lifecycle: object = _NoBridges()
diff --git a/core/tests/session_counting.py b/core/tests/session_counting.py
new file mode 100644
index 000000000..2a812ff17
--- /dev/null
+++ b/core/tests/session_counting.py
@@ -0,0 +1,20 @@
+"""A session factory that counts the sessions opened through it, for tests
+that pin how many pool checkouts a path costs. One session is one checkout."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+
+class CountingSessionFactory:
+ """The real factory, counting the sessions opened through it."""
+
+ def __init__(self, inner: async_sessionmaker[AsyncSession]) -> None:
+ self._inner = inner
+ self.opened = 0
+
+ def __call__(self) -> Any:
+ self.opened += 1
+ return self._inner()
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_session_slots.py b/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_session_slots.py
new file mode 100644
index 000000000..24e2b82ad
--- /dev/null
+++ b/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_session_slots.py
@@ -0,0 +1,109 @@
+"""`EventBuffer.one_session` lets at most `session_slots` blocks hold a
+connection at once, so the streams' passes cannot take the whole pool from
+requests when a pod is saturated."""
+
+from __future__ import annotations
+
+import asyncio
+
+import pytest
+from sqlalchemy import text
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.models import TENANT_ZERO_ID
+from tests.db_buffer import db_buffer_on
+
+
+async def _hold(
+ buffer, inside: list[int], most: list[int], release: asyncio.Event
+) -> None:
+ async with buffer.one_session(TENANT_ZERO_ID) as session:
+ await session.execute(text("select 1"))
+ inside[0] += 1
+ most[0] = max(most[0], inside[0])
+ await release.wait()
+ inside[0] -= 1
+
+
+async def test_blocks_over_the_limit_wait_their_turn(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ buffer = db_buffer_on(session_factory, session_slots=2)
+ inside, most, release = [0], [0], asyncio.Event()
+ tasks = [
+ asyncio.create_task(_hold(buffer, inside, most, release)) for _ in range(5)
+ ]
+ for _ in range(50):
+ if inside[0] == 2:
+ break
+ await asyncio.sleep(0.01)
+ await asyncio.sleep(0.05)
+
+ assert inside[0] == 2
+
+ release.set()
+ await asyncio.wait_for(asyncio.gather(*tasks), timeout=5)
+ assert most[0] == 2
+
+
+async def test_a_block_that_fails_gives_its_slot_back(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ buffer = db_buffer_on(session_factory, session_slots=1)
+ with pytest.raises(RuntimeError, match="boom"):
+ async with buffer.one_session(TENANT_ZERO_ID):
+ raise RuntimeError("boom")
+
+ async with asyncio.timeout(2):
+ async with buffer.one_session(TENANT_ZERO_ID) as session:
+ assert (await session.execute(text("select 1"))).scalar_one() == 1
+
+
+async def test_a_cancelled_waiter_takes_no_slot(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ buffer = db_buffer_on(session_factory, session_slots=1)
+ inside, most, release = [0], [0], asyncio.Event()
+ holder = asyncio.create_task(_hold(buffer, inside, most, release))
+ await asyncio.sleep(0.05)
+ waiter = asyncio.create_task(_hold(buffer, inside, most, release))
+ await asyncio.sleep(0.05)
+ waiter.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await waiter
+
+ release.set()
+ await asyncio.wait_for(holder, timeout=5)
+ async with asyncio.timeout(2):
+ async with buffer.one_session(TENANT_ZERO_ID):
+ pass
+
+
+async def test_without_a_limit_every_block_goes_in(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ buffer = db_buffer_on(session_factory)
+ inside, most, release = [0], [0], asyncio.Event()
+ tasks = [
+ asyncio.create_task(_hold(buffer, inside, most, release)) for _ in range(5)
+ ]
+ for _ in range(100):
+ if inside[0] == 5:
+ break
+ await asyncio.sleep(0.01)
+
+ assert inside[0] == 5
+
+ release.set()
+ await asyncio.wait_for(asyncio.gather(*tasks), timeout=5)
+
+
+async def test_a_block_inside_another_raises_rather_than_waiting_on_its_own_slot(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ buffer = db_buffer_on(session_factory, session_slots=1)
+ async with buffer.one_session(TENANT_ZERO_ID):
+ with pytest.raises(RuntimeError, match="inside another"):
+ async with asyncio.timeout(2):
+ async with buffer.one_session(TENANT_ZERO_ID):
+ pass
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_pool_checkouts.py b/core/tests/switch_core/bridges/agent/protocol/test_pool_checkouts.py
index 2e2468a29..ea69d001f 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_pool_checkouts.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_pool_checkouts.py
@@ -34,22 +34,7 @@ from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.budget_store import BudgetStore
from switch_core.db.stores.room_store import RoomStore
from switch_core.db.stores.usage_store import UsageStore
-
-
-class _CountingSessionFactory:
- """The real factory, counting the sessions opened through it.
-
- One session is one pool checkout, which is the quantity this whole change
- is about.
- """
-
- def __init__(self, inner: async_sessionmaker[AsyncSession]) -> None:
- self._inner = inner
- self.opened = 0
-
- def __call__(self) -> Any:
- self.opened += 1
- return self._inner()
+from tests.session_counting import CountingSessionFactory
class _NoBridges:
@@ -68,7 +53,7 @@ class _OneBridge:
self.typing.append((room_id, agent_name, is_typing))
-def _service(counting: _CountingSessionFactory) -> AgentCore:
+def _service(counting: CountingSessionFactory) -> AgentCore:
svc = object.__new__(AgentCore)
svc.session_factory = counting # type: ignore[attr-defined]
svc.agent_store = AgentStore() # type: ignore[attr-defined]
@@ -136,7 +121,7 @@ class TestRequireRoomMember:
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
room_id, member_id, _ = await _seed(session_factory)
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
svc = _service(counting)
room = await svc.require_room_member(member_id, room_id)
@@ -148,7 +133,7 @@ class TestRequireRoomMember:
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
room_id, _, outsider_id = await _seed(session_factory)
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
svc = _service(counting)
with pytest.raises(PermissionError):
@@ -162,7 +147,7 @@ class TestRequireRoomMember:
# "No such room" and "not your room" must stay distinguishable now that
# the two reads have been folded into one statement.
_, member_id, _ = await _seed(session_factory)
- svc = _service(_CountingSessionFactory(session_factory))
+ svc = _service(CountingSessionFactory(session_factory))
with pytest.raises(ValueError, match="Room not found"):
await svc.require_room_member(member_id, "no-such-room")
@@ -171,7 +156,7 @@ class TestRequireRoomMember:
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
room_id, member_id, _ = await _seed(session_factory)
- svc = _service(_CountingSessionFactory(session_factory))
+ svc = _service(CountingSessionFactory(session_factory))
room = await svc.require_room_member(member_id, room_id)
@@ -193,7 +178,7 @@ class TestCanAddress:
)
await session.commit()
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
svc = _service(counting)
async with session_factory() as session:
target = await svc.agent_store.get(session, member_id)
@@ -223,7 +208,7 @@ class TestCanAddress:
)
await session.commit()
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
svc = _service(counting)
async with session_factory() as session:
target = await svc.agent_store.get(session, member_id)
@@ -245,7 +230,7 @@ class TestCanAddress:
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
_, member_id, outsider_id = await _seed(session_factory)
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
svc = _service(counting)
async with session_factory() as session:
@@ -272,7 +257,7 @@ class TestTyping:
# Was three checkouts: membership took two of its own, then a third
# re-read the agent and the room the caller had already resolved.
room_id, member_id, _ = await _seed(session_factory)
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
svc = _service(counting)
await svc.set_typing(member_id, room_id, True)
@@ -283,7 +268,7 @@ class TestTyping:
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
_, member_id, _ = await _seed(session_factory)
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
svc = _service(counting)
bridge = _OneBridge()
svc.collab_lifecycle = bridge # type: ignore[assignment]
@@ -310,7 +295,7 @@ class TestSendMessage:
# agent for the typing indicator — and no more than that, because the
# room the caller already resolved is passed along instead of re-read.
room_id, member_id, _ = await _seed(session_factory)
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
svc = _service(counting)
sent: list[tuple[str, str]] = []
@@ -337,7 +322,7 @@ class TestSendMessage:
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
room_id, _, outsider_id = await _seed(session_factory)
- svc = _service(_CountingSessionFactory(session_factory))
+ svc = _service(CountingSessionFactory(session_factory))
with pytest.raises(PermissionError):
await svc.send_message(outsider_id, room_id, "hello")
@@ -367,7 +352,7 @@ class TestSendMessage:
amount=2,
)
await session.commit()
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
svc = _service(counting)
with pytest.raises(BudgetExceeded, match="member has reached its budget"):
@@ -383,7 +368,7 @@ class TestAgentStatuses:
# The gateway builds a room's detail inside its request transaction.
# Resolving statuses used to open a second one from underneath it.
room_id, member_id, _ = await _seed(session_factory)
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
svc = _service(counting)
svc.agent_session_store = AgentSessionStore() # type: ignore[attr-defined]
@@ -400,7 +385,7 @@ class TestAgentStatuses:
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
room_id, member_id, _ = await _seed(session_factory)
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
svc = _service(counting)
svc.agent_session_store = AgentSessionStore() # type: ignore[attr-defined]
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_stream.py b/core/tests/switch_core/bridges/agent/protocol/test_stream.py
index 8f68dfbf9..4d001b881 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_stream.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_stream.py
@@ -11,7 +11,7 @@ from __future__ import annotations
import asyncio
import inspect
from collections.abc import Callable
-from typing import Any
+from typing import Any, cast
import pytest
from sqlalchemy import select
@@ -61,6 +61,7 @@ from tests.db_buffer import (
seed_member,
sender_of,
)
+from tests.session_counting import CountingSessionFactory
AGENT = "agent-1"
ROOM_A = "room-a"
@@ -1638,3 +1639,30 @@ class TestBothTransportsEncodeTheSameFrames:
assert first.startswith(b"event: connection_state\n")
assert not conn.stream_attached
assert registry.get(conn.id) is conn
+
+
+async def test_a_pass_reads_in_one_session(
+ session_factory: async_sessionmaker[AsyncSession], room_bells: MessageListener
+) -> None:
+ """The reads of a pass that delivers (the agent's rooms, its head, its own
+ events, what was dropped after its position, its rooms' conversation and
+ the unread counts) cost one pool checkout and one transaction, not one of
+ each per read. A contract like those in `test_pool_checkouts.py`: every
+ message wakes a pass in each stream it is for."""
+ counting = CountingSessionFactory(session_factory)
+ buffer = db_buffer_on(cast(Any, counting), rooms=room_bells)
+ registry = AgentConnectionRegistry()
+ conn = _open(registry, scope="all", delivery_filter="all")
+ stream = event_frames(conn=conn, registry=registry, buffer=buffer, approvals=None)
+ await _addressed(session_factory, buffer, "first")
+ opening = await _take(stream, 2)
+ assert [name for name, _ in opening] == ["connection_state", "message"]
+
+ await _addressed(session_factory, buffer, "second")
+ counting.opened = 0
+ ((name, data),) = await _take(stream, 1)
+ await stream.aclose()
+
+ assert name == "message"
+ assert data["payload"]["body"] == "second"
+ assert counting.opened == 1
diff --git a/core/tests/switch_core/bridges/agent/test_api_key_cache.py b/core/tests/switch_core/bridges/agent/test_api_key_cache.py
index cb42e560d..302abb14e 100644
--- a/core/tests/switch_core/bridges/agent/test_api_key_cache.py
+++ b/core/tests/switch_core/bridges/agent/test_api_key_cache.py
@@ -30,6 +30,7 @@ from switch_core.db.models import Agent, ApiKey, Client, User
from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.api_key_store import ApiKeyStore
from switch_core.keys import Keyring
+from tests.session_counting import CountingSessionFactory
_PROFILE = IntegrationProfile(
connection_model="session_passive",
@@ -40,21 +41,6 @@ _PROFILE = IntegrationProfile(
)
-class _CountingSessionFactory:
- """Wraps the real factory and counts how many sessions are opened.
-
- A session is a pool checkout, so this is the number the incident is about.
- """
-
- def __init__(self, inner: async_sessionmaker[AsyncSession]) -> None:
- self._inner = inner
- self.opened = 0
-
- def __call__(self) -> Any:
- self.opened += 1
- return self._inner()
-
-
class _FakeClientLifecycle:
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
self._session_factory = session_factory
@@ -242,7 +228,7 @@ class TestResolutionHitsAndMisses:
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
await _seed_agent(session_factory, "tok")
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
counting.opened = 0
mw = _middleware(counting, ApiKeyCache(ttl_seconds=5, max_entries=8))
@@ -270,7 +256,7 @@ class TestResolutionHitsAndMisses:
# on a tenant_session to read the row itself. What the cache above
# saves is these two round trips, not one.
await _seed_agent(session_factory, "tok")
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
counting.opened = 0
mw = _middleware(counting, ApiKeyCache(ttl_seconds=0, max_entries=8))
@@ -285,7 +271,7 @@ class TestResolutionHitsAndMisses:
# tenant to open the second, row-reading session under. This is the
# load-shape that matters — an unauthenticated flood never reaches
# the second query.
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
cache = ApiKeyCache(ttl_seconds=5, max_entries=8)
mw = _middleware(counting, cache)
@@ -298,7 +284,7 @@ class TestResolutionHitsAndMisses:
# A token that does not resolve must reach Postgres every time, or a
# key that starts existing would keep failing — and, far worse, the
# memo would become a place to park an attacker-chosen string.
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
cache = ApiKeyCache(ttl_seconds=5, max_entries=8)
mw = _middleware(counting, cache)
@@ -318,7 +304,7 @@ class TestResolutionHitsAndMisses:
key_type="registration",
with_agent=False,
)
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
counting.opened = 0
cache = ApiKeyCache(ttl_seconds=5, max_entries=8)
mw = _middleware(counting, cache)
@@ -337,7 +323,7 @@ class TestResolutionHitsAndMisses:
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
await _seed_agent(session_factory, "orphan", name="orphan", with_agent=False)
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
counting.opened = 0
cache = ApiKeyCache(ttl_seconds=5, max_entries=8)
mw = _middleware(counting, cache)
@@ -354,7 +340,7 @@ class TestResolutionHitsAndMisses:
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
await _seed_agent(session_factory, "tok")
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
counting.opened = 0
mw = _middleware(counting, ApiKeyCache(ttl_seconds=0, max_entries=8))
@@ -373,7 +359,7 @@ class TestResolutionHitsAndMisses:
monkeypatch.setattr(
"switch_core.bridges.agent.api_key_cache.time.monotonic", lambda: now[0]
)
- counting = _CountingSessionFactory(session_factory)
+ counting = CountingSessionFactory(session_factory)
counting.opened = 0
mw = _middleware(counting, ApiKeyCache(ttl_seconds=5, max_entries=8))
diff --git a/core/tests/switch_core/db/test_pool_pre_ping.py b/core/tests/switch_core/db/test_pool_pre_ping.py
new file mode 100644
index 000000000..a7b695248
--- /dev/null
+++ b/core/tests/switch_core/db/test_pool_pre_ping.py
@@ -0,0 +1,105 @@
+"""What turning the pool's pre-ping off costs (`db_pool_pre_ping`).
+
+A pooled connection the server closed while it sat in the pool is the case
+the pre-ping exists for: a database restart closes every connection the pool
+holds. With the pre-ping off, the first query on one of them fails, and
+SQLAlchemy then treats every connection the pool held at that moment as
+stale and replaces it on its next checkout. So a restart costs one failed
+query, not one per pooled connection, which is the trade the default makes.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+import pytest
+from sqlalchemy import text
+from sqlalchemy.exc import DBAPIError
+from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine
+
+from switch_core.config import SwitchConfig
+from switch_core.db.engine import create_engine_from_config, create_unpooled_engine
+from tests.switch_core.db.test_dead_peer_detection import _config_for
+
+POOLED = 3
+
+
+async def _pid(conn: AsyncConnection) -> int:
+ return int((await conn.execute(text("select pg_backend_pid()"))).scalar_one())
+
+
+async def _fill_and_close_pool(
+ postgres_url: str, *, pre_ping: bool
+) -> tuple[AsyncEngine, set[int]]:
+ """An engine whose pool holds `POOLED` connections, every one of which the
+ server has since closed, as a restart does. Returns the engine and the
+ closed connections' backend pids."""
+ engine = create_engine_from_config(
+ _config_for(
+ postgres_url,
+ db_pool_size=POOLED,
+ db_max_overflow=0,
+ db_pool_pre_ping=pre_ping,
+ )
+ )
+ held = [await engine.connect() for _ in range(POOLED)]
+ pids = {await _pid(conn) for conn in held}
+ for conn in held:
+ await conn.close()
+ closer = create_unpooled_engine(_config_for(postgres_url))
+ try:
+ async with closer.connect() as conn:
+ for pid in pids:
+ await conn.execute(
+ text("select pg_terminate_backend(:pid)"), {"pid": pid}
+ )
+ # Termination is a signal: wait until the backends have exited, or
+ # one could still answer a ping and then die under the next query.
+ for _ in range(100):
+ left = await conn.scalar(
+ text(
+ "select count(*) from pg_stat_activity where pid = any(:pids)"
+ ),
+ {"pids": list(pids)},
+ )
+ if not left:
+ break
+ await asyncio.sleep(0.05)
+ else:
+ raise AssertionError("the closed backends did not exit")
+ finally:
+ await closer.dispose()
+ return engine, pids
+
+
+def test_the_pre_ping_is_off_by_default() -> None:
+ assert SwitchConfig.model_fields["db_pool_pre_ping"].default is False
+
+
+async def test_without_the_pre_ping_a_restart_costs_one_failed_query(
+ postgres_url: str,
+) -> None:
+ engine, closed = await _fill_and_close_pool(postgres_url, pre_ping=False)
+ try:
+ with pytest.raises(DBAPIError) as failed:
+ async with engine.connect() as conn:
+ await _pid(conn)
+ assert failed.value.connection_invalidated
+
+ # Every connection after it is a new one, though two more of the
+ # closed ones were still in the pool when the first query failed.
+ for _ in range(POOLED):
+ async with engine.connect() as conn:
+ assert await _pid(conn) not in closed
+ finally:
+ await engine.dispose()
+
+
+async def test_with_the_pre_ping_no_query_fails(postgres_url: str) -> None:
+ engine, closed = await _fill_and_close_pool(postgres_url, pre_ping=True)
+ try:
+ for _ in range(POOLED):
+ async with engine.connect() as conn:
+ assert await _pid(conn) not in closed
+ finally:
+ await engine.dispose()
diff --git a/core/tests/switch_core/db/test_session_scope.py b/core/tests/switch_core/db/test_session_scope.py
index 9b172a19e..3f8d7160e 100644
--- a/core/tests/switch_core/db/test_session_scope.py
+++ b/core/tests/switch_core/db/test_session_scope.py
@@ -18,12 +18,16 @@ is pinned in `test_tenant_exemption_allowlist.py`.
from __future__ import annotations
+import asyncio
+from typing import Any
+
import pytest
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-from switch_core.db.session_scope import tenant_session
+from switch_core.db.session_scope import shared_tenant_session, tenant_session
from switch_core.tenant_context import current_tenant_id, tenant_scope
+from tests.session_counting import CountingSessionFactory
TENANT_A = "tenant-session-scope-a"
TENANT_B = "tenant-session-scope-b"
@@ -86,3 +90,108 @@ class TestTenantSession:
async with tenant_session(session_factory, TENANT_B) as session:
assert await _current_setting(session) == TENANT_B
await session.commit()
+
+
+class TestSharedTenantSession:
+ async def test_a_session_opened_inside_the_block_is_the_shared_session(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ counting: Any = CountingSessionFactory(session_factory)
+ async with shared_tenant_session(counting, TENANT_A) as shared:
+ async with tenant_session(counting, TENANT_A) as first:
+ assert await _current_setting(first) == TENANT_A
+ async with tenant_session(counting, TENANT_A) as second:
+ assert await _current_setting(second) == TENANT_A
+ assert first is shared
+ assert second is shared
+ assert counting.opened == 1
+
+ async def test_another_tenant_opens_its_own(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ async with shared_tenant_session(session_factory, TENANT_A) as shared:
+ assert await _current_setting(shared) == TENANT_A
+ async with tenant_session(session_factory, TENANT_B) as other:
+ assert other is not shared
+ assert await _current_setting(other) == TENANT_B
+ assert await _current_setting(shared) == TENANT_A
+
+ async def test_another_task_opens_its_own(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """A task started inside the block inherits its context, but two tasks
+ must never use one `AsyncSession` at once."""
+
+ async def opened() -> AsyncSession:
+ async with tenant_session(session_factory, TENANT_A) as session:
+ await _current_setting(session)
+ return session
+
+ async with shared_tenant_session(session_factory, TENANT_A) as shared:
+ assert await asyncio.create_task(opened()) is not shared
+
+ async def test_nothing_is_lent_once_the_block_exits(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ async with shared_tenant_session(session_factory, TENANT_A) as shared:
+ pass
+ async with tenant_session(session_factory, TENANT_A) as after:
+ assert after is not shared
+
+ async def test_an_inner_block_leaves_the_session_open(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ async with shared_tenant_session(session_factory, TENANT_A) as shared:
+ async with tenant_session(session_factory, TENANT_A) as inner:
+ await _current_setting(inner)
+ assert await _current_setting(shared) == TENANT_A
+
+ async def test_a_commit_inside_the_block_keeps_the_tenant(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """A commit ends the transaction, and `set_config` is local to it; the
+ next transaction is stamped again by `after_begin`."""
+ async with shared_tenant_session(session_factory, TENANT_A):
+ async with tenant_session(session_factory, TENANT_A) as session:
+ assert await _current_setting(session) == TENANT_A
+ await session.commit()
+ async with tenant_session(session_factory, TENANT_A) as session:
+ assert await _current_setting(session) == TENANT_A
+
+ @pytest.mark.no_ambient_tenant
+ async def test_the_tenant_is_unbound_once_the_block_exits(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ with pytest.raises(RuntimeError, match="boom"):
+ async with shared_tenant_session(session_factory, TENANT_A):
+ async with tenant_session(session_factory, TENANT_A):
+ raise RuntimeError("boom")
+ assert current_tenant_id() is None
+ async with tenant_session(session_factory, TENANT_A) as session:
+ assert await _current_setting(session) == TENANT_A
+
+ async def test_a_block_inside_another_for_the_same_tenant_raises(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ async with shared_tenant_session(session_factory, TENANT_A):
+ with pytest.raises(RuntimeError, match="already open"):
+ async with shared_tenant_session(session_factory, TENANT_A):
+ pass
+
+ async def test_a_block_left_by_closing_its_generator_lends_nothing_after(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """Yielding inside a block is forbidden, but if a generator does and is
+ then closed, later sessions in the same task are not handed the closed
+ one."""
+
+ async def holds() -> Any:
+ async with shared_tenant_session(session_factory, TENANT_A) as session:
+ yield session
+
+ frames = holds()
+ shared = await anext(frames)
+ await frames.aclose()
+
+ async with tenant_session(session_factory, TENANT_A) as after:
+ assert after is not shared
diff --git a/core/tests/switch_core/gateway/test_gateway_ui.py b/core/tests/switch_core/gateway/test_gateway_ui.py
index c069434c0..fc5c9c4c3 100644
--- a/core/tests/switch_core/gateway/test_gateway_ui.py
+++ b/core/tests/switch_core/gateway/test_gateway_ui.py
@@ -168,7 +168,9 @@ def _agent_bridge_app(gateway_ui: GatewayUi | None) -> FastAPI:
room_service=object(), # type: ignore[arg-type]
client_lifecycle=object(), # type: ignore[arg-type]
collab_lifecycle=object(), # type: ignore[arg-type]
- event_buffer=EventBuffer(object(), sequence_base=0, rooms=QuietRooms()), # type: ignore[arg-type]
+ event_buffer=EventBuffer(
+ object(), sequence_base=0, rooms=QuietRooms(), session_slots=None
+ ), # type: ignore[arg-type]
resource_service=object(), # type: ignore[arg-type]
api_key_store=object(), # type: ignore[arg-type]
external_user_store=object(), # type: ignore[arg-type]
diff --git a/core/tests/switch_core/management/test_middleware.py b/core/tests/switch_core/management/test_middleware.py
index 14b5951bb..06f9626a9 100644
--- a/core/tests/switch_core/management/test_middleware.py
+++ b/core/tests/switch_core/management/test_middleware.py
@@ -310,7 +310,9 @@ class TestTheFlagOff:
room_service=object(), # type: ignore[arg-type]
client_lifecycle=object(), # type: ignore[arg-type]
collab_lifecycle=object(), # type: ignore[arg-type]
- event_buffer=EventBuffer(object(), sequence_base=0, rooms=QuietRooms()), # type: ignore[arg-type]
+ event_buffer=EventBuffer(
+ object(), sequence_base=0, rooms=QuietRooms(), session_slots=None
+ ), # type: ignore[arg-type]
resource_service=object(), # type: ignore[arg-type]
api_key_store=object(), # type: ignore[arg-type]
external_user_store=object(), # type: ignore[arg-type]
diff --git a/core/tests/switch_core/management/test_stream.py b/core/tests/switch_core/management/test_stream.py
index 7c3471caa..39b3b3288 100644
--- a/core/tests/switch_core/management/test_stream.py
+++ b/core/tests/switch_core/management/test_stream.py
@@ -38,6 +38,7 @@ from switch_core.management.notifier import (
ControllerNotifier,
)
from tests.db_buffer import age_events, say
+from tests.session_counting import CountingSessionFactory
from tests.switch_core.management.harness import (
Harness,
add_member,
@@ -224,6 +225,32 @@ class TestOpening:
assert data["event"]["payload"]["body"] == "@you"
assert data["event"]["missed"] == {"count": 2, "reason": None}
+ async def test_an_agents_pass_reads_in_one_session(
+ self, harness: Harness, monkeypatch: pytest.MonkeyPatch
+ ) -> None:
+ """The agent's head, its own events, what was dropped after its cursor
+ and the unread counts cost one pool checkout and one transaction, not
+ one of each per read."""
+ owner = await add_member(harness.session_factory, "ada")
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+ room_id = await add_room(harness.session_factory, agent_id)
+ opened = await open_connection(client, controller)
+
+ stream = await open_stream(harness, controller, opened)
+ await take(stream, 2)
+ await _ask(harness, agent_id, room_id, "@you")
+ buffer = harness.protocol.event_buffer
+ counting = CountingSessionFactory(buffer._session_factory)
+ monkeypatch.setattr(buffer, "_session_factory", counting)
+ [(name, data)] = await take(stream, 1)
+ await stream.aclose()
+
+ assert name == "agent.event"
+ assert data["event"]["payload"]["body"] == "@you"
+ assert counting.opened == 1
+
async def test_a_cursor_from_before_a_restart_is_a_gap(
self, harness: Harness
) -> None:
diff --git a/core/tests/switch_core/observability/test_connection_metrics.py b/core/tests/switch_core/observability/test_connection_metrics.py
index 5f11b48bb..094caaa21 100644
--- a/core/tests/switch_core/observability/test_connection_metrics.py
+++ b/core/tests/switch_core/observability/test_connection_metrics.py
@@ -251,7 +251,12 @@ class _EmptyBuffer(EventBuffer):
"""
def __init__(self) -> None:
- super().__init__(object(), sequence_base=0, rooms=QuietRooms()) # type: ignore[arg-type]
+ super().__init__(
+ object(), # type: ignore[arg-type]
+ sequence_base=0,
+ rooms=QuietRooms(),
+ session_slots=None,
+ )
async def head(self, tenant_id: str, agent_id: str) -> int:
return self.sequence_floor - 1
diff --git a/core/tests/switch_core/telemetry/test_emitters_are_live.py b/core/tests/switch_core/telemetry/test_emitters_are_live.py
index 59bb2afdf..f763415c6 100644
--- a/core/tests/switch_core/telemetry/test_emitters_are_live.py
+++ b/core/tests/switch_core/telemetry/test_emitters_are_live.py
@@ -53,7 +53,9 @@ def _build(telemetry: TelemetryService) -> Any:
room_service=object(), # type: ignore[arg-type]
client_lifecycle=object(), # type: ignore[arg-type]
collab_lifecycle=object(), # type: ignore[arg-type]
- event_buffer=EventBuffer(object(), sequence_base=0, rooms=QuietRooms()), # type: ignore[arg-type]
+ event_buffer=EventBuffer(
+ object(), sequence_base=0, rooms=QuietRooms(), session_slots=None
+ ), # type: ignore[arg-type]
resource_service=object(), # type: ignore[arg-type]
api_key_store=object(), # type: ignore[arg-type]
external_user_store=object(), # type: ignore[arg-type]
diff --git a/core/tests/switch_core/test_ingress_agent_api_paths.py b/core/tests/switch_core/test_ingress_agent_api_paths.py
index c9a88450d..9c525acdb 100644
--- a/core/tests/switch_core/test_ingress_agent_api_paths.py
+++ b/core/tests/switch_core/test_ingress_agent_api_paths.py
@@ -54,7 +54,9 @@ def _served_prefixes() -> set[str]:
room_service=object(), # type: ignore[arg-type]
client_lifecycle=object(), # type: ignore[arg-type]
collab_lifecycle=object(), # type: ignore[arg-type]
- event_buffer=EventBuffer(object(), sequence_base=0, rooms=QuietRooms()), # type: ignore[arg-type]
+ event_buffer=EventBuffer(
+ object(), sequence_base=0, rooms=QuietRooms(), session_slots=None
+ ), # type: ignore[arg-type]
resource_service=object(), # type: ignore[arg-type]
api_key_store=object(), # type: ignore[arg-type]
external_user_store=object(), # type: ignore[arg-type]
diff --git a/docs/old/rds-migration.md b/docs/old/rds-migration.md
index c07977655..632c40fd0 100644
--- a/docs/old/rds-migration.md
+++ b/docs/old/rds-migration.md
@@ -157,8 +157,9 @@ investigated.
Two related points for whoever watches this:
-- `pool_pre_ping` is on, so pooled connections recover on their own after a
- failover rather than serving one error each.
+- `pool_pre_ping` is off by default since switch#1111. After a failover the
+ first query on each pod fails, along with any query that was running, and
+ SQLAlchemy then replaces every connection the pool held.
- The notification queue is bounded (8 GB) and fills only if a listener sits in
a long transaction. Ours does not, but a queue-full alarm is cheap.