Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Take an agent's head from the notification of its new event (CHOO-3667)
Abel Dantas16242358d4e8
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6376e560e..a0426c9ac 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -223,6 +223,12 @@ version of their own to them without also giving them a release of their own.group whose subgroup would move up next to a namesake is refused too, andthe dashboard now shows why. Names can no longer contain `/` or be blank.Existing duplicates are left as they are and can be reached by id.+- **Delivering an event to an agent behind a controller no longer reads the+ agent's head first.** The notification of a new agent event already carries+ its number, and the controller's stream now keeps it as the agent's head, so+ a delivery costs the agent's own read alone, where it read the heads of all+ the controller's agents first. The heads are still read after each idle+ interval, when an agent is attached, and after the listener reconnects.#### Fixed- **The agent HTTP API answers a refused or malformed request with a 4xx, notdiff --git a/core/switch_core/bridges/agent/protocol/agent_event_listener.py b/core/switch_core/bridges/agent/protocol/agent_event_listener.pyindex 37641cd39..4af301c59 100644--- a/core/switch_core/bridges/agent/protocol/agent_event_listener.py+++ b/core/switch_core/bridges/agent/protocol/agent_event_listener.py@@ -8,12 +8,14 @@ connection, so every process listens here and rings the agent's doorbell(`EventBuffer.ring`). The stream behind the connection then reads the rowspast its position from the table.-A notification only wakes a stream; the event itself is always read from the-table. If a notification is lost, the stream finds the event when its wait-times out and it reads again, so the event arrives later but still arrives.-Notifications sent while the listening connection is down are lost, so after-every reconnect every stream in this process is woken to look for events it-may have missed.+A notification wakes a stream and tells it the new event's number, which the+stream takes as the agent's head instead of reading it; the event itself is+always read from the table. If a notification is lost, the stream finds the+event when its wait times out and it reads the heads again, so the event+arrives later but still arrives. Notifications sent while the listening+connection is down are lost, so each time it starts or stops listening every+head this process was told is forgotten, and when it starts every stream is+woken to look for events it may have missed (`EventBuffer.listening`).Connection handling follows `MessageListener`: a dedicated connection outsidethe pool, a periodic heartbeat query, and reconnects with a capped backoff.@@ -104,7 +106,7 @@ class AgentEventListener:)await driver.add_listener(AGENT_EVENT_CHANNEL, self._on_notify)try:- self._buffer.ring_everyone()+ self._buffer.listening(True)self.connected.set()while self._running:await asyncio.sleep(HEARTBEAT_SECONDS)@@ -113,6 +115,7 @@ class AgentEventListener:)finally:self.connected.clear()+ self._buffer.listening(False)with contextlib.suppress(Exception):await driver.remove_listener(AGENT_EVENT_CHANNEL, self._on_notify@@ -123,16 +126,22 @@ class AgentEventListener:def _on_notify(self, _connection: Any, _pid: int, _channel: str, payload: str) -> None:- """Handle one notification by ringing the agent's doorbell.+ """Handle one notification by ringing the agent's doorbell with the+ new event's number.asyncpg calls this from the connection's read loop, so it does nothingthat waits and returns straight away."""try:- agent_id = json.loads(payload)["agent_id"]+ announced = json.loads(payload)+ agent_id = announced["agent_id"]except (ValueError, TypeError, KeyError):logger.warning("Ignoring an unreadable agent event announcement: %r", payload[:200])return- self._buffer.ring(agent_id)+ seq = announced.get("seq")+ self._buffer.ring(+ agent_id,+ seq if isinstance(seq, int) and not isinstance(seq, bool) else None,+ )diff --git a/core/switch_core/bridges/agent/protocol/controller_stream.py b/core/switch_core/bridges/agent/protocol/controller_stream.pyindex c51849558..cdff0bd71 100644--- a/core/switch_core/bridges/agent/protocol/controller_stream.py+++ b/core/switch_core/bridges/agent/protocol/controller_stream.py@@ -186,6 +186,12 @@ class _ControllerStream:)self._tenant: str = self._tenant_idself._attached: dict[str, _Attached] = {}+ # Whether the next pass reads every head from the database rather+ # than taking the heads this process was told (`EventBuffer.heads`):+ # the first pass, and the pass after an idle interval with no wake,+ # which is when a notification lost without its connection dropping+ # is caught.+ self._fresh_heads = Trueasync def run(self, opening: dict[str, Any]) -> AsyncGenerator[ControllerFrame]:conn = self._conn@@ -280,6 +286,7 @@ class _ControllerStream:continueif not await self._wait():+ self._fresh_heads = Trueyield IDLEfinally:for attached in self._attached.values():@@ -466,12 +473,17 @@ class _ControllerStream:async def _due(self) -> list[tuple[_Attached, int]]:"""The attached agents a delivery pass has work for, each with itshead: events past its cursor, or rooms without an unread-count baseline- yet. The heads are read in one query, so a pass over a controller's- agents, such as the one after each idle interval, costs one read when- nothing is new rather than a session per agent."""+ yet. The heads are the ones this process was told with each event+ (`EventBuffer.ring`), and those it was not told are read in one query,+ so a pass woken by an event reads nothing to find it, and a pass over+ a controller's agents after each idle interval costs one read rather+ than a session per agent."""if not self._attached:return []- heads = await self._buffer.heads(self._tenant, set(self._attached))+ heads = await self._buffer.heads(+ self._tenant, set(self._attached), fresh=self._fresh_heads+ )+ self._fresh_heads = Falsereturn [(attached, heads[attached.agent_id])for attached in list(self._attached.values())diff --git a/core/switch_core/bridges/agent/protocol/event_buffer.py b/core/switch_core/bridges/agent/protocol/event_buffer.pyindex d1a80a1c5..b4e2d22fa 100644--- a/core/switch_core/bridges/agent/protocol/event_buffer.py+++ b/core/switch_core/bridges/agent/protocol/event_buffer.py@@ -418,6 +418,19 @@ class EventBuffer:# listener sets it for own events written by other processes. The events# themselves are always read from `agent_events`.self._notify: dict[str, asyncio.Event] = {}+ # The head of each agent read here, as this process was last told it:+ # by the notification of an own event written anywhere, or by a write+ # of its own. Kept only while the notification listener is listening,+ # since a notification sent while it was not is lost, and forgotten+ # whenever that may have happened. A head only ever rises, so of two+ # values for one agent the larger is kept. A delivery pass takes its+ # heads from here and reads the database for an agent with none+ # (`heads`).+ self._told_heads: dict[str, int] = {}+ self._listening = False+ # Advanced each time the told heads are forgotten, so a head read from+ # the database before that is not kept after it (`heads`).+ self._told_epoch = 0@asynccontextmanagerasync def one_session(self, tenant_id: str) -> AsyncIterator[AsyncSession]:@@ -685,7 +698,7 @@ class EventBuffer:event.type,seq,)- self.ring(agent_id)+ self.ring(agent_id, seq)return seqasync def _store_own_event(@@ -846,26 +859,45 @@ class EventBuffer:"""return self._notify.setdefault(agent_id, asyncio.Event())- def ring(self, agent_id: str) -> None:- """Set the agent's doorbell in this process.+ def ring(self, agent_id: str, seq: int | None = None) -> None:+ """Set the agent's doorbell in this process, and keep `seq`, the+ number of the own event that rang it, as the agent's head here.- `enqueue` calls this after inserting an own event, and the notification- listener calls it when another process inserts one.+ `enqueue` and the router call this after inserting an own event, and+ the notification listener calls it when another process inserts one.+ An agent's numbers are handed out in commit order (`store`), so the+ event numbered `seq` has committed after every lower one: once it has,+ the agent's head is at least `seq`. A ring without a number forgets the+ head kept, and the next pass reads it."""notify = self._notify.get(agent_id)- if notify is not None:- notify.set()+ if notify is None:+ return+ if seq is None or not self._listening:+ self._told_heads.pop(agent_id, None)+ else:+ self._told_heads[agent_id] = max(seq, self._told_heads.get(agent_id, seq))+ notify.set()def ring_everyone(self) -> None:- """Set every doorbell in this process.+ """Set every doorbell in this process, and forget every head kept.- The notification listener calls this each time it starts listening on- a Postgres connection, because it may have missed notifications while- it was not listening.+ Called whenever notifications may have been missed, so every reader+ looks for events it was not told of."""+ self._told_heads.clear()+ self._told_epoch += 1for notify in self._notify.values():notify.set()+ def listening(self, now_listening: bool) -> None:+ """Told by the notification listener each time it starts or stops+ listening on a Postgres connection. Notifications sent while it was+ not listening are lost, so the heads kept are forgotten either way,+ and from the start of listening every reader looks again."""+ self._listening = now_listening+ self.ring_everyone()+async def head(self, tenant_id: str, agent_id: str) -> int:"""The number of the agent's newest own event, or the number justbefore its first one if it has none yet. This is the agent's head."""@@ -879,25 +911,49 @@ class EventBuffer:last = await session.scalar(query)return self.sequence_floor - 1 if last is None else last- async def heads(self, tenant_id: str, agent_ids: set[str]) -> dict[str, int]:- """`head` for several agents in one query, for a connection that- serves many agents."""+ async def heads(+ self, tenant_id: str, agent_ids: set[str], *, fresh: bool = False+ ) -> dict[str, int]:+ """`head` for several agents, for a connection that serves many agents.++ An agent's head is the one this process was told (`ring`), when it+ has one and `fresh` is not asked for. The others are read in one+ query, and kept as told while the listener goes on listening, since+ any event numbered after the read rings with its number.+ """+ heads: dict[str, int] = {}+ if not fresh:+ heads = {+ agent_id: self._told_heads[agent_id]+ for agent_id in agent_ids+ if agent_id in self._told_heads+ }+ unknown = agent_ids - heads.keys()+ if not unknown:+ return headsdef query(ids: set[str]) -> Any:return select(AgentEventHead.agent_id, AgentEventHead.last_seq).where(AgentEventHead.agent_id.in_(ids))+ epoch = self._told_epochasync with tenant_session(self._session_factory, tenant_id) as session:- found = dict((await session.execute(query(agent_ids))).tuples().all())- missing = agent_ids - found.keys()+ found = dict((await session.execute(query(unknown))).tuples().all())+ missing = unknown - found.keys()if missing:await self._start_numbering(session, missing)found.update((await session.execute(query(missing))).tuples().all())- return {- agent_id: found.get(agent_id, self.sequence_floor - 1)- for agent_id in agent_ids- }+ keep = self._listening and epoch == self._told_epoch+ for agent_id in unknown:+ head = max(+ found.get(agent_id, self.sequence_floor - 1),+ self._told_heads.get(agent_id, 0),+ )+ heads[agent_id] = head+ if keep and agent_id in self._notify:+ self._told_heads[agent_id] = head+ return headsasync def first_seq(self, tenant_id: str, agent_id: str) -> int:"""The number of the agent's first own event.@@ -1774,6 +1830,7 @@ class EventBuffer:row in `agents` (`ON DELETE CASCADE`)."""self._notify.pop(agent_id, None)+ self._told_heads.pop(agent_id, None)# ------------------------------------------------------------------# Retentiondiff --git a/core/switch_core/delivery/router.py b/core/switch_core/delivery/router.pyindex 44c8d416d..518a1c146 100644--- a/core/switch_core/delivery/router.py+++ b/core/switch_core/delivery/router.py@@ -538,7 +538,7 @@ class Routed:row: InboundEvent,*,seq: int,- stored: list[str],+ stored: list[tuple[str, int]],deadline_in: float | None,) -> None:self._router = router@@ -571,8 +571,8 @@ class Routed:def _committed(self) -> None:router = self._router- for agent_id in self._stored:- router.services.event_buffer.ring(agent_id)+ for agent_id, seq in self._stored:+ router.services.event_buffer.ring(agent_id, seq)reads = self._readsif reads is None:return@@ -883,7 +883,7 @@ class MessageRouter:)row = to_inbound(message, attachments, transport_room_id=transport_room_id)deadline_in: float | None = None- stored: list[str] = []+ stored: list[tuple[str, int]] = []if reads is not None:if plan.group is not None:deadline_in = await self._step_group(session, reads, plan, row)@@ -1332,10 +1332,11 @@ class MessageRouter:reads: RoomReads,deliveries: list[tuple[AgentRecipient, Delivery]],row: InboundEvent,- ) -> list[str]:+ ) -> list[tuple[str, int]]:"""Store every own event decided for `row`, in the order of the- agents' ids, and return the agents whose events were stored."""- stored: list[str] = []+ agents' ids, and return the agents whose events were stored, each with+ its event's number."""+ stored: list[tuple[str, int]] = []for recipient, delivery in sorted(deliveries, key=lambda decided: decided[0].agent.id):@@ -1366,7 +1367,7 @@ class MessageRouter:)continueif seq is not None:- stored.append(agent_id)+ stored.append((agent_id, seq))return storedasync def _step_group(diff --git a/core/tests/switch_core/bridges/agent/protocol/test_agent_event_listener.py b/core/tests/switch_core/bridges/agent/protocol/test_agent_event_listener.pynew file mode 100644index 000000000..fabb52e40--- /dev/null+++ b/core/tests/switch_core/bridges/agent/protocol/test_agent_event_listener.py@@ -0,0 +1,41 @@+"""What the agent event listener does with a notification: ring the agent's+doorbell with the new event's number, which the controller stream takes as+the agent's head instead of reading it."""++from __future__ import annotations++from switch_core.bridges.agent.protocol.agent_event_listener import (+ AgentEventListener,+)+++class _Doorbells:+ def __init__(self) -> None:+ self.rung: list[tuple[str, int | None]] = []++ def ring(self, agent_id: str, seq: int | None = None) -> None:+ self.rung.append((agent_id, seq))+++def _heard(payload: str) -> list[tuple[str, int | None]]:+ doorbells = _Doorbells()+ listener = AgentEventListener(lambda: None, doorbells) # type: ignore[arg-type,return-value]+ listener._on_notify(None, 0, "switch_agent_event", payload)+ return doorbells.rung+++def test_a_notification_rings_with_the_events_number() -> None:+ assert _heard('{"agent_id": "a1", "seq": 42}') == [("a1", 42)]+++def test_a_notification_without_a_usable_number_rings_without_one() -> None:+ """The stream then reads the agent's head, as it did before notifications+ carried it."""+ assert _heard('{"agent_id": "a1"}') == [("a1", None)]+ assert _heard('{"agent_id": "a1", "seq": "42"}') == [("a1", None)]+ assert _heard('{"agent_id": "a1", "seq": true}') == [("a1", None)]+++def test_an_unreadable_notification_rings_nothing() -> None:+ assert _heard("not json") == []+ assert _heard('["a1", 42]') == []diff --git a/core/tests/switch_core/delivery/test_router_sessions.py b/core/tests/switch_core/delivery/test_router_sessions.pyindex a277fed2d..073fa4735 100644--- a/core/tests/switch_core/delivery/test_router_sessions.py+++ b/core/tests/switch_core/delivery/test_router_sessions.py@@ -207,3 +207,22 @@ class TestWritingAMessage:assert (scope.taken, scope.peak) == (1, 1)assert len(await own_events(session_factory, agents["a3"][0].id)) == 1+++class TestTheWritersOwnProcess:+ async def test_an_addressed_agent_is_told_its_new_head_here(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """The process that stores an agent's own event knows its number, so+ a stream of the agent's in this process takes it as the agent's head+ rather than reading it (`EventBuffer.ring`)."""+ process, room, louisa, agents = await _seed(session_factory, names=["member"])+ agent = agents["member"][0]+ process.buffer.listening(True)+ process.buffer.doorbell(agent.id)++ await process.say(room, "@member can you look at this", sender=louisa)+ await process.idle()++ [event] = await own_events(session_factory, agent.id)+ assert process.buffer._told_heads == {agent.id: event.seq}diff --git a/core/tests/switch_core/management/test_stream.py b/core/tests/switch_core/management/test_stream.pyindex 495a1df63..6f3d2716c 100644--- a/core/tests/switch_core/management/test_stream.py+++ b/core/tests/switch_core/management/test_stream.py@@ -10,6 +10,7 @@ from __future__ import annotationsimport asyncioimport contextlibimport time+from typing import Anyimport pytestfrom sqlalchemy.exc import TimeoutError as PoolTimeoutError@@ -1018,3 +1019,211 @@ async def test_an_attach_that_fails_part_way_is_done_again_and_told_once("agent.attached",]assert name == "agent.event"+++async def test_a_delivery_told_its_head_reads_only_the_agents_events(+ harness: Harness, monkeypatch: pytest.MonkeyPatch+) -> None:+ """While the notification listener listens, an event rings its agent's+ doorbell with its number, and the pass it wakes takes the agent's head+ from that rather than reading the heads: the delivery costs the agent's+ pass alone."""+ monkeypatch.setattr(controller_routes, "IDLE_INTERVAL_SECONDS", 60.0)+ buffer = harness.protocol.event_buffer+ buffer.listening(True)+ owner = await add_member(harness.session_factory, "ada")+ async with harness.client() as client:+ controller = await enroll_console(harness, client, owner)+ agent_id = await place_agent(client, controller, name="reviewer")+ room_id = await add_room(harness.session_factory, agent_id)+ opened = await open_connection(client, controller)+ stream = await open_stream(harness, controller, opened)+ await take(stream, 2)+ # The stream's first pass reads every head; it then waits for a doorbell.+ delivered = asyncio.ensure_future(take(stream, 1))+ await asyncio.sleep(0.2)+ await _ask(harness, agent_id, room_id, "@you")+ counting = CountingSessionFactory(buffer._session_factory)+ monkeypatch.setattr(buffer, "_session_factory", counting)++ [(name, data)] = await delivered+ await stream.aclose()++ assert name == "agent.event"+ assert data["event"]["payload"]["body"] == "@you"+ assert counting.opened == 1+++async def test_an_event_never_told_is_found_after_the_idle_interval(+ harness: Harness, monkeypatch: pytest.MonkeyPatch+) -> None:+ """A notification can be lost without its connection dropping. The pass+ after each idle interval reads the heads rather than taking the ones it+ was told, so the event is sent then."""+ monkeypatch.setattr(controller_routes, "IDLE_INTERVAL_SECONDS", 0.05)+ buffer = harness.protocol.event_buffer+ buffer.listening(True)+ owner = await add_member(harness.session_factory, "ada")+ async with harness.client() as client:+ controller = await enroll_console(harness, client, owner)+ agent_id = await place_agent(client, controller, name="reviewer")+ room_id = await add_room(harness.session_factory, agent_id)+ opened = await open_connection(client, controller)+ stream = await open_stream(harness, controller, opened)+ await take(stream, 2)+ delivered = asyncio.ensure_future(take(stream, 1, timeout=5))+ await asyncio.sleep(0.2)+ # Stored the way `enqueue` stores it, without ringing anything.+ row = await say(harness.session_factory, room_id, "@you", sender="@ada:test")+ await buffer._store_own_event(+ TENANT_ZERO_ID,+ agent_id,+ room_id,+ _message(room_id, "@you", message_id=row.transport_event_id),+ row.seq,+ row.transport_event_id,+ )++ [(name, data)] = await delivered+ await stream.aclose()++ assert name == "agent.event"+ assert data["event"]["payload"]["body"] == "@you"+++class TestToldHeads:+ """What `EventBuffer.heads` takes from the doorbells instead of reading."""++ @staticmethod+ async def _agent(harness: Harness) -> str:+ owner = await add_member(harness.session_factory, "ada")+ async with harness.client() as client:+ controller = await enroll_console(harness, client, owner)+ agent_id = await place_agent(client, controller, name="reviewer")+ harness.protocol.event_buffer.doorbell(agent_id)+ return agent_id++ @staticmethod+ def _counting(harness: Harness, monkeypatch: pytest.MonkeyPatch) -> Any:+ buffer = harness.protocol.event_buffer+ counting = CountingSessionFactory(buffer._session_factory)+ monkeypatch.setattr(buffer, "_session_factory", counting)+ return counting++ async def test_a_head_told_while_listening_is_not_read(+ self, harness: Harness, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ agent_id = await self._agent(harness)+ buffer = harness.protocol.event_buffer+ buffer.listening(True)+ buffer.ring(agent_id, 41)+ counting = self._counting(harness, monkeypatch)++ assert await buffer.heads(TENANT_ZERO_ID, {agent_id}) == {agent_id: 41}+ assert counting.opened == 0++ async def test_a_lower_number_never_lowers_a_head(+ self, harness: Harness, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ """Two writers' notifications can arrive in either order."""+ agent_id = await self._agent(harness)+ buffer = harness.protocol.event_buffer+ buffer.listening(True)+ buffer.ring(agent_id, 41)+ buffer.ring(agent_id, 40)++ assert await buffer.heads(TENANT_ZERO_ID, {agent_id}) == {agent_id: 41}++ async def test_no_head_is_kept_unless_the_listener_listens(+ self, harness: Harness, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ agent_id = await self._agent(harness)+ buffer = harness.protocol.event_buffer+ buffer.ring(agent_id, 41)+ counting = self._counting(harness, monkeypatch)++ await buffer.heads(TENANT_ZERO_ID, {agent_id})+ await buffer.heads(TENANT_ZERO_ID, {agent_id})++ assert counting.opened == 2++ async def test_a_ring_without_a_number_forgets_the_head(+ self, harness: Harness, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ agent_id = await self._agent(harness)+ buffer = harness.protocol.event_buffer+ buffer.listening(True)+ buffer.ring(agent_id, 41)+ buffer.ring(agent_id)+ counting = self._counting(harness, monkeypatch)++ await buffer.heads(TENANT_ZERO_ID, {agent_id})++ assert counting.opened == 1++ async def test_a_reconnect_forgets_every_head_told(+ self, harness: Harness, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ """Notifications sent while the listener was not listening are lost."""+ agent_id = await self._agent(harness)+ buffer = harness.protocol.event_buffer+ buffer.listening(True)+ buffer.ring(agent_id, 41)+ buffer.listening(False)+ buffer.listening(True)+ counting = self._counting(harness, monkeypatch)++ await buffer.heads(TENANT_ZERO_ID, {agent_id})++ assert counting.opened == 1++ async def test_a_head_read_is_kept_and_not_read_again(+ self, harness: Harness, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ """Any event numbered after the read rings with its number."""+ agent_id = await self._agent(harness)+ buffer = harness.protocol.event_buffer+ buffer.listening(True)+ counting = self._counting(harness, monkeypatch)++ first = await buffer.heads(TENANT_ZERO_ID, {agent_id})+ second = await buffer.heads(TENANT_ZERO_ID, {agent_id})++ assert first == second+ assert counting.opened == 1++ async def test_a_head_read_across_a_reconnect_is_not_kept(+ self, harness: Harness, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ """The read may have seen the agent's events before ones whose+ notifications the reconnect lost."""+ agent_id = await self._agent(harness)+ buffer = harness.protocol.event_buffer+ buffer.listening(True)+ inner = buffer._session_factory++ def reconnecting() -> Any:+ buffer.listening(True)+ return inner()++ monkeypatch.setattr(buffer, "_session_factory", reconnecting)+ await buffer.heads(TENANT_ZERO_ID, {agent_id})+ counting = CountingSessionFactory(inner)+ monkeypatch.setattr(buffer, "_session_factory", counting)++ await buffer.heads(TENANT_ZERO_ID, {agent_id})++ assert counting.opened == 1++ async def test_a_fresh_read_reads_the_heads_told(+ self, harness: Harness, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ agent_id = await self._agent(harness)+ buffer = harness.protocol.event_buffer+ buffer.listening(True)+ buffer.ring(agent_id, 41)+ counting = self._counting(harness, monkeypatch)++ await buffer.heads(TENANT_ZERO_ID, {agent_id}, fresh=True)++ assert counting.opened == 1