Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Put the π on first and stop clock redraws starving new turns on Slack (CHOO-3947) (#1161)
Christian McDermott78ed6ade825a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c62bb23d0..50877317d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -209,6 +209,19 @@ version of their own to them without also giving them a release of their own.Existing duplicates are left as they are and can be reached by id.#### Fixed+- **On Slack, the π and the activity card no longer arrive after the agent's+ reply.** Every running turn's card was redrawn every 5 seconds to tick its+ clock. Slack closes a streamed card after 10 minutes, so on a longer turn+ each tick was a full message edit, and Slack allows about 50 of those a+ minute for the whole workspace. A few long turns used that up, Slack made the+ bridge wait, and new turns' cards and π waited behind the clock redraws. The+ clock now moves once a minute, and a running card shows whole minutes+ ("Workingβ¦ 3m"). Clock redraws stop while Slack is making the bridge wait,+ and new turns go ahead of them. The π goes on as soon as a turn starts+ rather than once its card has posted. A card is redrawn as soon as Slack+ closes its stream, instead of showing still-running steps as failed until+ the next change. A turn whose first draw into its stream fails keeps that+ card rather than posting a second one under it.- **An agent is told why a machine's CLI cannot be used.** When`create_agent` or `update_agent_detail` was refused because a provider wasnot logged in, it said the owner needed to log in, even when the machine haddiff --git a/core/switch_core/bridges/collaboration/adapter.py b/core/switch_core/bridges/collaboration/adapter.pyindex b1672b339..6a59269e3 100644--- a/core/switch_core/bridges/collaboration/adapter.py+++ b/core/switch_core/bridges/collaboration/adapter.py@@ -312,6 +312,37 @@ class RichContentThrottled(RichContentFailed):self.retry_after = retry_after+class RichContentFailedAfterPost(RichContentFailed):+ """The message was posted, but what goes in it could not be drawn.++ Slack opens a turn's stream and then takes its content in a separate call,+ and the second can fail after the first has made the message. `ref` names+ that message, so the caller keeps it and draws into it later rather than+ posting again, which would leave a second message in the thread and the+ first one empty.+ """++ def __init__(self, message: str, *, ref: str, text: str) -> None:+ super().__init__(message, text=text)+ self.ref = ref+++class RichContentThrottledAfterPost(RichContentThrottled, RichContentFailedAfterPost):+ """The message was posted, and the platform asked for a wait before what+ goes in it could be drawn.++ A `RichContentThrottled` as well, so the wait is the one asked for.+ """++ def __init__(self, *, ref: str, retry_after: float, text: str) -> None:+ # Not through `RichContentThrottled.__init__`: its `super()` call+ # reaches `RichContentFailedAfterPost` here, which needs the ref.+ RichContentFailedAfterPost.__init__(+ self, "Platform updates are rate limited.", ref=ref, text=text+ )+ self.retry_after = retry_after++class RichContentWedged(RichContentFailed):"""The platform will not change this message, and never will again.@@ -1393,6 +1424,26 @@ class PlatformAdapter(ABC):`supports_queue_reaction` is never asked for the queued one."""+ def activity_closes_in(self, message_ref: str) -> float | None:+ """Seconds until the platform closes this message to further streaming+ on its own, or None where it never will.++ Closing it rewrites what it shows (Slack marks the steps still running+ as failed), so a redraw is owed just after, whatever the clock says.+ Asked once, as the message is posted.+ """+ return None++ def rich_updates_held_for(self) -> float:+ """Seconds before this adapter will try another edit to rich content.++ Nonzero while it is holding edits back after a rate limit. Asked before+ work that can wait, such as redrawing a running turn's clock, so that+ work is not queued only to be refused. Only an adapter that+ `redraws_for_elapsed_time` is asked.+ """+ return 0.0+async def notify_working(self, channel_id: str, agent_name: str, thread_root_id: str | None) -> None:diff --git a/core/switch_core/bridges/collaboration/session/renderers/__init__.py b/core/switch_core/bridges/collaboration/session/renderers/__init__.pyindex 462bec54c..ba46aa9cb 100644--- a/core/switch_core/bridges/collaboration/session/renderers/__init__.py+++ b/core/switch_core/bridges/collaboration/session/renderers/__init__.py@@ -26,6 +26,11 @@ from switch_core.sessions.contract import (TurnUpsert,)+# How finely a running turn's clock is shown, and so how often the clock alone+# is worth redrawing a message for. Seconds would sit still between those+# redraws and read as a card that had stopped.+RUNNING_CLOCK_SECONDS = 60+# Where the turn itself has got to. One message per turn is edited in place, so# without this a turn that finished and a turn that stalled read identically:# the same tool log, the last line still marked in progress, and nothing to say@@ -61,7 +66,9 @@ def turn_state(`elapsed_seconds` comes from outside: neither a turn nor an item carriesa timestamp, so a caller that tracked one against the session's own event- log supplies it. A running turn shows the live duration. A completed turn shows it in+ log supplies it. A running turn shows the whole minutes it has run, and+ nothing until the first, because that is as often as anything redraws it+ for the clock alone. A completed turn shows the exact duration inplace of the plain "Turn complete." β that phrase said only that the turnwas over, and this says what happened while it ran. An interrupted orerrored turn keeps its own phrase, since that is still worth knowing on@@ -76,8 +83,8 @@ def turn_state("""state = TURN_STATE[turn.status]if turn.status not in TURN_ENDED:- if elapsed_seconds is not None:- return f"{state} {_format_duration(elapsed_seconds)}"+ if elapsed_seconds is not None and elapsed_seconds >= RUNNING_CLOCK_SECONDS:+ return f"{state} {int(elapsed_seconds // RUNNING_CLOCK_SECONDS)}m"return stateif elapsed_seconds is not None:worked = _worked_for(elapsed_seconds, items, tool_detail=tool_detail)diff --git a/core/switch_core/bridges/collaboration/slack/adapter.py b/core/switch_core/bridges/collaboration/slack/adapter.pyindex 82f388c14..9e0457554 100644--- a/core/switch_core/bridges/collaboration/slack/adapter.py+++ b/core/switch_core/bridges/collaboration/slack/adapter.py@@ -4,6 +4,7 @@ import asyncioimport loggingimport mathimport re+import timeimport uuidfrom collections import OrderedDictfrom collections.abc import Awaitable, Callable, Sequence@@ -31,7 +32,9 @@ from switch_core.bridges.collaboration.adapter import (RequestCard,RichContent,RichContentFailed,+ RichContentFailedAfterPost,RichContentThrottled,+ RichContentThrottledAfterPost,RichContentWedged,TurnActivity,)@@ -316,6 +319,11 @@ _SLACK_FILE_DOMAINS = ("slack.com", "slack-gov.com")_MAX_OPEN_STREAMS = 100+# Slack's own cap on a stream: it closes ten minutes after `chat.startStream`,+# however recently it was appended to, and marks the steps still running as+# failed.+_STREAM_LIFETIME_SECONDS = 600.0+# Far above the open-stream bound on purpose. An entry here is what stops a# message nothing can change from being asked about again, so evicting one# resumes that asking for as long as the process lives. Only turns still being@@ -364,6 +372,16 @@ _STREAMING_CONFLICT = "streaming_state_conflict"_STRANDED_BLOCKS: dict[str, dict[str, Any]] = {slot: {} for slot in STREAM_SLOTS}+class _StreamClosed(RichContentFailed):+ """An append refused because Slack has stopped taking appends for good.++ Its own type so that `update_rich` can draw the turn as an ordinary edit+ in the same call, rather than leaving the message as Slack left it until+ the turn next changes. To every other caller it is the `RichContentFailed`+ it inherits from.+ """++class _StreamNotClosed(RichContentFailed):"""A turn drawn to the end on a stream Slack would not then stop.@@ -388,11 +406,14 @@ class _ActivityStream:cards that has not changed costs an append and risks nothing useful.`blocks` is the last thing written to each section, by `block_id`, so a- redraw sends only what moved.+ redraw sends only what moved. `opened_at` is when this process opened it,+ on the monotonic clock; None for one taken back after a restart, whose+ opening this process never saw."""channel_id: strts: str+ opened_at: float | Noneblocks: dict[str, dict[str, Any]] = field(default_factory=dict)@@ -995,8 +1016,16 @@ class SlackAdapter(PlatformAdapter):)stream = self._streams.get(message_ref)if stream is not None and isinstance(content, TurnActivity):- await self._extend_stream(stream, message_ref, content)- return+ try:+ await self._extend_stream(stream, message_ref, content)+ except _StreamClosed:+ # Slack closes every stream ten minutes after it opens, however+ # recently it was appended to, and marks the steps still+ # running as failed. Edited now, those read right again+ # straight away rather than at the turn's next change.+ pass+ else:+ returnif message_ref in self._unredrawable:if not self._unredrawable[message_ref]:self._unredrawable[message_ref] = True@@ -1081,7 +1110,10 @@ class SlackAdapter(PlatformAdapter):f"Cannot recover an activity stream: invalid message ref {message_ref}.")stream = _ActivityStream(- channel_id=channel_id, ts=ts, blocks=dict(_STRANDED_BLOCKS)+ channel_id=channel_id,+ ts=ts,+ opened_at=None,+ blocks=dict(_STRANDED_BLOCKS),)self._register_stream(message_ref, stream)try:@@ -1256,6 +1288,18 @@ class SlackAdapter(PlatformAdapter):return 30.0return max(1.0, delay) if math.isfinite(delay) else 30.0+ def rich_updates_held_for(self) -> float:+ return self._rich_update_cooldown.remaining()++ def activity_closes_in(self, message_ref: str) -> float | None:+ """Only a stream closes on its own; an ordinary post never does."""+ stream = self._streams.get(message_ref)+ if stream is None or stream.opened_at is None:+ return None+ return max(+ 0.0, _STREAM_LIFETIME_SECONDS - (time.monotonic() - stream.opened_at)+ )+def _streamable(self, content: RichContent, thread_root_id: str | None) -> bool:"""Whether this is a turn's activity, in a thread a stream can open in.@@ -1392,9 +1436,22 @@ class SlackAdapter(PlatformAdapter):)return Noneref = f"{channel_id}:{ts}"- stream = _ActivityStream(channel_id=channel_id, ts=str(ts))+ stream = _ActivityStream(+ channel_id=channel_id, ts=str(ts), opened_at=time.monotonic()+ )self._register_stream(ref, stream)- await self._extend_stream(stream, ref, content)+ # The message exists now. Reported as not posted, the turn would be+ # posted again on the retry, leaving this one empty above it.+ try:+ await self._extend_stream(stream, ref, content)+ except RichContentThrottled as throttled:+ raise RichContentThrottledAfterPost(+ ref=ref, retry_after=throttled.retry_after, text=throttled.text+ ) from throttled+ except RichContentFailed as failed:+ raise RichContentFailedAfterPost(+ str(failed), ref=ref, text=failed.text+ ) from failedreturn refdef _register_stream(self, message_ref: str, stream: _ActivityStream) -> None:@@ -1575,20 +1632,21 @@ class SlackAdapter(PlatformAdapter):) -> NoReturn:"""Turn a refused append into the failure the caller already handles.- A stream that has been stopped, or that this app no longer owns, is- gone rather than temporarily unwell: forget it so the publication is- redrawn as an ordinary post rather than appended to forever.+ A stream that has been stopped, timed out, or that this app no longer+ owns, is gone rather than temporarily unwell: forget it so the+ publication is redrawn as an ordinary post rather than appended to+ forever."""code = error.response.get("error")if code in _STREAM_CLOSED_ERRORS:self._streams.pop(message_ref, None)logger.warning("The activity stream %s is no longer accepting appends (%s); "- "later updates will be drawn as an ordinary message.",+ "the turn is drawn as an ordinary message from here.",message_ref,code,)- raise RichContentFailed(+ raise _StreamClosed(f"Slack closed the activity stream {message_ref}: {code}", text=text) from errorthrottled = self._throttled(error, text)diff --git a/core/switch_core/session_activity/bridge_turns.py b/core/switch_core/session_activity/bridge_turns.pyindex e4cf9139f..9defa6d44 100644--- a/core/switch_core/session_activity/bridge_turns.py+++ b/core/switch_core/session_activity/bridge_turns.py@@ -16,6 +16,7 @@ from __future__ import annotationsfrom dataclasses import dataclassfrom datetime import datetime+from switch_core.bridges.collaboration.session.renderers import RUNNING_CLOCK_SECONDSfrom switch_core.db.models import AgentSessionActivityItemfrom switch_core.db.stores.session_activity_store import TURN_ITEM_IDfrom switch_core.sessions.contract import TURN_ENDED, Item, TurnUpsert@@ -94,18 +95,21 @@ def status_state(elapsed_seconds: float | None,interrupt_turn_id: str | None,clock_redraws: bool,+ closed_by_platform: bool,) -> str:"""What the turn's message is showing, so an unchanged redraw can be skipped.The turn's state and its tools count. The clock counts only where the- platform redraws for it. What the agent is saying does not: prose is+ platform redraws for it, and then in the whole minutes a running turn+ shows. What the agent is saying does not: prose isrevised on every token, and it goes out with the next change to either,and with the edit that ends the turn. The turn a stop control namescounts, so a queued turn's message stops offering to stop a turn that- has ended.+ has ended. So does the platform having closed the message on its own,+ since closing it changed what it shows."""clock = (- str(int(elapsed_seconds))+ str(int(elapsed_seconds // RUNNING_CLOCK_SECONDS))if clock_redraws and elapsed_seconds is not Noneelse "")@@ -114,4 +118,5 @@ def status_state(for item in view.itemsif item.kind == "tool-activity")- return f"{view.turn.status}:{interrupt_turn_id or ''}:{clock}/{tools}"+ closed = ":closed" if closed_by_platform else ""+ return f"{view.turn.status}:{interrupt_turn_id or ''}:{clock}/{tools}{closed}"diff --git a/core/switch_core/session_activity/publisher.py b/core/switch_core/session_activity/publisher.pyindex 4bb6228d4..fa39b3cd6 100644--- a/core/switch_core/session_activity/publisher.py+++ b/core/switch_core/session_activity/publisher.py@@ -5,6 +5,12 @@ bridge's tenant to `_on_change`, which only queues a key. A task of thisbridge's own drains the queue and does the platform work, so a slow or ratelimited platform never holds up the listener's other subscribers.+The queue has three lanes, drained most urgent first. A change just announced+is somebody waiting: a new turn's card and its π, a step. Retries, a resync's+catch-up and an agent going offline or coming back are owed but older. A+running turn's clock is cosmetic, so it waits behind both, and one the+platform refuses is dropped rather than retried: the next minute draws it.+A key queued while the same key is still waiting is dropped, and each key ishandled by reading the rows as they are now. So a burst of steps in one turncosts one redraw rather than one per step, and a request answered before its@@ -23,9 +29,10 @@ platform that removes answered cards.**Turns.** Each turn is drawn in one message, from its rows, with the adapter'sown activity rendering: every step, the stop control while it runs, and howlong it took. The asking message carries a queued or working marker while the-turn waits or runs, and a platform that says a turn is stuck in a message of-its own gets one, cleared again once it no longer applies. All of it sits in-`turn_status_posts`.+turn waits or runs β put on before the message is posted, so a platform short+of room for the message does not hold it back β and a platform that says a+turn is stuck in a message of its own gets one, cleared again once it no+longer applies. All of it sits in `turn_status_posts`.Failures of a key are retried on a widening interval; a platform that asks fora wait gets exactly that wait.@@ -36,7 +43,7 @@ from __future__ import annotationsimport asyncioimport loggingimport secrets-from collections import OrderedDict+from collections import OrderedDict, dequefrom collections.abc import Callablefrom dataclasses import dataclass, field, replacefrom datetime import UTC, datetime, timedelta@@ -52,11 +59,13 @@ from switch_core.bridges.collaboration.adapter import (PlatformAdapter,RemovalFailed,RichContentFailed,+ RichContentFailedAfterPost,RichContentThrottled,RichContentWedged,ThreadUnavailable,TurnActivity,)+from switch_core.bridges.collaboration.session.renderers import RUNNING_CLOCK_SECONDSfrom switch_core.db.models import (Agent,AgentSessionActivityItem,@@ -116,9 +125,13 @@ _THREAD_WAIT_ATTEMPTS = 5# corrected by a later resync already, and redrawing everything on each start# would spend a platform's rate budget on messages nobody is looking at._RESYNC_WINDOW = timedelta(minutes=15)-# How often running turns are looked at again: for the clock, where the-# platform redraws for it, and for their agent going offline or coming back.+# How often running turns are looked at again: for their agent going offline+# or coming back, and, where the platform redraws for it, for a clock that has+# moved on a minute._TICK_SECONDS = 5.0+# How long after a platform closes a turn's message on its own the redraw that+# repairs it is due, so that the redraw finds it closed.+_AFTER_PLATFORM_CLOSE = timedelta(seconds=2)_RETRY_MIN_SECONDS = 5.0_RETRY_MAX_SECONDS = 600.0# How many turns' drawn state one bridge remembers. Forgetting one costs a@@ -137,6 +150,11 @@ _HOST_OFFLINE = "Host offline. Answers are unavailable until the session reconneMark = Literal["queued", "working"]+# The queue's lanes, most urgent first; see the module docstring.+_NOW = 0+_SOON = 1+_CLOCK = 2+@dataclass(frozen=True)class _Target:@@ -170,6 +188,11 @@ class _TurnDrawn:status: str | None = Nonefinal: bool = Falsewedged: bool = False+ # When a running turn's message is next due a redraw without a change of+ # its own: its clock showing another minute, or the platform closing it.+ clock_due: datetime | None = None+ # When the platform closes the turn's message on its own, if it does.+ platform_closes_at: datetime | None = None@dataclass@@ -228,12 +251,18 @@ class AgentSessionActivityPublisher:self._activity = AgentSessionActivityStore()self._card_posts = ApprovalRequestPostStore()self._turn_posts = TurnStatusPostStore()- self._queue: asyncio.Queue[_Key] = asyncio.Queue()- self._pending: set[_Key] = set()+ self._lanes: tuple[deque[_Key], ...] = (deque(), deque(), deque())+ self._pending: dict[_Key, int] = {}+ self._wake = asyncio.Event()self._thread_waits: dict[_Key, int] = {}self._retries = _Backoff()self._drawn: OrderedDict[_Key, _TurnDrawn] = OrderedDict()self._live_turns: dict[_Key, str] = {}+ # A turn's row before its message is posted, holding the marker+ # already on the asking message until the row can be stored. Only in+ # memory: a restart in that window forgets the marker, and the turn+ # takes it off only if it is still running when the publisher is back.+ self._unposted: dict[_Key, TurnStatusPost] = {}self._open_cards: dict[_Key, str] = {}self._online_seen: dict[str, bool] = {}self._unsure_marks: set[_Key] = set()@@ -260,8 +289,9 @@ class AgentSessionActivityPublisher:),]# The listener resyncs its subscribers when it connects, which may have- # happened before this bridge started.- self._enqueue(_RESYNC)+ # happened before this bridge started. The resync itself only reads and+ # queues, so it goes first and what it finds owed waits its turn.+ self._enqueue(_RESYNC, _NOW)async def stop(self) -> None:if self._unsubscribe is not None:@@ -286,36 +316,60 @@ class AgentSessionActivityPublisher:async def _on_change(self, change: Change) -> None:if change.kind.startswith("approval."):- self._enqueue(("approval", change.agent_id, change.session_id, change.key))+ self._enqueue(+ ("approval", change.agent_id, change.session_id, change.key), _NOW+ )elif change.kind == "activity":- self._enqueue(("turn", change.agent_id, change.session_id, change.key))+ self._enqueue(+ ("turn", change.agent_id, change.session_id, change.key), _NOW+ )async def _on_resync(self) -> None:- self._enqueue(_RESYNC)+ self._enqueue(_RESYNC, _NOW)- def _enqueue(self, key: _Key) -> None:- if key in self._pending:+ def _enqueue(self, key: _Key, lane: int) -> None:+ """Queue a key in a lane, or move it up to one if it waits in a lower one."""+ queued = self._pending.get(key)+ if queued is not None and queued <= lane:return- self._pending.add(key)- self._queue.put_nowait(key)+ self._pending[key] = lane+ self._lanes[lane].append(key)+ self._wake.set()- def _enqueue_later(self, key: _Key, delay: float) -> None:- asyncio.get_running_loop().call_later(delay, self._enqueue, key)+ def _enqueue_later(self, key: _Key, delay: float, lane: int) -> None:+ asyncio.get_running_loop().call_later(delay, self._enqueue, key, lane)def _retry_later(self, key: _Key) -> None:- self._enqueue_later(key, self._retries.next(key))+ self._enqueue_later(key, self._retries.next(key), _SOON)++ async def _next(self) -> tuple[_Key, int]:+ """The first key in the most urgent lane holding one, and that lane.++ A key moved up a lane leaves its old entry behind, which is passed over+ here because the key is no longer pending in that lane.+ """+ while True:+ for lane, queue in enumerate(self._lanes):+ while queue:+ key = queue.popleft()+ if self._pending.get(key) == lane:+ del self._pending[key]+ return key, lane+ self._wake.clear()+ await self._wake.wait()# ββ The bridge's own tasks ββββββββββββββββββββββββββββββββββββββββββββββββasync def _run(self) -> None:while True:- key = await self._queue.get()- self._pending.discard(key)+ key, lane = await self._next()try:await self._handle(key)except asyncio.CancelledError:raiseexcept RichContentThrottled as throttled:+ if lane == _CLOCK:+ continuelogger.warning("%s asked bridge %s to wait %.1fs; retrying %s then.",self._adapter.platform_name,@@ -323,9 +377,9 @@ class AgentSessionActivityPublisher:throttled.retry_after,key,)- self._enqueue_later(key, throttled.retry_after)+ self._enqueue_later(key, throttled.retry_after, _SOON)except _ThreadNotMappedYet:- self._enqueue_later(key, _THREAD_WAIT_SECONDS)+ self._enqueue_later(key, _THREAD_WAIT_SECONDS, lane)except Exception:# The delivery loop: one key failing must not stop the rest.delay = self._retries.next(key)@@ -335,18 +389,41 @@ class AgentSessionActivityPublisher:self._bridge_id,delay,)- self._enqueue_later(key, delay)+ self._enqueue_later(key, delay, _SOON)async def _tick_forever(self) -> None:while True:await asyncio.sleep(_TICK_SECONDS)- clock = self._adapter.redraws_for_elapsed_time- for keys in (self._live_turns, self._open_cards):- for key, agent_id in list(keys.items()):- online = self._agent_online(agent_id)- moved = self._online_seen.get(agent_id) != online- if moved or (clock and online and keys is self._live_turns):- self._enqueue(key)+ self._tick(datetime.now(UTC))++ def _tick(self, now: datetime) -> None:+ """Queue what moved without a change of its own: an agent going offline+ or coming back, and a running turn's clock once it shows another minute.++ No clock while the platform is holding edits back. Each redraw would+ only be refused, and the first tick after the wait draws whichever+ clocks are due by then.+ """+ clock = (+ self._adapter.redraws_for_elapsed_time+ and self._adapter.rich_updates_held_for() <= 0+ )+ for keys in (self._live_turns, self._open_cards):+ for key, agent_id in list(keys.items()):+ online = self._agent_online(agent_id)+ if self._online_seen.get(agent_id) != online:+ self._enqueue(key, _SOON)+ elif (+ clock+ and online+ and keys is self._live_turns+ and self._clock_due(key, now)+ ):+ self._enqueue(key, _CLOCK)++ def _clock_due(self, key: _Key, now: datetime) -> bool:+ drawn = self._drawn.get(key)+ return drawn is None or drawn.clock_due is None or drawn.clock_due <= nowasync def _handle(self, key: _Key) -> None:kind = key[0]@@ -425,14 +502,14 @@ class AgentSessionActivityPublisher:).all()marked = await self._turn_posts.marked(db, self._bridge_id)for agent_id, session_id, request_id in requests:- self._enqueue(("approval", agent_id, session_id, request_id))+ self._enqueue(("approval", agent_id, session_id, request_id), _SOON)for agent_id, session_id, turn_id, changed, running, drawn_at in turns:# An ended turn whose message was last drawn after its last change# already shows how it ended.if running or drawn_at is None or drawn_at < changed:- self._enqueue(("turn", agent_id, session_id, turn_id))+ self._enqueue(("turn", agent_id, session_id, turn_id), _SOON)for post in marked:- self._enqueue(("turn", post.agent_id, post.session_id, post.turn_id))+ self._enqueue(("turn", post.agent_id, post.session_id, post.turn_id), _SOON)# ββ Where a room's publications go ββββββββββββββββββββββββββββββββββββββββ@@ -816,7 +893,7 @@ class AgentSessionActivityPublisher:post.external_channel_id,delay,)- self._enqueue_later(key, delay)+ self._enqueue_later(key, delay, _SOON)return Falseasync with tenant_session(self._sessions, self._tenant_id) as db, db.begin():stored = await self._card_posts.get(@@ -933,7 +1010,7 @@ class AgentSessionActivityPublisher:self._adapter.platform_name,delay,)- self._enqueue_later(key, delay)+ self._enqueue_later(key, delay, _SOON)return Noneasync with tenant_session(self._sessions, self._tenant_id) as db, db.begin():stored = await self._card_posts.get(@@ -995,11 +1072,12 @@ class AgentSessionActivityPublisher:await self._activity.turn(db, agent_id, session_id, turn_id))if view is None or view.row.room_id is None:+ self._forget_turn(key)returnroom_id = view.row.room_idtarget = await self._target(db, key, agent_id, room_id, view.row.thread_id)if target is None:- self._live_turns.pop(key, None)+ self._forget_turn(key)returnasked_on = (await self._platform_ref(db, target.channel_id, view.row.message_id)@@ -1043,19 +1121,32 @@ class AgentSessionActivityPublisher:elapsed_seconds=elapsed,interrupt_turn_id=interrupt_turn_id,clock_redraws=self._adapter.redraws_for_elapsed_time,+ closed_by_platform=drawn.platform_closes_at is not None+ and drawn.platform_closes_at <= datetime.now(UTC),)if post is None:- post = await self._begin_turn(key, view, target, asked_on, content)- if post is not None:- drawn.status_state = state- drawn.final = view.ended+ unposted = self._unposted.get(key)+ # Rebuilt until it holds a marker, so the marker goes where the+ # asking message is known to be now: its post can be mapped+ # between one attempt and the next.+ if unposted is None or unposted.mark is None:+ unposted = self._unposted[key] = self._unposted_turn(+ view, target, asked_on+ )+ await self._marks(key, unposted, view, target, waiting_behind=bool(unended))+ post = await self._begin_turn(+ key, view, target, unposted, content, state, drawn+ )+ if post is not None or (view.ended and unposted.mark is None):+ self._unposted.pop(key, None)else:await self._redraw_turn(key, post, view, target, content, state, drawn)- if post is not None:await self._marks(key, post, view, target, waiting_behind=bool(unended))+ if post is not None:await self._attention(post, view, target, error_summary, recipient, drawn)if post is not None and not view.ended:self._live_turns[key] = agent_id+ drawn.clock_due = self._redraw_due(drawn, elapsed)else:self._live_turns.pop(key, None)if post is not None and drawn.final and not was_final:@@ -1065,7 +1156,39 @@ class AgentSessionActivityPublisher:# The other waiting turns' stop controls name this session's# running turn, which may just have changed.for other in unended:- self._enqueue(("turn", agent_id, session_id, other))+ self._enqueue(("turn", agent_id, session_id, other), _SOON)++ def _redraw_due(self, drawn: _TurnDrawn, elapsed: float | None) -> datetime:+ """When a running turn is next due a redraw with no change of its own."""+ now = datetime.now(UTC)+ due = now + timedelta(seconds=_until_the_clock_moves(elapsed))+ closes_at = drawn.platform_closes_at+ if closes_at is not None and now < closes_at + _AFTER_PLATFORM_CLOSE < due:+ due = closes_at + _AFTER_PLATFORM_CLOSE+ return due++ def _forget_turn(self, key: _Key) -> None:+ """Stop watching a turn that is gone, or no longer published here."""+ self._live_turns.pop(key, None)+ self._drop_unposted(key)++ def _drop_unposted(self, key: _Key) -> None:+ """Forget a turn's unposted row, saying so if it leaves a marker behind.++ Nothing is asked of the platform: a turn that stops being published+ here keeps its marker whether or not its message was posted.+ """+ unposted = self._unposted.pop(key, None)+ if unposted is not None and unposted.mark is not None:+ logger.warning(+ "No longer publishing turn %s of session %s, whose message was "+ "never posted; the %s marker on %s in channel %s stays there.",+ unposted.turn_id,+ unposted.session_id,+ unposted.mark,+ unposted.reaction_message_ref,+ unposted.external_channel_id,+ )async def _stamp_drawn(self, post: TurnStatusPost) -> None:"""Record that the turn's message shows how it ended, for the next resync."""@@ -1101,16 +1224,40 @@ class AgentSessionActivityPublisher:]return running[-1] if running else None+ def _unposted_turn(+ self, view: TurnView, target: _Target, asked_on: str | None+ ) -> TurnStatusPost:+ """The turn's row before its message exists, to carry its marker until then."""+ return TurnStatusPost(+ tenant_id=self._tenant_id,+ bridge_id=self._bridge_id,+ agent_id=view.row.agent_id,+ session_id=view.row.session_id,+ turn_id=view.turn.turn_id,+ external_channel_id=target.channel_id,+ thread_ref=target.thread_ref,+ reaction_message_ref=asked_on,+ mark=None,+ attention_post_id=None,+ )+async def _begin_turn(self,key: _Key,view: TurnView,target: _Target,- asked_on: str | None,+ post: TurnStatusPost,content: TurnActivity,+ state: str,+ drawn: _TurnDrawn,) -> TurnStatusPost | None:"""Post the turn in its thread, or at the channel root with none to thread under.+ `post` is the turn's row as it stood before the message, marker and+ all, and is stored once the platform has the message. One the platform+ made but could not draw into is kept and drawn into later, rather than+ posted a second time: after the wait, where it asked for one.+Once per turn, which is why the "agent has started" nudge belongs here.A refusal is logged; the next change to the turn tries again, and anended turn is tried again after a wait.@@ -1121,12 +1268,42 @@ class AgentSessionActivityPublisher:target.agent_name,# Asked at the channel root, the turn threads under what was# said, and whoever is waiting is watching the root.- None if asked_on == target.thread_ref else target.thread_ref,+ None+ if post.reaction_message_ref == target.thread_ref+ else target.thread_ref,)+ drew = Truetry:ref = await self._adapter.post_rich(target.channel_id, target.agent_name, content, target.thread_ref)+ except RichContentFailedAfterPost as failed:+ ref = failed.ref+ drew = False+ if isinstance(failed, RichContentThrottled):+ logger.warning(+ "%s made the message for turn %s in channel %s but asked for "+ "a %.1fs wait before drawing the turn into it; drawing it then.",+ self._adapter.platform_name,+ view.turn.turn_id,+ target.channel_id,+ failed.retry_after,+ )+ self._enqueue_later(key, failed.retry_after, _SOON)+ else:+ logger.error(+ "%s made the message for turn %s in channel %s but could not "+ "draw the turn into it: %s. %s",+ self._adapter.platform_name,+ view.turn.turn_id,+ target.channel_id,+ failed,+ "The turn has ended; trying again after a wait."+ if view.ended+ else "The next change to the turn draws into the same message.",+ )+ if view.ended:+ self._retry_later(key)except RichContentThrottled:raiseexcept RichContentFailed as error:@@ -1142,23 +1319,20 @@ class AgentSessionActivityPublisher:if view.ended:self._retry_later(key)return None- post = TurnStatusPost(- tenant_id=self._tenant_id,- bridge_id=self._bridge_id,- agent_id=view.row.agent_id,- session_id=view.row.session_id,- turn_id=view.turn.turn_id,- external_channel_id=target.channel_id,- external_post_id=ref,- thread_ref=target.thread_ref,- reaction_message_ref=asked_on,- mark=None,- attention_post_id=None,+ closes_in = self._adapter.activity_closes_in(ref)+ drawn.platform_closes_at = (+ None+ if closes_in is None+ else datetime.now(UTC) + timedelta(seconds=closes_in))+ post.external_post_id = refasync with tenant_session(self._sessions, self._tenant_id) as db, db.begin():await self._turn_posts.create(db, post)db.expunge(post)- self._retries.clear(key)+ if drew:+ self._retries.clear(key)+ drawn.status_state = state+ drawn.final = view.endedreturn postasync def _redraw_turn(@@ -1334,6 +1508,7 @@ class AgentSessionActivityPublisher:mark,same_agent=self._adapter.activity_reactions_per_agent,)+ holders += self._unposted_holders(post, mark)if not holders:try:await self._adapter.mark_activity(@@ -1358,6 +1533,21 @@ class AgentSessionActivityPublisher:await self._record_mark(post, None)return True+ def _unposted_holders(self, post: TurnStatusPost, mark: str) -> int:+ """Other turns holding `mark` on the same asking message whose own+ message is not posted yet. Their rows are not stored, so the database+ cannot count them."""+ per_agent = self._adapter.activity_reactions_per_agent+ return sum(+ 1+ for other in self._unposted.values()+ if other is not post+ and other.mark == mark+ and other.external_channel_id == post.external_channel_id+ and other.reaction_message_ref == post.reaction_message_ref+ and (not per_agent or other.agent_id == post.agent_id)+ )+async def _record_mark(self, post: TurnStatusPost, mark: Mark | None) -> None:async with tenant_session(self._sessions, self._tenant_id) as db, db.begin():stored = await self._turn_posts.get(@@ -1509,6 +1699,16 @@ class AgentSessionActivityPublisher:return view, await self._running_turn(db, post.agent_id, post.session_id)+def _until_the_clock_moves(elapsed: float | None) -> float:+ """Seconds until a running turn's clock shows another minute.++ A queued turn shows no clock, and is looked at again after the same while.+ """+ if elapsed is None:+ return RUNNING_CLOCK_SECONDS+ return RUNNING_CLOCK_SECONDS - elapsed % RUNNING_CLOCK_SECONDS++def _asked_at_root(turn_row: AgentSessionActivityItem | None) -> bool:"""Whether the turn a request belongs to was asked at the channel root.diff --git a/core/tests/switch_core/bridges/collaboration/golden/slack/activity_plan.json b/core/tests/switch_core/bridges/collaboration/golden/slack/activity_plan.jsonindex 725cd9d63..d27691d14 100644--- a/core/tests/switch_core/bridges/collaboration/golden/slack/activity_plan.json+++ b/core/tests/switch_core/bridges/collaboration/golden/slack/activity_plan.json@@ -29,7 +29,7 @@"title": "Run pytest"}],- "title": "Workingβ¦ 5s Β· Running: Run pytest",+ "title": "Workingβ¦ Β· Running: Run pytest","type": "plan"},{@@ -57,7 +57,7 @@},"event_type": "switch_publication"},- "text": "Workingβ¦ 5s Β· Running: Run pytest",+ "text": "Workingβ¦ Β· Running: Run pytest","thread_ts": null,"unfurl_links": false,"unfurl_media": false,diff --git a/core/tests/switch_core/bridges/collaboration/golden/slack/activity_stream.json b/core/tests/switch_core/bridges/collaboration/golden/slack/activity_stream.jsonindex fb95c9db0..baf72981b 100644--- a/core/tests/switch_core/bridges/collaboration/golden/slack/activity_stream.json+++ b/core/tests/switch_core/bridges/collaboration/golden/slack/activity_stream.json@@ -35,7 +35,7 @@"title": "Read README.md"}],- "title": "Workingβ¦ 3s Β· Running: Read README.md",+ "title": "Workingβ¦ Β· Running: Read README.md","type": "plan"}],diff --git a/core/tests/switch_core/bridges/collaboration/golden/slack/status_card.json b/core/tests/switch_core/bridges/collaboration/golden/slack/status_card.jsonindex dd1de8b60..463efb366 100644--- a/core/tests/switch_core/bridges/collaboration/golden/slack/status_card.json+++ b/core/tests/switch_core/bridges/collaboration/golden/slack/status_card.json@@ -6,7 +6,7 @@"block_id": "switch-request:tok-status","status": "in_progress","task_id": "turn-1",- "title": "Workingβ¦ 12s",+ "title": "Workingβ¦","type": "task_card"},{@@ -27,7 +27,7 @@},"event_type": "switch_publication"},- "text": "<@U0OWNER> Workingβ¦ 12s",+ "text": "<@U0OWNER> Workingβ¦","thread_ts": null,"unfurl_links": false,"unfurl_media": false,diff --git a/core/tests/switch_core/bridges/collaboration/test_session_activity.py b/core/tests/switch_core/bridges/collaboration/test_session_activity.pyindex 62d7f3cb3..7a3c0c0f7 100644--- a/core/tests/switch_core/bridges/collaboration/test_session_activity.py+++ b/core/tests/switch_core/bridges/collaboration/test_session_activity.py@@ -385,11 +385,18 @@ async def test_an_interrupted_turn_keeps_its_own_phrase_and_says_how_long_too()assert state == "Turn interrupted. Worked for 45s."-async def test_a_running_turn_shows_live_elapsed_seconds() -> None:- """The publisher refreshes this duration while the turn is running."""- state = turn_state([], _turn("running"), tool_detail=True, elapsed_seconds=80)+async def test_a_running_turn_shows_the_whole_minutes_it_has_run() -> None:+ """The clock alone redraws a running turn once a minute, so seconds would+ sit still between redraws and read as a card that had stopped."""- assert state == "Workingβ¦ 1m 20s"+ def running(elapsed: float) -> str:+ return turn_state(+ [], _turn("running"), tool_detail=True, elapsed_seconds=elapsed+ )++ assert running(59) == "Workingβ¦"+ assert running(80) == "Workingβ¦ 1m"+ assert running(3725) == "Workingβ¦ 62m"async def test_with_no_elapsed_seconds_a_completed_turn_says_only_that() -> None:@@ -686,12 +693,12 @@ async def test_the_one_message_leads_with_the_state_and_the_step_it_is_on() -> Nitems = await _items()running = _item(itemId="live", status="in-progress", title="Read")- drawn = render_activity_plan([*items, running], _turn(), elapsed_seconds=40)+ drawn = render_activity_plan([*items, running], _turn(), elapsed_seconds=140)plan = drawn.blocks[0]assert plan["type"] == "plan"assert plan["title"].endswith("Β· Running: Read")- assert "40s" in plan["title"]+ assert "2m" in plan["title"]assert _turn_cards(plan)[-1]["title"] == "Read"diff --git a/core/tests/switch_core/bridges/collaboration/test_session_slack_streaming.py b/core/tests/switch_core/bridges/collaboration/test_session_slack_streaming.pyindex 8be4c9be1..eec9f75d1 100644--- a/core/tests/switch_core/bridges/collaboration/test_session_slack_streaming.py+++ b/core/tests/switch_core/bridges/collaboration/test_session_slack_streaming.py@@ -39,7 +39,9 @@ from slack_sdk.errors import SlackApiErrorfrom switch_core.bridges.collaboration.adapter import (RichContentFailed,+ RichContentFailedAfterPost,RichContentThrottled,+ RichContentThrottledAfterPost,RichContentWedged,TurnActivity,)@@ -204,7 +206,7 @@ async def test_only_the_sections_that_moved_are_appended() -> None:assert [c["type"] for c in _chunks(client)[0]] == ["blocks"]later = _chunks(client)[1]assert [c["type"] for c in later] == ["blocks"]- assert later[0]["blocks"][0]["title"] == "Workingβ¦ 9s Β· Running: Grep"+ assert later[0]["blocks"][0]["title"] == "Workingβ¦ Β· Running: Grep"assert [(t["title"], t["status"]) for t in later[0]["blocks"][0]["tasks"][1:]] == [("Read", "complete"),("Grep", "in_progress"),@@ -243,7 +245,7 @@ async def test_a_section_that_did_not_move_is_not_sent_again() -> None:later = [c for c in _chunks(client)[1] if c["type"] == "blocks"]assert len(later) == 1- assert later[0]["blocks"][0]["title"] == "Workingβ¦ 9s Β· Last: Tool 50"+ assert later[0]["blocks"][0]["title"] == "Workingβ¦ Β· Last: Tool 50"async def test_a_publish_that_changed_nothing_appends_nothing() -> None:@@ -271,16 +273,16 @@ async def test_the_clock_moves_the_live_section_and_nothing_above_it() -> None:many = [_tool(f"t{n}", f"Tool {n}", status="completed") for n in range(50)]ref = await adapter.post_rich(- CHANNEL, "Agent", TurnActivity(many, _turn(), 5.0), THREAD+ CHANNEL, "Agent", TurnActivity(many, _turn(), 65.0), THREAD)await adapter.update_rich(- CHANNEL, "Agent", ref, TurnActivity(many, _turn(), 10.0), THREAD+ CHANNEL, "Agent", ref, TurnActivity(many, _turn(), 125.0), THREAD)assert [c["blocks"][0]["block_id"] for c in _chunks(client)[1]] == ["switch-interrupt"]- assert _chunks(client)[1][0]["blocks"][0]["title"] == "Workingβ¦ 10s Β· Last: Tool 49"+ assert _chunks(client)[1][0]["blocks"][0]["title"] == "Workingβ¦ 2m Β· Last: Tool 49"async def test_a_detail_is_rich_text_because_that_is_what_a_card_takes() -> None:@@ -396,7 +398,7 @@ async def test_a_turn_of_any_length_draws_the_same_three_step_blocks() -> None:assert gone["elements"][0]["text"] == "_Activity 1β147 no longer shown_"assert older["title"] == "Activity 148β196"assert [task["title"] for task in older["tasks"][1:]][:1] == ["Tool 147"]- assert newer["title"] == "Workingβ¦ 4m 0s Β· Last: Tool 239"+ assert newer["title"] == "Workingβ¦ 4m Β· Last: Tool 239"assert [task["title"] for task in newer["tasks"][1:]][-1:] == ["Tool 239"]@@ -425,7 +427,7 @@ async def test_what_is_no_longer_shown_is_one_line_above_the_sections() -> None:assert [block["title"] for block in before] == ["Activity 1β49",- "Workingβ¦ 1s Β· Last: Tool 97",+ "Workingβ¦ Β· Last: Tool 97",]after = list(_drawn(client).values())assert [block["type"] for block in after] == ["context", "plan", "plan"]@@ -433,7 +435,7 @@ async def test_what_is_no_longer_shown_is_one_line_above_the_sections() -> None:assert after[0]["elements"][0]["text"] == "_Activity 1β49 no longer shown_"assert [block["title"] for block in after[1:]] == ["Activity 50β98",- "Workingβ¦ 9s Β· Last: Tool 98",+ "Workingβ¦ Β· Last: Tool 98",]@@ -455,7 +457,7 @@ async def test_a_step_never_moves_between_sections_once_it_has_landed() -> None:moved = [c for c in _chunks(client)[1] if c["type"] == "blocks"]assert [chunk["blocks"][0]["title"] for chunk in moved] == [- "Workingβ¦ 9s Β· Last: Tool 50"+ "Workingβ¦ Β· Last: Tool 50"]older, newer = _pages(client)assert [task["title"] for task in older["tasks"][1:]][:1] == ["Tool 0"]@@ -485,7 +487,7 @@ async def test_the_live_step_is_named_on_its_own_section_and_not_in_the_header()assert [page["title"] for page in _pages(client)] == ["Activity 1β49 Β· Running: Grep",- "Workingβ¦ 40s",+ "Workingβ¦",]@@ -512,12 +514,12 @@ async def test_the_label_leaves_a_settled_section_when_the_live_step_moves_past_first = [c for c in _chunks(client)[0] if c["type"] == "blocks"]assert [chunk["blocks"][0]["title"] for chunk in first] == [- "Workingβ¦ 1s Β· Last: Tool 48"+ "Workingβ¦ Β· Last: Tool 48"]later = [c for c in _chunks(client)[1] if c["type"] == "blocks"]assert [chunk["blocks"][0]["title"] for chunk in later] == ["Activity 1β49",- "Workingβ¦ 9s Β· Last: Tool 49",+ "Workingβ¦ Β· Last: Tool 49",]@@ -534,7 +536,7 @@ async def test_a_step_title_is_not_escaped_because_nothing_in_it_is_parsed() ->)assert _steps(client)[0]["title"] == shell- assert _pages(client)[0]["title"] == f"Workingβ¦ 1s Β· Running: {shell}"+ assert _pages(client)[0]["title"] == f"Workingβ¦ Β· Running: {shell}"# ββ Ending βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ@@ -858,7 +860,7 @@ async def test_an_attention_post_is_never_streamed() -> None:# ββ When an append is refused ββββββββββββββββββββββββββββββββββββββββββββββββ-async def test_a_closed_stream_is_forgotten_so_later_updates_are_edits(+async def test_a_closed_stream_is_forgotten_and_the_turn_edited_at_once(caplog: Any,) -> None:"""A stopped stream, or one this app no longer owns, is gone rather than@@ -872,18 +874,41 @@ async def test_a_closed_stream_is_forgotten_so_later_updates_are_edits()client.append_error = "stopped_by_user"- with pytest.raises(RichContentFailed), caplog.at_level(logging.WARNING):+ with caplog.at_level(logging.WARNING):await adapter.update_rich(- CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 9.0), THREAD+ CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 61.0), THREAD)- client.append_error = Noneawait adapter.update_rich(- CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 14.0), THREAD+ CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 121.0), THREAD)assert "stopped_by_user" in caplog.textassert adapter._streams == {}- assert [call["ts"] for call in client.updated] == ["1.0"]+ assert [call["ts"] for call in client.updated] == ["1.0", "1.0"]+ assert len(client.appended) == 1+++async def test_a_stream_slack_timed_out_has_its_turn_redrawn_in_the_same_call() -> None:+ """Slack closes every stream ten minutes after it opens, appended to or+ not, and marks whatever was still running as failed. Left for the turn's+ next change, a long tool call would sit there marked failed until it+ finished."""+ client = FakeWebClient()+ adapter = _adapter(client)+ tool = _tool("t1", "Run the tests")+ ref = await adapter.post_rich(+ CHANNEL, "Agent", TurnActivity([tool], _turn(), 1.0), THREAD+ )++ client.append_error = "message_not_in_streaming_state"+ await adapter.update_rich(+ CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 600.0), THREAD+ )++ assert ref not in adapter._streams+ [edit] = client.updated+ assert edit["ts"] == "1.0"+ assert "Workingβ¦ 10m" in edit["text"]async def test_a_turn_of_two_sections_is_left_as_the_stream_drew_it(@@ -937,11 +962,6 @@ async def test_a_turn_whose_last_append_was_refused_is_still_drawn_by_an_edit()CHANNEL, "Agent", TurnActivity(many, _turn(), 1.0), THREAD)client.append_error = "stopped_by_user"- with pytest.raises(RichContentFailed):- await adapter.update_rich(- CHANNEL, "Agent", ref, TurnActivity(many, _turn("completed"), 9.0), THREAD- )- client.append_error = Noneawait adapter.update_rich(CHANNEL, "Agent", ref, TurnActivity(many, _turn("completed"), 9.0), THREAD)@@ -1047,16 +1067,115 @@ async def test_a_throttled_append_asks_the_caller_to_wait_and_keeps_the_stream(monkeypatch.setattr(client, "chat_appendStream", refuse)with pytest.raises(RichContentThrottled) as refused:await adapter.update_rich(- CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 9.0), THREAD+ CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 61.0), THREAD)assert refused.value.retry_after == 7.0assert ref in adapter._streamsassert [block["title"] for block in adapter._streams[ref].blocks.values()] == [- "Workingβ¦ 1s Β· Running: Read"+ "Workingβ¦ Β· Running: Read"]+async def test_a_stream_opened_but_throttled_before_its_first_append_is_kept() -> None:+ """The message exists once the stream opens. Reported as never posted, the+ retry would open a second one under it and leave this one empty."""+ client = FakeWebClient()+ adapter = _adapter(client)+ tool = _tool("t1", "Read")++ client.append_error = FakeResponse(+ {"error": "ratelimited"}, headers={"Retry-After": "3"}+ )+ with pytest.raises(RichContentThrottledAfterPost) as throttled:+ await adapter.post_rich(+ CHANNEL, "Agent", TurnActivity([tool], _turn(), 1.0), THREAD+ )+ assert throttled.value.retry_after == 3.0+ ref = throttled.value.ref+ assert ref == f"{CHANNEL}:1.0"++ client.append_error = None+ adapter._rich_update_cooldown._until = 0.0+ await adapter.update_rich(+ CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 2.0), THREAD+ )++ assert len(client.started) == 1+ assert [call["ts"] for call in client.appended] == ["1.0"]+ assert _steps(client)[0]["title"] == "Read"+++async def test_a_stream_whose_first_append_fails_is_kept_rather_than_posted_again() -> (+ None+):+ """Any failure once the stream has made the message, not only a rate+ limit. One Slack may still take an append on stays a stream."""+ client = FakeWebClient()+ adapter = _adapter(client)+ tool = _tool("t1", "Read")++ client.append_error = "internal_error"+ with pytest.raises(RichContentFailedAfterPost) as failed:+ await adapter.post_rich(+ CHANNEL, "Agent", TurnActivity([tool], _turn(), 1.0), THREAD+ )+ assert not isinstance(failed.value, RichContentThrottled)+ ref = failed.value.ref+ assert ref in adapter._streams++ client.append_error = None+ await adapter.update_rich(+ CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 2.0), THREAD+ )+ assert len(client.started) == 1+ assert [call["ts"] for call in client.appended] == ["1.0"]+++async def test_a_stream_closed_before_its_first_append_is_kept_and_edited() -> None:+ client = FakeWebClient()+ adapter = _adapter(client)+ tool = _tool("t1", "Read")++ client.append_error = "message_not_owned_by_app"+ with pytest.raises(RichContentFailedAfterPost) as failed:+ await adapter.post_rich(+ CHANNEL, "Agent", TurnActivity([tool], _turn(), 1.0), THREAD+ )+ ref = failed.value.ref+ assert ref not in adapter._streams++ await adapter.update_rich(+ CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 2.0), THREAD+ )+ assert len(client.started) == 1+ assert [call["ts"] for call in client.updated] == ["1.0"]+++async def test_only_a_stream_is_said_to_close_on_its_own() -> None:+ """Slack closes a stream ten minutes after opening it. An ordinary post it+ never closes, and a redraw scheduled for one would be an edit wasted."""+ client = FakeWebClient()+ adapter = _adapter(client)+ tool = _tool("t1", "Read")+ streamed = await adapter.post_rich(+ CHANNEL, "Agent", TurnActivity([tool], _turn(), 1.0), THREAD+ )+ posted = await adapter.post_rich(+ CHANNEL, "Agent", TurnActivity([tool], _turn(), 1.0), None+ )++ closes = adapter.activity_closes_in(streamed)+ assert closes is not None and 590.0 < closes <= 600.0+ assert adapter.activity_closes_in(posted) is None++ client.append_error = "message_not_in_streaming_state"+ await adapter.update_rich(+ CHANNEL, "Agent", streamed, TurnActivity([tool], _turn(), 61.0), THREAD+ )+ assert adapter.activity_closes_in(streamed) is None++# ββ When a restart strands a stream ββββββββββββββββββββββββββββββ## The open-stream registry is held on the adapter and written nowhere else, so@@ -1335,7 +1454,7 @@ async def test_a_stranded_turn_still_running_carries_on_streaming() -> None:assert ref in adapter._streamsawait adapter.update_rich(- CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 14.0), THREAD+ CHANNEL, "Agent", ref, TurnActivity([tool], _turn(), 74.0), THREAD)assert len(client.update_attempts) == 1assert len(client.appended) == 3diff --git a/core/tests/switch_core/session_activity/bridge_fixtures.py b/core/tests/switch_core/session_activity/bridge_fixtures.pyindex 5b781bb12..a15715184 100644--- a/core/tests/switch_core/session_activity/bridge_fixtures.py+++ b/core/tests/switch_core/session_activity/bridge_fixtures.py@@ -11,7 +11,13 @@ from typing import Anyfrom sqlalchemy import insert-from switch_core.bridges.collaboration.adapter import RichContent, RichContentFailed+from switch_core.bridges.collaboration.adapter import (+ RichContent,+ RichContentFailed,+ RichContentFailedAfterPost,+ RichContentThrottledAfterPost,+ TurnActivity,+)from switch_core.db.models import Client, CollaborationBridge, Room, room_agents@@ -69,6 +75,8 @@ class RecordingPlatform:separate_attention_slot: bool = Truenotifies_only_by_mention: bool = Falseredraws_for_elapsed_time: bool = False+ # What `activity_closes_in` answers for every message posted.+ closes_in: float | None = Nonesupports_activity_reactions: bool = Truesupports_queue_reaction: bool = Trueactivity_reactions_per_agent: bool = False@@ -79,6 +87,14 @@ class RecordingPlatform:drawn: list[Drawn] = field(default_factory=list)marks: list[tuple[str, str, bool]] = field(default_factory=list)refuse_posts: bool = False+ # Raised for every post of a turn's activity while its turn id is here.+ turn_post_errors: dict[str, RichContentFailed] = field(default_factory=dict)+ # Turns whose next post opens a message, then is asked to wait before+ # drawing into it.+ open_undrawn: set[str] = field(default_factory=set)+ # Turns whose next post opens a message, then cannot draw into it.+ open_unfilled: set[str] = field(default_factory=set)+ held_for: float = 0.0found_card: str | None = Nonefirst_reply: bool = True_next: int = 0@@ -96,7 +112,22 @@ class RecordingPlatform:) -> str:if self.refuse_posts:raise RichContentFailed("refused", text="")+ turn_id = (+ content.turn.turn_id+ if isinstance(content, TurnActivity) and not content.status_only+ else None+ )+ if turn_id in self.turn_post_errors:+ raise self.turn_post_errors[turn_id]ref = self._ref()+ if turn_id in self.open_undrawn:+ self.open_undrawn.discard(turn_id)+ self.drawn.append(Drawn("opened", channel_id, ref, thread_root_id, None))+ raise RichContentThrottledAfterPost(ref=ref, retry_after=0.2, text="")+ if turn_id in self.open_unfilled:+ self.open_unfilled.discard(turn_id)+ self.drawn.append(Drawn("opened", channel_id, ref, thread_root_id, None))+ raise RichContentFailedAfterPost("refused", ref=ref, text="")self.drawn.append(Drawn("post_rich", channel_id, ref, thread_root_id, content))return ref@@ -112,6 +143,12 @@ class RecordingPlatform:Drawn("update_rich", channel_id, message_ref, thread_root_id, content))+ def rich_updates_held_for(self) -> float:+ return self.held_for++ def activity_closes_in(self, message_ref: str) -> float | None:+ return self.closes_in+async def send_message(self,channel_id: str,diff --git a/core/tests/switch_core/session_activity/test_publisher.py b/core/tests/switch_core/session_activity/test_publisher.pyindex 45cf07091..b94ea2482 100644--- a/core/tests/switch_core/session_activity/test_publisher.py+++ b/core/tests/switch_core/session_activity/test_publisher.py@@ -3,18 +3,25 @@ from __future__ import annotationsimport asyncioimport uuidfrom collections.abc import AsyncIterator-from datetime import UTC, datetime+from datetime import UTC, datetime, timedeltafrom types import SimpleNamespacefrom unittest.mock import AsyncMock, MagicMockimport pytest-from sqlalchemy import select+from sqlalchemy import delete, selectfrom sqlalchemy.ext.asyncio import create_async_enginefrom sqlalchemy.pool import NullPool-from switch_core.bridges.collaboration.adapter import RequestCard, TurnActivity+from switch_core.bridges.collaboration.adapter import (+ ActivityMarkRefused,+ RequestCard,+ RichContentFailed,+ RichContentThrottled,+ TurnActivity,+)from switch_core.db.models import (Agent,+ AgentSessionActivityItem,ApprovalRequestPost,BridgeMessageMap,Client,@@ -701,6 +708,260 @@ async def test_a_restart_redraws_running_turns_but_not_ones_already_shown_ended(await publisher.stop()+# ββ Under a rate limit βββββββββββββββββββββββββββββββββββββββββββββββββββββββ+++def _key(turn_id: str) -> tuple[str, ...]:+ return ("turn", AGENT, SESSION, turn_id)+++async def test_the_asking_message_is_marked_before_its_card_can_be_posted(+ service, publisher, platform, bridged, session_factory+):+ """The π says the agent has the message, so it goes on as the turn starts+ rather than once the platform has room for the card."""+ platform.turn_post_errors["turn-1"] = RichContentThrottled(retry_after=0.2, text="")+ await _turn(service, bridged.room_id, "running", 1)+ await platform.wait_until(lambda: platform.marks == [(ASKED_POST, "working", True)])+ assert _turns(platform) == []++ del platform.turn_post_errors["turn-1"]+ await platform.wait_until(lambda: len(_turns(platform)) == 1)+ assert platform.marks == [(ASKED_POST, "working", True)]+ assert (await _turn_post(session_factory)).mark == "working"+++async def test_a_turn_that_ends_before_its_card_posts_takes_its_mark_off(+ service, publisher, platform, bridged+):+ platform.turn_post_errors["turn-1"] = RichContentFailed("refused", text="")+ await _turn(service, bridged.room_id, "running", 1)+ await platform.wait_until(lambda: platform.marks == [(ASKED_POST, "working", True)])++ await _turn(service, bridged.room_id, "completed", 2)+ await platform.wait_until(lambda: len(platform.marks) == 2)+ assert platform.marks[-1] == (ASKED_POST, "working", False)+++async def test_a_mark_put_on_before_its_card_is_not_taken_off_by_another_turn(+ service, publisher, platform, bridged+):+ """Every agent shares one π here, and a turn whose card has not posted yet+ still holds it."""+ await _turn(service, bridged.room_id, "running", 1, turn_id="turn-1")+ await platform.wait_until(lambda: len(platform.marks) == 1)+ platform.turn_post_errors["turn-2"] = RichContentThrottled(retry_after=0.2, text="")+ await _turn(service, bridged.room_id, "running", 1, turn_id="turn-2")+ await platform.wait_until(lambda: len(platform.marks) == 2)++ await _turn(service, bridged.room_id, "completed", 2, turn_id="turn-1")+ await asyncio.sleep(0.5)+ assert all(on for _, _, on in platform.marks)+++async def test_a_card_opened_but_not_yet_drawn_is_drawn_into_rather_than_posted_again(+ service, publisher, platform, bridged, session_factory+):+ platform.open_undrawn.add("turn-1")+ await _turn(service, bridged.room_id, "running", 1)+ await platform.wait_until(lambda: len(_turns(platform)) == 1)++ [opened] = platform.of("opened")+ [drawn] = _turns(platform)+ assert (drawn.call, drawn.ref) == ("update_rich", opened.ref)+ assert platform.of("post_rich") == []+ assert (await _turn_post(session_factory)).external_post_id == opened.ref+++async def test_a_card_opened_but_not_drawn_into_is_kept_for_the_next_change(+ service, publisher, platform, bridged, session_factory+):+ """Any failure once the message exists, not only a rate limit: posting+ again would leave the empty one above the new one."""+ platform.open_unfilled.add("turn-1")+ await _turn(service, bridged.room_id, "running", 1)+ await platform.wait_until(lambda: len(platform.of("opened")) == 1)+ [opened] = platform.of("opened")+ assert (await _turn_post(session_factory)).external_post_id == opened.ref++ await _step(+ service,+ bridged.room_id,+ item_id="tool-1",+ kind="tool-activity",+ status="in-progress",+ revision=1,+ title="Ran `pytest`",+ )+ await platform.wait_until(lambda: len(_turns(platform)) == 1)+ [drawn] = _turns(platform)+ assert (drawn.call, drawn.ref) == ("update_rich", opened.ref)+ assert platform.of("post_rich") == []+++async def test_an_ended_turn_whose_card_could_not_be_drawn_backs_off_as_usual(+ service, publisher, platform, bridged+):+ """Keeping the message is not the draw succeeding: the retry still widens."""+ platform.open_unfilled.add("turn-1")+ await _turn(service, bridged.room_id, "completed", 1)+ await platform.wait_until(lambda: len(platform.of("opened")) == 1)++ assert publisher._retries.waits[_key("turn-1")] == 10.0+++async def test_the_asking_message_is_marked_once_on_a_turns_first_draw(+ service, publisher, platform, bridged+):+ """A platform that refuses the mark is asked once per pass, not twice."""+ asked = 0+ refuse = platform.mark_activity++ async def refused(*args, **kwargs):+ nonlocal asked+ asked += 1+ await refuse(*args, **kwargs)+ raise ActivityMarkRefused("No permission to react here.")++ platform.mark_activity = refused # type: ignore[method-assign]+ await _turn(service, bridged.room_id, "running", 1)+ await platform.wait_until(lambda: len(_turns(platform)) == 1)++ assert asked == 1+++async def test_a_running_turn_is_not_redrawn_every_few_seconds_for_its_clock(+ service, session_factory, bridged, listener, online, monkeypatch+):+ """Slack takes about fifty edits a minute for a whole workspace. A clock+ redrawn every five seconds spends that on cards where nothing happened, so+ it moves once a minute and every real change still goes out at once."""+ monkeypatch.setattr(publisher_module, "_TICK_SECONDS", 0.05)+ platform = RecordingPlatform(redraws_for_elapsed_time=True)+ publisher = _publisher(session_factory, bridged, listener, platform, online)+ publisher.start()+ try:+ await _turn(service, bridged.room_id, "running", 1)+ await platform.wait_until(lambda: len(_turns(platform)) == 1)+ await asyncio.sleep(2.5)+ assert len(_turns(platform)) == 1++ await _step(+ service,+ bridged.room_id,+ item_id="tool-1",+ kind="tool-activity",+ status="in-progress",+ revision=1,+ title="Ran `pytest`",+ )+ await platform.wait_until(lambda: len(_turns(platform)) == 2)+ finally:+ await publisher.stop()+++async def test_a_message_the_platform_closes_on_its_own_is_redrawn_just_after(+ service, session_factory, bridged, listener, online, monkeypatch+):+ """Slack closes a stream ten minutes after it opens and marks the steps+ still running as failed. Left to the clock, that could show for a minute."""+ monkeypatch.setattr(publisher_module, "_TICK_SECONDS", 0.05)+ monkeypatch.setattr(+ publisher_module, "_AFTER_PLATFORM_CLOSE", timedelta(seconds=0.1)+ )+ platform = RecordingPlatform(redraws_for_elapsed_time=True, closes_in=0.5)+ publisher = _publisher(session_factory, bridged, listener, platform, online)+ publisher.start()+ try:+ await _turn(service, bridged.room_id, "running", 1)+ await platform.wait_until(lambda: len(_turns(platform)) == 1)+ await platform.wait_until(lambda: len(_turns(platform)) == 2)+ assert _turns(platform)[1].call == "update_rich"++ await asyncio.sleep(1.0)+ assert len(_turns(platform)) == 2, "once, and then the clock again"+ finally:+ await publisher.stop()+++async def test_the_clock_asks_for_a_redraw_when_its_minute_turns_and_edits_are_taken():+ publisher = _bare_publisher()+ platform = publisher._adapter+ platform.redraws_for_elapsed_time = True # type: ignore[misc]+ now = datetime.now(UTC)+ for turn_id in ("due", "later"):+ publisher._live_turns[_key(turn_id)] = AGENT+ publisher._online_seen[AGENT] = True+ publisher._remembered(_key("due")).clock_due = now - timedelta(seconds=1)+ publisher._remembered(_key("later")).clock_due = now + timedelta(seconds=30)++ platform.held_for = 4.0 # type: ignore[attr-defined]+ publisher._tick(now)+ assert publisher._pending == {}++ platform.held_for = 0.0 # type: ignore[attr-defined]+ publisher._tick(now)+ assert publisher._pending == {_key("due"): publisher_module._CLOCK}+++async def test_new_work_goes_ahead_of_retries_and_retries_ahead_of_the_clock():+ """A new turn's card and π must not wait behind every running card's clock,+ nor behind every card a rate limit sent back at the same moment."""+ publisher = _bare_publisher()+ for turn_id in ("old-1", "old-2", "old-3"):+ publisher._enqueue(_key(turn_id), publisher_module._CLOCK)+ publisher._enqueue(_key("retry"), publisher_module._SOON)+ publisher._enqueue(_key("new"), publisher_module._NOW)+ # Something happened on a turn that was only waiting for its clock.+ publisher._enqueue(_key("old-3"), publisher_module._NOW)++ order = [(await publisher._next())[0][3] for _ in range(5)]++ assert order == ["new", "old-3", "retry", "old-1", "old-2"]+ assert publisher._pending == {}+++async def test_a_clock_redraw_the_platform_refuses_is_dropped_but_a_change_is_retried(+ monkeypatch,+):+ """The next minute redraws the clock anyway; a real change is still owed."""+ publisher = _bare_publisher()+ handled: list[tuple[str, ...]] = []++ async def handle(key: tuple[str, ...]) -> None:+ handled.append(key)+ raise RichContentThrottled(retry_after=0.05, text="")++ monkeypatch.setattr(publisher, "_handle", handle)+ publisher._enqueue(_key("clock"), publisher_module._CLOCK)+ publisher._enqueue(_key("change"), publisher_module._NOW)+ running = asyncio.create_task(publisher._run())+ await asyncio.sleep(0.4)+ running.cancel()+ with pytest.raises(asyncio.CancelledError):+ await running++ assert handled.count(_key("clock")) == 1+ assert handled.count(_key("change")) > 1+++async def test_a_turn_whose_rows_were_pruned_is_no_longer_watched(+ service, publisher, platform, bridged, session_factory+):+ await _turn(service, bridged.room_id, "running", 1)+ await platform.wait_until(lambda: len(_turns(platform)) == 1)+ assert _key("turn-1") in publisher._live_turns++ async with session_factory() as db, db.begin():+ await db.execute(+ delete(AgentSessionActivityItem).where(+ AgentSessionActivityItem.turn_id == "turn-1"+ )+ )+ publisher._enqueue(_key("turn-1"), publisher_module._NOW)+ await platform.wait_until(lambda: _key("turn-1") not in publisher._live_turns)++# ββ Stopping βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ