Enterprise agent platform for building, deploying, and operating AI teammates.
feat(connectors): add smoke V2 Tier 1 read checks for Slack (#3747)
leyoonafrf67c1bcf9ace
diff --git a/docs/operations/connector-smoke-v2.md b/docs/operations/connector-smoke-v2.mdindex ca35810bd..819b412a3 100644--- a/docs/operations/connector-smoke-v2.md+++ b/docs/operations/connector-smoke-v2.md@@ -134,7 +134,7 @@ At run time (see [Running write checks](#running-write-checks)):Review each write flow against these, which validation cannot check:-- Cleanup and sweep use the connector's own tools, so a connector whose server cannot delete what its write check creates declares no write check until its Tier 1 issue settles how to clean up. Today that includes the HubSpot MCP server (#3572), the in-repo GitHub server (#3568), Slack's messages (#3571), Gmail's drafts (#3567) and Teams' messages (#3569).+- Cleanup and sweep use the connector's own tools, so a connector whose server cannot delete what its write check creates declares no write check until its Tier 1 issue settles how to clean up. Today that includes the HubSpot MCP server (#3572), the in-repo GitHub server (#3568), Slack's messages (#3743, split from its Tier 1 issue #3571), Gmail's drafts (#3567) and Teams' messages (#3569).- The sweep's `find` lists the same container `create` writes into. Jira's search takes only JQL, so its sweep reads a query fixture rather than the project key `create` takes.- Each step's target comes from its reference or a write-area fixture, never from a literal, or from the pin's key, next to one: validation sees that a reference is present, not which argument the tool treats as its target.- The cleanup guard finds the object by its `CreatedRef` alone, not by a query fixture or a literal that sits next to it. Only then does it catch a wrongly declared path before anything is deleted.@@ -199,6 +199,8 @@ The six document connectors (`google-docs`, `google-sheets`, `google-slides`, `p`jira` has adopted V2 at the Tier 1 bar (#3570). It keeps its L3 call and adds four read checks on the test tenant's seeded read project: `jira_search_issues` with the query `jira.seed.search_jql`, whose first issue is `jira.seed.issue_key`; `jira_list_comments` on that issue, whose first comment is `jira.seed.comment_id`; `jira_get_issue` on it; and `jira_get_current_user`. Its write check is the round trip shown under [Write checks and the sweep](#write-checks-and-the-sweep), with the sweep reading `jira.write_sweep_jql`. Each step that reads or writes issues passes `cloud_id` from `jira.tenant_id` instead of relying on the tool's single-site auto-detection; the tenant pin still refuses the write when the grant reaches any other site. The other seven tools are exempt: three reads the checked tools make unnecessary (they carry over 80% of Jira's calls), and four writes the write-check format cannot express, since each edits or adds to an existing issue rather than creating one it can delete.+`slack` has adopted V2 at the Tier 1 bar (#3571), with read checks only. It keeps its L3 call and adds four read checks on the QA workspace's seed, a public channel named `xagent-smoke-seed` with a set topic and one thread in it: `slack_list_channels` finds that channel by name and asserts its id, `slack.seed.channel_id`; `slack_get_channel_history` reads the channel up to the thread's reply, `slack.seed.reply_ts`, and asserts the thread's parent, `slack.seed.thread_ts`; `slack_get_thread_replies` reads that thread; and `slack_get_channel_info` asserts the channel's name and topic. The seed posts the thread last, after its bots joined and it set the topic, each of which leaves a message in the channel; the history read stops at the reply, which `latest` excludes, so a message posted to the channel later shifts nothing the checks assert. A later reply in the thread comes after the one `slack_get_thread_replies` asserts, and the history read asserts only that the parent has a reply count, which grows with it. `slack.seed.thread_ts` and `slack.seed.reply_ts` must be strings, as Slack writes a `ts` (`"1762905600.000100"`): as a tool argument a number is refused before the call (`call_failed`), in an assertion it never equals the string the tools return, and either way it loses digits. A new install of an app not on the Slack Marketplace may call `conversations.history` and `conversations.replies` once a minute and gets at most 15 messages back, so a run calls each of them once with a small limit, and `slack_search_messages`, which calls both again, is exempt. A second run on the same Slack app within about a minute gets a 429 whose `Retry-After` exceeds the 30 s the tool waits, so the throttled reads fail as `assertion_failed` with reason `status_success`, as they do when the seed's channel or thread is gone; leave a minute between runs before reading such a failure as drift. The QA workspace is on Slack's free plan, which hides messages older than 90 days, so the thread must be re-seeded before then, and the new `ts` values replace `slack.seed.thread_ts` and `slack.seed.reply_ts` in the tenant registry. `slack_list_direct_messages` is exempt because a Toby run's bot grant admits public channels only (xorbitsai/xagent-saas#2553), so its listing is empty. No write tool has a check: `slack_post_message` and `slack_upload_file` create what no tool of the server deletes, and whether `slack_join_channel` and the reaction tools get a write check is decided in #3743, since no tool reads the workspace's team id for a tenant pin and the bot grant only reads.+`gmail`, `google-analytics`, `google-calendar`, `google-docs`, `google-drive`, `google-sheets` and `google-slides` have adopted V2 against the Google test tenant (#3567). Every read check but the probe-derived `l3_probe` is bound to a fixture key the tenant registry records for the seed layout: `gmail.seed_label_id` (the label's one seeded message), `google-drive.seed_folder_id` (the closed seed folder, found by a Drive search), `google-docs.document_id` (read directly, and again through Drive's text export), `google-sheets.spreadsheet_id` (the spreadsheet's one tab, and the seeded `item`/`quantity` block through `google_sheets_read_range`), `google-slides.presentation_id` (the deck's title and its one slide), and `google-analytics.property_id`. The checks assert the seed's own values where the seed pins one, such as the document's text `xagent smoke seed document`, the spreadsheet's `alpha`/`1` row and the deck's one slide, so a change to that seed is a change to the checks that read it. Assertions the seed does not pin are shape guards: a spreadsheet tab's `row_count` and `column_count` are Google's default grid, and an Analytics report's `rows` is an array. `google_analytics_run_report` returns no property field of GA's own: `query.property_id` is its normalization of the argument it was given, so no assertion reads it, and `google-analytics.property_id` must be a bare-digit string (`563334905`), the form the tenant registry records. `google-analytics` also runs its probe-derived `l3_probe`, which takes no fixture. `google-calendar` keeps only that check, because every Calendar tool addresses `primary`, while the seed's calendar is a secondary one, so no seeded event is reachable.Every other Google tool is exempt: the reads outside the 80% cut; the writes, whose checks are decided with the write flows (#3567); the writes that create an object no tool of their server deletes (`google_docs_create_document`, `google_sheets_create_spreadsheet`, `google_slides_create_presentation`, `google_slides_import_pptx`, `gmail_send_messages`); and the in-place Docs edits (`google_docs_append_text`, `google_docs_replace_text`, `google_docs_batch_update`), which stay pending: a write check creates, verifies and deletes its own object and has no step for an edit, and the tenant registry sets aside no write-area document to edit. `google_slides_get_slide_thumbnail` is exempt because the tenant registry records no slide-id fixture: the deck the seed creates holds Google's default slide, but no fixture names it, and `gmail_manage_labels` is exempt because it is the only tool that reads a label and the only one that creates or deletes one, so a label write check's verify or cleanup guard would call its own create and cleanup tool (`#3653`); no separate label-reading tool is added.diff --git a/src/xagent/web/connector_checks/registry.py b/src/xagent/web/connector_checks/registry.pyindex a3236f6af..27454844e 100644--- a/src/xagent/web/connector_checks/registry.py+++ b/src/xagent/web/connector_checks/registry.py@@ -44,6 +44,11 @@ from .fixtures import FixtureRef_GOOGLE_TIER_1 = "#3567"_JIRA_TIER_1 = "#3570"_MICROSOFT_365_TIER_1 = "#3569"+_SLACK_TIER_1 = "#3571"+# Whether Slack gets write checks at all: no tool of its server deletes a+# message or a file, a reaction is named by an existing emoji, no tool reads+# the tenant id, and the bot grant only reads.+_SLACK_WRITE_CHECKS = "#3743"# Tier 2 runs structural reads only: no seeded objects and no write checks._TIER_2 = "#3573"WRITE_WITHOUT_CLEANUP = (@@ -343,6 +348,141 @@ def _jira_checks() -> ConnectorChecks:)+def _slack_checks() -> ConnectorChecks:+ """Slack's Tier 1 checks (#3571).++ Reads only, against the QA workspace's seed: a public channel with a set+ topic, and one thread in it, a parent with one reply. The thread is+ posted last, after the bots joined and the topic was set, each of which+ leaves a message in the channel. The checks cover the tools that carry+ most of Slack's calls; no write tool was called in the window the tools+ were picked from.++ A new installation of an app that is not on the Slack Marketplace may call+ ``conversations.history`` and ``conversations.replies`` once a minute and+ gets at most 15 messages back, so a run calls each of them once with a+ small limit. History is read up to the reply, which is exclusive, so a+ message posted to the channel after the seed shifts nothing the check+ asserts. A later reply in the thread comes after the one the thread read+ asserts; the parent's reply count grows with it, so the history read+ asserts only that the count is there.++ A second run on the same Slack app within about a minute gets a 429 whose+ ``Retry-After`` exceeds the 30 s the tool waits, so the throttled reads+ fail as ``assertion_failed`` (``status_success``), as they do when the+ seed's channel or thread is gone. The QA workspace is on Slack's free+ plan, which hides messages older than 90 days, so the thread must be+ re-seeded before then, with its new ``ts`` values in the registry.+ """+ channel = FixtureRef("slack.seed.channel_id")+ thread = FixtureRef("slack.seed.thread_ts")+ reply = FixtureRef("slack.seed.reply_ts")+ channel_name = "xagent-smoke-seed"+ return ConnectorChecks(+ "slack",+ reads=(+ _declared_l3("slack"),+ ReadCheck(+ "slack.find_seed_channel",+ "slack_list_channels",+ {"name_contains": channel_name, "limit": 10},+ assertions=(+ StatusSuccess(),+ HasKeys((), ("channels", "truncated")),+ Length(("channels",), 1, 1),+ EachHasKeys(("channels",), ("id", "name", "is_archived")),+ Equals(("channels", 0, "id"), channel),+ Equals(("channels", 0, "name"), channel_name),+ ),+ ),+ ReadCheck(+ "slack.read_seed_history",+ "slack_get_channel_history",+ {"channel": channel, "latest": reply, "limit": 1},+ assertions=(+ StatusSuccess(),+ HasKeys((), ("messages", "has_more", "next_cursor")),+ Length(("messages",), 1, 1),+ EachHasKeys(+ ("messages",),+ ("ts", "user", "text", "thread_ts", "reply_count"),+ ),+ Equals(("messages", 0, "ts"), thread),+ Equals(("messages", 0, "thread_ts"), thread),+ Equals(("messages", 0, "text"), "xagent smoke seed thread"),+ IsType(("messages", 0, "reply_count"), "number"),+ ),+ ),+ ReadCheck(+ "slack.read_seed_thread",+ "slack_get_thread_replies",+ {"channel": channel, "thread_ts": thread, "limit": 15},+ assertions=(+ StatusSuccess(),+ HasKeys((), ("messages", "has_more", "next_cursor")),+ Length(("messages",), 2),+ EachHasKeys(("messages",), ("ts", "user", "text")),+ Equals(("messages", 0, "ts"), thread),+ Equals(("messages", 1, "ts"), reply),+ Equals(("messages", 1, "text"), "xagent smoke seed reply"),+ ),+ ),+ ReadCheck(+ "slack.read_seed_channel",+ "slack_get_channel_info",+ {"channel": channel},+ assertions=(+ StatusSuccess(),+ HasKeys(+ (),+ ("id", "name", "topic", "purpose", "is_archived", "is_private"),+ ),+ Equals(("id",), channel),+ Equals(("name",), channel_name),+ Equals(("topic",), "xagent smoke seed topic"),+ Equals(("is_archived",), False),+ Equals(("is_private",), False),+ ),+ ),+ ),+ exemptions=(+ ToolExemption(+ "slack_search_messages",+ "read left unchecked: it calls conversations.history and "+ "conversations.replies again, which a new install of an app not "+ "on the Slack Marketplace may each call once a minute, after "+ "slack.read_seed_history and slack.read_seed_thread have; the checked "+ f"tools carry over 80% of Slack's calls, as decided in {_SLACK_TIER_1}",+ ),+ ToolExemption(+ "slack_list_direct_messages",+ "read left unchecked: a Toby run's bot grant admits public channels "+ "only (xorbitsai/xagent-saas#2553), so the listing it sees is empty "+ "and holds nothing seeded; the checked tools carry over 80% of "+ f"Slack's calls, as decided in {_SLACK_TIER_1}",+ ),+ ToolExemption("slack_post_message", WRITE_WITHOUT_CLEANUP),+ ToolExemption("slack_upload_file", WRITE_WITHOUT_CLEANUP),+ ToolExemption(+ "slack_join_channel",+ "write that adds the bot to a channel, which no tool of this "+ "server leaves again, so a write check could not undo it; whether "+ f"Slack gets write checks is decided in {_SLACK_WRITE_CHECKS}",+ ),+ *(+ ToolExemption(+ tool,+ "write on a reaction, which is named by an existing emoji, so a "+ "write check cannot give it the run-scoped name its create must "+ "use; "+ f"whether Slack gets write checks is decided in {_SLACK_WRITE_CHECKS}",+ )+ for tool in ("slack_add_reaction", "slack_remove_reaction")+ ),+ ),+ )++def _onedrive_checks() -> ConnectorChecks:"""OneDrive's Tier 1 read checks (#3569), on the closed seed folder ofthe test account's drive and the text note in it."""@@ -1136,6 +1276,7 @@ CONNECTOR_CHECKS: Mapping[str, ConnectorChecks] = {in_place_edits=("sharepoint_update_list_item",),),),+ "slack": _slack_checks(),"teams": _teams_checks(),"word": ConnectorChecks("word",@@ -1321,7 +1462,6 @@ LEGACY_CONNECTORS: frozenset[str] = frozenset("rocketlane","salesforce","shopify",- "slack","stripe","whatsapp","xero",diff --git a/tests/web/connector_checks/test_cli.py b/tests/web/connector_checks/test_cli.pyindex b543d2664..ba2bd723e 100644--- a/tests/web/connector_checks/test_cli.py+++ b/tests/web/connector_checks/test_cli.py@@ -85,8 +85,14 @@ def _slack(tool: FakeTool | None = None) -> FakeAdapter:return FakeAdapter([ConnectorTarget("slack")], {"slack": listed(tool)})+def _stripe() -> FakeAdapter:+ """A legacy connector, whose one check is its L3 probe."""+ tool = FakeTool("stripe_list_customers")+ return FakeAdapter([ConnectorTarget("stripe")], {"stripe": listed(tool)})++def test_a_passing_run_prints_its_document_and_exits_0(monkeypatch, capfd):- names = _install_adapter(monkeypatch, _slack())+ names = _install_adapter(monkeypatch, _stripe())code = cli.main(["--user", "monitor"])@@ -163,7 +169,7 @@ def test_a_run_the_total_timeout_cut_exits_1(monkeypatch, capfd):def test_skipped_rows_do_not_fail_the_run(monkeypatch, capfd):adapter = FakeAdapter(- [ConnectorTarget("slack", skip_reason="not_catalog_identity")]+ [ConnectorTarget("stripe", skip_reason="not_catalog_identity")])_install_adapter(monkeypatch, adapter)diff --git a/tests/web/connector_checks/test_registry.py b/tests/web/connector_checks/test_registry.pyindex 4773dbb82..d1d9f8ccb 100644--- a/tests/web/connector_checks/test_registry.py+++ b/tests/web/connector_checks/test_registry.py@@ -80,12 +80,12 @@ def _fixture_refs(value: Any) -> Iterator[FixtureRef]:def test_a_probed_connector_gets_its_l3_probe_as_one_read_check() -> None:- assert connector_checks_for("slack") == ConnectorChecks(- "slack",+ assert connector_checks_for("stripe") == ConnectorChecks(+ "stripe",reads=(ReadCheck(- "slack.l3_probe",- "slack_list_channels",+ "stripe.l3_probe",+ "stripe_list_customers",{"limit": 1},assertions=(L3Envelope(),),),diff --git a/tests/web/connector_checks/test_slack_checks.py b/tests/web/connector_checks/test_slack_checks.pynew file mode 100644index 000000000..c08420cd9--- /dev/null+++ b/tests/web/connector_checks/test_slack_checks.py@@ -0,0 +1,458 @@+"""Slack's smoke V2 Tier 1 checks (#3571), run end to end.++The declaration in ``registry.py`` runs through the real engine, and every+check calls the real tool function in ``web/tools/mcp/slack.py``. Only+Slack's Web API is replaced, by an in-memory workspace laid out like the QA+workspace's seed: one public channel whose topic is set, and one thread in+it, a parent message with one reply, posted after the bots joined and the+topic was set.+"""++from __future__ import annotations++import json+import time+from collections import Counter+from collections.abc import Callable+from typing import Any++import pytest++from xagent.web.connector_checks.engine import run_checks+from xagent.web.connector_checks.registry import CONNECTOR_CHECKS+from xagent.web.connector_checks.results import RunResult+from xagent.web.tools.mcp import slack++from .fakes import FakeAdapter, FakeTool, listed, rows, targets++CHANNEL = "C0SEED00001"+OTHER_CHANNEL = "C0GENERAL01"+CHANNEL_NAME = "xagent-smoke-seed"+TOPIC = "xagent smoke seed topic"+THREAD = "1762905600.000100"+THREAD_TEXT = "xagent smoke seed thread"+REPLY = "1762905601.000200"+REPLY_TEXT = "xagent smoke seed reply"+FIXTURES = {+ "slack.seed.channel_id": CHANNEL,+ "slack.seed.thread_ts": THREAD,+ "slack.seed.reply_ts": REPLY,+}+++class _Response:+ status_code = 200+ headers: dict[str, str] = {}++ def __init__(self, payload: dict[str, Any]) -> None:+ self._payload = payload++ def raise_for_status(self) -> None:+ return None++ def json(self) -> dict[str, Any]:+ return self._payload+++class Workspace:+ """An in-memory Slack workspace answering the requests slack.py sends."""++ def __init__(self) -> None:+ self.revoked = False+ self.bot_is_member = True+ self.channels: dict[str, dict[str, Any]] = {+ CHANNEL: self._channel(CHANNEL, CHANNEL_NAME, TOPIC),+ OTHER_CHANNEL: self._channel(OTHER_CHANNEL, "general", ""),+ }+ # Top-level messages per channel, oldest first: the bots joined and+ # the topic was set before the seed posted its thread.+ self.messages: dict[str, list[dict[str, Any]]] = {+ CHANNEL: [+ {+ "ts": "1762905500.000100",+ "user": "U0BOT000001",+ "subtype": "channel_join",+ "text": "<@U0BOT000001> has joined the channel",+ },+ {+ "ts": "1762905510.000100",+ "user": "U0SEED00001",+ "subtype": "channel_topic",+ "text": f"set the channel topic: {TOPIC}",+ },+ {+ "ts": THREAD,+ "user": "U0SEED00001",+ "text": THREAD_TEXT,+ "thread_ts": THREAD,+ "reply_count": 1,+ },+ ],+ OTHER_CHANNEL: [],+ }+ self.replies: dict[str, list[dict[str, Any]]] = {+ THREAD: [+ {+ "ts": REPLY,+ "user": "U0SEED00001",+ "text": REPLY_TEXT,+ "thread_ts": THREAD,+ }+ ]+ }+ self.calls: list[tuple[str, str]] = []++ @staticmethod+ def _channel(channel_id: str, name: str, topic: str) -> dict:+ return {+ "id": channel_id,+ "name": name,+ "is_archived": False,+ "is_private": False,+ "is_im": False,+ "is_mpim": False,+ "topic": {"value": topic},+ "purpose": {"value": ""},+ }++ def request(self, *, method: str, url: str, **kwargs: Any) -> _Response:+ api = url.removeprefix(f"{slack.SLACK_BASE_URL}/")+ self.calls.append((method, api))+ return _Response(self._answer(api, kwargs.get("params") or {}))++ def _answer(self, api: str, params: dict[str, Any]) -> dict[str, Any]:+ if self.revoked:+ return {"ok": False, "error": "token_revoked"}+ if api == "conversations.list":+ assert params["types"] == "public_channel", params+ return {+ "ok": True,+ "channels": list(self.channels.values()),+ "response_metadata": {"next_cursor": ""},+ }+ channel = params["channel"]+ if channel not in self.channels:+ return {"ok": False, "error": "channel_not_found"}+ # Slack documents not_in_channel for conversations.history; the fake+ # gates no other method on membership.+ if (+ api == "conversations.history"+ and channel == CHANNEL+ and not self.bot_is_member+ ):+ return {"ok": False, "error": "not_in_channel"}+ if api == "conversations.info":+ return {"ok": True, "channel": self.channels[channel]}+ if api == "conversations.history":+ return self._history(channel, params)+ if api == "conversations.replies":+ parents = [m for m in self.messages[channel] if m["ts"] == params["ts"]]+ if not parents:+ return {"ok": False, "error": "thread_not_found"}+ found = [*parents, *self.replies.get(params["ts"], [])]+ limit = int(params["limit"])+ return {+ "ok": True,+ "messages": found[:limit],+ "has_more": len(found) > limit,+ }+ raise AssertionError(api)++ def _history(self, channel: str, params: dict[str, Any]) -> dict[str, Any]:+ # Newest first; ``latest`` is exclusive by default.+ found = [+ message+ for message in reversed(self.messages[channel])+ if float(message["ts"]) < float(params.get("latest") or "inf")+ ]+ limit = int(params["limit"])+ return {"ok": True, "messages": found[:limit], "has_more": len(found) > limit}++ def post(self, ts: str, text: str, *, thread: str | None = None) -> None:+ message = {"ts": ts, "user": "U0OTHER0001", "text": text}+ if thread is None:+ self.messages[CHANNEL].append(message)+ else:+ self.replies[thread].append({**message, "thread_ts": thread})+ # Slack counts a parent's replies as they are posted.+ (parent,) = [m for m in self.messages[CHANNEL] if m["ts"] == thread]+ parent["reply_count"] += 1+++def _tool(name: str) -> FakeTool:+ function: Callable[..., str] = getattr(slack, name)++ def respond(args: dict[str, Any]) -> Any:+ return {+ "content": [{"type": "text", "text": function(**args)}],+ "structured_content": None,+ "is_error": False,+ }++ return FakeTool(name, respond=respond)+++# Every tool the server registers, so the session lists what production does+# without a channel access policy.+_TOOLS = (+ "slack_join_channel",+ "slack_list_channels",+ "slack_post_message",+ "slack_get_channel_history",+ "slack_get_thread_replies",+ "slack_get_channel_info",+ "slack_list_direct_messages",+ "slack_search_messages",+ "slack_add_reaction",+ "slack_remove_reaction",+ "slack_upload_file",+)+++def workspace(monkeypatch: pytest.MonkeyPatch) -> Workspace:+ workspace = Workspace()+ monkeypatch.setenv("SLACK_ACCESS_TOKEN", "test-token")+ monkeypatch.delenv(slack._CHANNEL_ACCESS_POLICY_ENV_VAR, raising=False)+ monkeypatch.setattr(slack.requests, "request", workspace.request)+ return workspace+++def _adapter(fixtures: dict[str, Any] = FIXTURES) -> FakeAdapter:+ tools = [_tool(name) for name in _TOOLS]+ return FakeAdapter(targets("slack"), {"slack": listed(*tools)}, fixtures=fixtures)+++async def _run(adapter: FakeAdapter) -> RunResult:+ return await run_checks(adapter, check_timeout=5.0, total_timeout=60.0)+++def _check_rows(result: RunResult) -> dict[str, tuple[Any, ...]]:+ # The session row carries no check id.+ return {row[1] or "session": row[2:] for row in rows(result) if row[0] == "slack"}+++_CHECKS = (+ "slack.l3_probe",+ "slack.find_seed_channel",+ "slack.read_seed_history",+ "slack.read_seed_thread",+ "slack.read_seed_channel",+)+_PASSED = {+ "session": ("pass", None, None),+ **{check: ("pass", None, None) for check in _CHECKS},+}+++# The declaration+++def test_slack_checks_the_tools_that_carry_its_calls() -> None:+ checks = CONNECTOR_CHECKS["slack"]++ assert [check.id for check in checks.reads] == list(_CHECKS)+ assert checks.writes == ()+ assert checks.sweep is None+ assert checks.checked_tools() == {+ "slack_list_channels",+ "slack_get_channel_history",+ "slack_get_thread_replies",+ "slack_get_channel_info",+ }+ assert sorted(exemption.tool for exemption in checks.exemptions) == [+ "slack_add_reaction",+ "slack_join_channel",+ "slack_list_direct_messages",+ "slack_post_message",+ "slack_remove_reaction",+ "slack_search_messages",+ "slack_upload_file",+ ]+++def test_each_exemption_names_the_issue_that_decides_it() -> None:+ reasons = {e.tool: e.reason for e in CONNECTOR_CHECKS["slack"].exemptions}++ for tool in ("slack_join_channel", "slack_add_reaction", "slack_remove_reaction"):+ assert reasons[tool].endswith("decided in #3743"), tool+ for tool in ("slack_search_messages", "slack_list_direct_messages"):+ assert reasons[tool].endswith("decided in #3571"), tool+++# A run against the seeded workspace+++async def test_every_check_passes_on_the_seeded_workspace_and_writes_nothing(+ workspace: Workspace,+) -> None:+ result = await _run(_adapter())++ assert _check_rows(result) == _PASSED+ assert result.complete+ assert {method for method, _ in workspace.calls} == {"GET"}+++async def test_a_run_calls_each_throttled_method_once(workspace: Workspace) -> None:+ await _run(_adapter())++ # Slack allows a new non-Marketplace installation one call a minute to each.+ called = Counter(api for _, api in workspace.calls)+ assert called["conversations.history"] == 1+ assert called["conversations.replies"] == 1+++async def test_every_check_passes_under_the_bot_grants_read_policy(+ workspace: Workspace, monkeypatch: pytest.MonkeyPatch+) -> None:+ # What a Toby run starts Slack with: a read-only policy that admits the+ # public seed channel (xorbitsai/xagent-saas#2553).+ monkeypatch.setenv(+ slack._CHANNEL_ACCESS_POLICY_ENV_VAR,+ json.dumps(+ {+ "version": 2,+ "channel_ids": [CHANNEL],+ "expires_at": time.time() + 300,+ "capabilities": ["read"],+ }+ ),+ )++ result = await _run(_adapter())++ assert _check_rows(result) == _PASSED+++async def test_later_messages_in_the_seed_channel_break_no_check(+ workspace: Workspace,+) -> None:+ # Another bot joins after the seed, someone posts, and the thread grows.+ workspace.post("1762909200.000100", "<@U0BOT000002> has joined the channel")+ workspace.post("1762909300.000100", "an unrelated message")+ workspace.post("1762909400.000100", "a later reply", thread=THREAD)++ result = await _run(_adapter())++ assert _check_rows(result) == _PASSED+++async def test_without_the_seeds_fixtures_only_the_probe_runs(+ workspace: Workspace,+) -> None:+ result = await _run(_adapter({}))++ checks = _check_rows(result)+ assert checks.pop("slack.l3_probe") == ("pass", None, None)+ assert checks.pop("session") == ("pass", None, None)+ assert {outcome[:2] for outcome in checks.values()} == {("skip", "fixture_missing")}+++async def test_a_revoked_grant_fails_every_check(workspace: Workspace) -> None:+ workspace.revoked = True++ result = await _run(_adapter())++ checks = _check_rows(result)+ assert checks.pop("session") == ("pass", None, None)+ assert set(checks) == set(_CHECKS)+ assert {outcome for outcome, _, _ in checks.values()} == {"fail"}+++async def test_a_bot_outside_the_seed_channel_fails_the_history_read(+ workspace: Workspace,+) -> None:+ workspace.bot_is_member = False++ result = await _run(_adapter())++ outcome, _, _ = _check_rows(result)["slack.read_seed_history"]+ assert outcome == "fail"+++ ("key", "value", "check"),+ [+ ("slack.seed.channel_id", OTHER_CHANNEL, "slack.find_seed_channel"),+ ("slack.seed.channel_id", OTHER_CHANNEL, "slack.read_seed_channel"),+ ("slack.seed.thread_ts", "1762905599.000100", "slack.read_seed_history"),+ ("slack.seed.thread_ts", "1762905599.000100", "slack.read_seed_thread"),+ ("slack.seed.reply_ts", "1762905602.000300", "slack.read_seed_thread"),+ ],+)+async def test_a_seeded_object_that_drifted_fails_its_read(+ workspace: Workspace, key: str, value: str, check: str+) -> None:+ result = await _run(_adapter({**FIXTURES, key: value}))++ outcome, failure_class, _ = _check_rows(result)[check]+ assert (outcome, failure_class) == ("fail", "assertion_failed")+++ ("key", "check"),+ [+ ("slack.seed.thread_ts", "slack.read_seed_history"),+ ("slack.seed.reply_ts", "slack.read_seed_thread"),+ ],+)+async def test_a_ts_fixture_recorded_as_a_number_fails_its_read(+ workspace: Workspace, key: str, check: str+) -> None:+ # A registry that records a ts unquoted loads it as a float. Here it only+ # reaches an Equals, which never matches the string the tools return. As+ # a tool argument (thread_ts, latest) FastMCP refuses it before the call,+ # which this fake, calling the functions directly, does not model.+ result = await _run(_adapter({**FIXTURES, key: float(FIXTURES[key])}))++ outcome, failure_class, _ = _check_rows(result)[check]+ assert (outcome, failure_class) == ("fail", "assertion_failed")+++ ("drift", "check"),+ [+ (+ lambda w: w.channels[CHANNEL].update(name="renamed"),+ "slack.find_seed_channel",+ ),+ (+ lambda w: w.channels[CHANNEL]["topic"].update(value="changed"),+ "slack.read_seed_channel",+ ),+ (+ lambda w: w.messages[CHANNEL][-1].update(text="edited"),+ "slack.read_seed_history",+ ),+ (+ lambda w: w.messages[CHANNEL][-1].pop("reply_count"),+ "slack.read_seed_history",+ ),+ (+ lambda w: w.replies[THREAD][0].update(text="edited"),+ "slack.read_seed_thread",+ ),+ ],+)+async def test_seeded_content_that_changed_fails_its_read(+ workspace: Workspace, drift: Callable[[Workspace], None], check: str+) -> None:+ drift(workspace)++ result = await _run(_adapter())++ outcome, failure_class, _ = _check_rows(result)[check]+ assert (outcome, failure_class) == ("fail", "assertion_failed")+++async def test_a_seed_channel_whose_name_matches_twice_fails_the_lookup(+ workspace: Workspace,+) -> None:+ workspace.channels["C0SEED00002"] = Workspace._channel(+ "C0SEED00002", f"{CHANNEL_NAME}-copy", ""+ )++ result = await _run(_adapter())++ outcome, failure_class, _ = _check_rows(result)["slack.find_seed_channel"]+ assert (outcome, failure_class) == ("fail", "assertion_failed")diff --git a/tests/web/connector_checks/test_tool_coverage.py b/tests/web/connector_checks/test_tool_coverage.pyindex a3d3dde49..ef47eb081 100644--- a/tests/web/connector_checks/test_tool_coverage.py+++ b/tests/web/connector_checks/test_tool_coverage.py@@ -57,7 +57,6 @@ _PINNED_LEGACY = frozenset("rocketlane","salesforce","shopify",- "slack","stripe","whatsapp","xero",