Skip to content

Enterprise agent platform for building, deploying, and operating AI teammates.

feat(connectors): add smoke V2 Tier 1 read checks for Slack (#3747)

leyoonafrf67c1bcf9ace
diff --git a/docs/operations/connector-smoke-v2.md b/docs/operations/connector-smoke-v2.md
index ca35810bd..819b412a3 100644
--- a/docs/operations/connector-smoke-v2.md
+++ b/docs/operations/connector-smoke-v2.md
@@ -134,7 +134,7 @@ At run time (see [Running write checks](#running-write-checks)):
Review each write flow against these, which validation cannot check:
-- Cleanup and sweep use the connector's own tools, so a connector whose server cannot delete what its write check creates declares no write check until its Tier 1 issue settles how to clean up. Today that includes the HubSpot MCP server (#3572), the in-repo GitHub server (#3568), Slack's messages (#3571), Gmail's drafts (#3567) and Teams' messages (#3569).
+- Cleanup and sweep use the connector's own tools, so a connector whose server cannot delete what its write check creates declares no write check until its Tier 1 issue settles how to clean up. Today that includes the HubSpot MCP server (#3572), the in-repo GitHub server (#3568), Slack's messages (#3743, split from its Tier 1 issue #3571), Gmail's drafts (#3567) and Teams' messages (#3569).
- The sweep's `find` lists the same container `create` writes into. Jira's search takes only JQL, so its sweep reads a query fixture rather than the project key `create` takes.
- Each step's target comes from its reference or a write-area fixture, never from a literal, or from the pin's key, next to one: validation sees that a reference is present, not which argument the tool treats as its target.
- The cleanup guard finds the object by its `CreatedRef` alone, not by a query fixture or a literal that sits next to it. Only then does it catch a wrongly declared path before anything is deleted.
@@ -199,6 +199,8 @@ The six document connectors (`google-docs`, `google-sheets`, `google-slides`, `p
`jira` has adopted V2 at the Tier 1 bar (#3570). It keeps its L3 call and adds four read checks on the test tenant's seeded read project: `jira_search_issues` with the query `jira.seed.search_jql`, whose first issue is `jira.seed.issue_key`; `jira_list_comments` on that issue, whose first comment is `jira.seed.comment_id`; `jira_get_issue` on it; and `jira_get_current_user`. Its write check is the round trip shown under [Write checks and the sweep](#write-checks-and-the-sweep), with the sweep reading `jira.write_sweep_jql`. Each step that reads or writes issues passes `cloud_id` from `jira.tenant_id` instead of relying on the tool's single-site auto-detection; the tenant pin still refuses the write when the grant reaches any other site. The other seven tools are exempt: three reads the checked tools make unnecessary (they carry over 80% of Jira's calls), and four writes the write-check format cannot express, since each edits or adds to an existing issue rather than creating one it can delete.
+`slack` has adopted V2 at the Tier 1 bar (#3571), with read checks only. It keeps its L3 call and adds four read checks on the QA workspace's seed, a public channel named `xagent-smoke-seed` with a set topic and one thread in it: `slack_list_channels` finds that channel by name and asserts its id, `slack.seed.channel_id`; `slack_get_channel_history` reads the channel up to the thread's reply, `slack.seed.reply_ts`, and asserts the thread's parent, `slack.seed.thread_ts`; `slack_get_thread_replies` reads that thread; and `slack_get_channel_info` asserts the channel's name and topic. The seed posts the thread last, after its bots joined and it set the topic, each of which leaves a message in the channel; the history read stops at the reply, which `latest` excludes, so a message posted to the channel later shifts nothing the checks assert. A later reply in the thread comes after the one `slack_get_thread_replies` asserts, and the history read asserts only that the parent has a reply count, which grows with it. `slack.seed.thread_ts` and `slack.seed.reply_ts` must be strings, as Slack writes a `ts` (`"1762905600.000100"`): as a tool argument a number is refused before the call (`call_failed`), in an assertion it never equals the string the tools return, and either way it loses digits. A new install of an app not on the Slack Marketplace may call `conversations.history` and `conversations.replies` once a minute and gets at most 15 messages back, so a run calls each of them once with a small limit, and `slack_search_messages`, which calls both again, is exempt. A second run on the same Slack app within about a minute gets a 429 whose `Retry-After` exceeds the 30 s the tool waits, so the throttled reads fail as `assertion_failed` with reason `status_success`, as they do when the seed's channel or thread is gone; leave a minute between runs before reading such a failure as drift. The QA workspace is on Slack's free plan, which hides messages older than 90 days, so the thread must be re-seeded before then, and the new `ts` values replace `slack.seed.thread_ts` and `slack.seed.reply_ts` in the tenant registry. `slack_list_direct_messages` is exempt because a Toby run's bot grant admits public channels only (xorbitsai/xagent-saas#2553), so its listing is empty. No write tool has a check: `slack_post_message` and `slack_upload_file` create what no tool of the server deletes, and whether `slack_join_channel` and the reaction tools get a write check is decided in #3743, since no tool reads the workspace's team id for a tenant pin and the bot grant only reads.
+
`gmail`, `google-analytics`, `google-calendar`, `google-docs`, `google-drive`, `google-sheets` and `google-slides` have adopted V2 against the Google test tenant (#3567). Every read check but the probe-derived `l3_probe` is bound to a fixture key the tenant registry records for the seed layout: `gmail.seed_label_id` (the label's one seeded message), `google-drive.seed_folder_id` (the closed seed folder, found by a Drive search), `google-docs.document_id` (read directly, and again through Drive's text export), `google-sheets.spreadsheet_id` (the spreadsheet's one tab, and the seeded `item`/`quantity` block through `google_sheets_read_range`), `google-slides.presentation_id` (the deck's title and its one slide), and `google-analytics.property_id`. The checks assert the seed's own values where the seed pins one, such as the document's text `xagent smoke seed document`, the spreadsheet's `alpha`/`1` row and the deck's one slide, so a change to that seed is a change to the checks that read it. Assertions the seed does not pin are shape guards: a spreadsheet tab's `row_count` and `column_count` are Google's default grid, and an Analytics report's `rows` is an array. `google_analytics_run_report` returns no property field of GA's own: `query.property_id` is its normalization of the argument it was given, so no assertion reads it, and `google-analytics.property_id` must be a bare-digit string (`563334905`), the form the tenant registry records. `google-analytics` also runs its probe-derived `l3_probe`, which takes no fixture. `google-calendar` keeps only that check, because every Calendar tool addresses `primary`, while the seed's calendar is a secondary one, so no seeded event is reachable.
Every other Google tool is exempt: the reads outside the 80% cut; the writes, whose checks are decided with the write flows (#3567); the writes that create an object no tool of their server deletes (`google_docs_create_document`, `google_sheets_create_spreadsheet`, `google_slides_create_presentation`, `google_slides_import_pptx`, `gmail_send_messages`); and the in-place Docs edits (`google_docs_append_text`, `google_docs_replace_text`, `google_docs_batch_update`), which stay pending: a write check creates, verifies and deletes its own object and has no step for an edit, and the tenant registry sets aside no write-area document to edit. `google_slides_get_slide_thumbnail` is exempt because the tenant registry records no slide-id fixture: the deck the seed creates holds Google's default slide, but no fixture names it, and `gmail_manage_labels` is exempt because it is the only tool that reads a label and the only one that creates or deletes one, so a label write check's verify or cleanup guard would call its own create and cleanup tool (`#3653`); no separate label-reading tool is added.
diff --git a/src/xagent/web/connector_checks/registry.py b/src/xagent/web/connector_checks/registry.py
index a3236f6af..27454844e 100644
--- a/src/xagent/web/connector_checks/registry.py
+++ b/src/xagent/web/connector_checks/registry.py
@@ -44,6 +44,11 @@ from .fixtures import FixtureRef
_GOOGLE_TIER_1 = "#3567"
_JIRA_TIER_1 = "#3570"
_MICROSOFT_365_TIER_1 = "#3569"
+_SLACK_TIER_1 = "#3571"
+# Whether Slack gets write checks at all: no tool of its server deletes a
+# message or a file, a reaction is named by an existing emoji, no tool reads
+# the tenant id, and the bot grant only reads.
+_SLACK_WRITE_CHECKS = "#3743"
# Tier 2 runs structural reads only: no seeded objects and no write checks.
_TIER_2 = "#3573"
WRITE_WITHOUT_CLEANUP = (
@@ -343,6 +348,141 @@ def _jira_checks() -> ConnectorChecks:
)
+def _slack_checks() -> ConnectorChecks:
+ """Slack's Tier 1 checks (#3571).
+
+ Reads only, against the QA workspace's seed: a public channel with a set
+ topic, and one thread in it, a parent with one reply. The thread is
+ posted last, after the bots joined and the topic was set, each of which
+ leaves a message in the channel. The checks cover the tools that carry
+ most of Slack's calls; no write tool was called in the window the tools
+ were picked from.
+
+ A new installation of an app that is not on the Slack Marketplace may call
+ ``conversations.history`` and ``conversations.replies`` once a minute and
+ gets at most 15 messages back, so a run calls each of them once with a
+ small limit. History is read up to the reply, which is exclusive, so a
+ message posted to the channel after the seed shifts nothing the check
+ asserts. A later reply in the thread comes after the one the thread read
+ asserts; the parent's reply count grows with it, so the history read
+ asserts only that the count is there.
+
+ A second run on the same Slack app within about a minute gets a 429 whose
+ ``Retry-After`` exceeds the 30 s the tool waits, so the throttled reads
+ fail as ``assertion_failed`` (``status_success``), as they do when the
+ seed's channel or thread is gone. The QA workspace is on Slack's free
+ plan, which hides messages older than 90 days, so the thread must be
+ re-seeded before then, with its new ``ts`` values in the registry.
+ """
+ channel = FixtureRef("slack.seed.channel_id")
+ thread = FixtureRef("slack.seed.thread_ts")
+ reply = FixtureRef("slack.seed.reply_ts")
+ channel_name = "xagent-smoke-seed"
+ return ConnectorChecks(
+ "slack",
+ reads=(
+ _declared_l3("slack"),
+ ReadCheck(
+ "slack.find_seed_channel",
+ "slack_list_channels",
+ {"name_contains": channel_name, "limit": 10},
+ assertions=(
+ StatusSuccess(),
+ HasKeys((), ("channels", "truncated")),
+ Length(("channels",), 1, 1),
+ EachHasKeys(("channels",), ("id", "name", "is_archived")),
+ Equals(("channels", 0, "id"), channel),
+ Equals(("channels", 0, "name"), channel_name),
+ ),
+ ),
+ ReadCheck(
+ "slack.read_seed_history",
+ "slack_get_channel_history",
+ {"channel": channel, "latest": reply, "limit": 1},
+ assertions=(
+ StatusSuccess(),
+ HasKeys((), ("messages", "has_more", "next_cursor")),
+ Length(("messages",), 1, 1),
+ EachHasKeys(
+ ("messages",),
+ ("ts", "user", "text", "thread_ts", "reply_count"),
+ ),
+ Equals(("messages", 0, "ts"), thread),
+ Equals(("messages", 0, "thread_ts"), thread),
+ Equals(("messages", 0, "text"), "xagent smoke seed thread"),
+ IsType(("messages", 0, "reply_count"), "number"),
+ ),
+ ),
+ ReadCheck(
+ "slack.read_seed_thread",
+ "slack_get_thread_replies",
+ {"channel": channel, "thread_ts": thread, "limit": 15},
+ assertions=(
+ StatusSuccess(),
+ HasKeys((), ("messages", "has_more", "next_cursor")),
+ Length(("messages",), 2),
+ EachHasKeys(("messages",), ("ts", "user", "text")),
+ Equals(("messages", 0, "ts"), thread),
+ Equals(("messages", 1, "ts"), reply),
+ Equals(("messages", 1, "text"), "xagent smoke seed reply"),
+ ),
+ ),
+ ReadCheck(
+ "slack.read_seed_channel",
+ "slack_get_channel_info",
+ {"channel": channel},
+ assertions=(
+ StatusSuccess(),
+ HasKeys(
+ (),
+ ("id", "name", "topic", "purpose", "is_archived", "is_private"),
+ ),
+ Equals(("id",), channel),
+ Equals(("name",), channel_name),
+ Equals(("topic",), "xagent smoke seed topic"),
+ Equals(("is_archived",), False),
+ Equals(("is_private",), False),
+ ),
+ ),
+ ),
+ exemptions=(
+ ToolExemption(
+ "slack_search_messages",
+ "read left unchecked: it calls conversations.history and "
+ "conversations.replies again, which a new install of an app not "
+ "on the Slack Marketplace may each call once a minute, after "
+ "slack.read_seed_history and slack.read_seed_thread have; the checked "
+ f"tools carry over 80% of Slack's calls, as decided in {_SLACK_TIER_1}",
+ ),
+ ToolExemption(
+ "slack_list_direct_messages",
+ "read left unchecked: a Toby run's bot grant admits public channels "
+ "only (xorbitsai/xagent-saas#2553), so the listing it sees is empty "
+ "and holds nothing seeded; the checked tools carry over 80% of "
+ f"Slack's calls, as decided in {_SLACK_TIER_1}",
+ ),
+ ToolExemption("slack_post_message", WRITE_WITHOUT_CLEANUP),
+ ToolExemption("slack_upload_file", WRITE_WITHOUT_CLEANUP),
+ ToolExemption(
+ "slack_join_channel",
+ "write that adds the bot to a channel, which no tool of this "
+ "server leaves again, so a write check could not undo it; whether "
+ f"Slack gets write checks is decided in {_SLACK_WRITE_CHECKS}",
+ ),
+ *(
+ ToolExemption(
+ tool,
+ "write on a reaction, which is named by an existing emoji, so a "
+ "write check cannot give it the run-scoped name its create must "
+ "use; "
+ f"whether Slack gets write checks is decided in {_SLACK_WRITE_CHECKS}",
+ )
+ for tool in ("slack_add_reaction", "slack_remove_reaction")
+ ),
+ ),
+ )
+
+
def _onedrive_checks() -> ConnectorChecks:
"""OneDrive's Tier 1 read checks (#3569), on the closed seed folder of
the test account's drive and the text note in it."""
@@ -1136,6 +1276,7 @@ CONNECTOR_CHECKS: Mapping[str, ConnectorChecks] = {
in_place_edits=("sharepoint_update_list_item",),
),
),
+ "slack": _slack_checks(),
"teams": _teams_checks(),
"word": ConnectorChecks(
"word",
@@ -1321,7 +1462,6 @@ LEGACY_CONNECTORS: frozenset[str] = frozenset(
"rocketlane",
"salesforce",
"shopify",
- "slack",
"stripe",
"whatsapp",
"xero",
diff --git a/tests/web/connector_checks/test_cli.py b/tests/web/connector_checks/test_cli.py
index b543d2664..ba2bd723e 100644
--- a/tests/web/connector_checks/test_cli.py
+++ b/tests/web/connector_checks/test_cli.py
@@ -85,8 +85,14 @@ def _slack(tool: FakeTool | None = None) -> FakeAdapter:
return FakeAdapter([ConnectorTarget("slack")], {"slack": listed(tool)})
+def _stripe() -> FakeAdapter:
+ """A legacy connector, whose one check is its L3 probe."""
+ tool = FakeTool("stripe_list_customers")
+ return FakeAdapter([ConnectorTarget("stripe")], {"stripe": listed(tool)})
+
+
def test_a_passing_run_prints_its_document_and_exits_0(monkeypatch, capfd):
- names = _install_adapter(monkeypatch, _slack())
+ names = _install_adapter(monkeypatch, _stripe())
code = cli.main(["--user", "monitor"])
@@ -163,7 +169,7 @@ def test_a_run_the_total_timeout_cut_exits_1(monkeypatch, capfd):
def test_skipped_rows_do_not_fail_the_run(monkeypatch, capfd):
adapter = FakeAdapter(
- [ConnectorTarget("slack", skip_reason="not_catalog_identity")]
+ [ConnectorTarget("stripe", skip_reason="not_catalog_identity")]
)
_install_adapter(monkeypatch, adapter)
diff --git a/tests/web/connector_checks/test_registry.py b/tests/web/connector_checks/test_registry.py
index 4773dbb82..d1d9f8ccb 100644
--- a/tests/web/connector_checks/test_registry.py
+++ b/tests/web/connector_checks/test_registry.py
@@ -80,12 +80,12 @@ def _fixture_refs(value: Any) -> Iterator[FixtureRef]:
def test_a_probed_connector_gets_its_l3_probe_as_one_read_check() -> None:
- assert connector_checks_for("slack") == ConnectorChecks(
- "slack",
+ assert connector_checks_for("stripe") == ConnectorChecks(
+ "stripe",
reads=(
ReadCheck(
- "slack.l3_probe",
- "slack_list_channels",
+ "stripe.l3_probe",
+ "stripe_list_customers",
{"limit": 1},
assertions=(L3Envelope(),),
),
diff --git a/tests/web/connector_checks/test_slack_checks.py b/tests/web/connector_checks/test_slack_checks.py
new file mode 100644
index 000000000..c08420cd9
--- /dev/null
+++ b/tests/web/connector_checks/test_slack_checks.py
@@ -0,0 +1,458 @@
+"""Slack's smoke V2 Tier 1 checks (#3571), run end to end.
+
+The declaration in ``registry.py`` runs through the real engine, and every
+check calls the real tool function in ``web/tools/mcp/slack.py``. Only
+Slack's Web API is replaced, by an in-memory workspace laid out like the QA
+workspace's seed: one public channel whose topic is set, and one thread in
+it, a parent message with one reply, posted after the bots joined and the
+topic was set.
+"""
+
+from __future__ import annotations
+
+import json
+import time
+from collections import Counter
+from collections.abc import Callable
+from typing import Any
+
+import pytest
+
+from xagent.web.connector_checks.engine import run_checks
+from xagent.web.connector_checks.registry import CONNECTOR_CHECKS
+from xagent.web.connector_checks.results import RunResult
+from xagent.web.tools.mcp import slack
+
+from .fakes import FakeAdapter, FakeTool, listed, rows, targets
+
+CHANNEL = "C0SEED00001"
+OTHER_CHANNEL = "C0GENERAL01"
+CHANNEL_NAME = "xagent-smoke-seed"
+TOPIC = "xagent smoke seed topic"
+THREAD = "1762905600.000100"
+THREAD_TEXT = "xagent smoke seed thread"
+REPLY = "1762905601.000200"
+REPLY_TEXT = "xagent smoke seed reply"
+FIXTURES = {
+ "slack.seed.channel_id": CHANNEL,
+ "slack.seed.thread_ts": THREAD,
+ "slack.seed.reply_ts": REPLY,
+}
+
+
+class _Response:
+ status_code = 200
+ headers: dict[str, str] = {}
+
+ def __init__(self, payload: dict[str, Any]) -> None:
+ self._payload = payload
+
+ def raise_for_status(self) -> None:
+ return None
+
+ def json(self) -> dict[str, Any]:
+ return self._payload
+
+
+class Workspace:
+ """An in-memory Slack workspace answering the requests slack.py sends."""
+
+ def __init__(self) -> None:
+ self.revoked = False
+ self.bot_is_member = True
+ self.channels: dict[str, dict[str, Any]] = {
+ CHANNEL: self._channel(CHANNEL, CHANNEL_NAME, TOPIC),
+ OTHER_CHANNEL: self._channel(OTHER_CHANNEL, "general", ""),
+ }
+ # Top-level messages per channel, oldest first: the bots joined and
+ # the topic was set before the seed posted its thread.
+ self.messages: dict[str, list[dict[str, Any]]] = {
+ CHANNEL: [
+ {
+ "ts": "1762905500.000100",
+ "user": "U0BOT000001",
+ "subtype": "channel_join",
+ "text": "<@U0BOT000001> has joined the channel",
+ },
+ {
+ "ts": "1762905510.000100",
+ "user": "U0SEED00001",
+ "subtype": "channel_topic",
+ "text": f"set the channel topic: {TOPIC}",
+ },
+ {
+ "ts": THREAD,
+ "user": "U0SEED00001",
+ "text": THREAD_TEXT,
+ "thread_ts": THREAD,
+ "reply_count": 1,
+ },
+ ],
+ OTHER_CHANNEL: [],
+ }
+ self.replies: dict[str, list[dict[str, Any]]] = {
+ THREAD: [
+ {
+ "ts": REPLY,
+ "user": "U0SEED00001",
+ "text": REPLY_TEXT,
+ "thread_ts": THREAD,
+ }
+ ]
+ }
+ self.calls: list[tuple[str, str]] = []
+
+ @staticmethod
+ def _channel(channel_id: str, name: str, topic: str) -> dict:
+ return {
+ "id": channel_id,
+ "name": name,
+ "is_archived": False,
+ "is_private": False,
+ "is_im": False,
+ "is_mpim": False,
+ "topic": {"value": topic},
+ "purpose": {"value": ""},
+ }
+
+ def request(self, *, method: str, url: str, **kwargs: Any) -> _Response:
+ api = url.removeprefix(f"{slack.SLACK_BASE_URL}/")
+ self.calls.append((method, api))
+ return _Response(self._answer(api, kwargs.get("params") or {}))
+
+ def _answer(self, api: str, params: dict[str, Any]) -> dict[str, Any]:
+ if self.revoked:
+ return {"ok": False, "error": "token_revoked"}
+ if api == "conversations.list":
+ assert params["types"] == "public_channel", params
+ return {
+ "ok": True,
+ "channels": list(self.channels.values()),
+ "response_metadata": {"next_cursor": ""},
+ }
+ channel = params["channel"]
+ if channel not in self.channels:
+ return {"ok": False, "error": "channel_not_found"}
+ # Slack documents not_in_channel for conversations.history; the fake
+ # gates no other method on membership.
+ if (
+ api == "conversations.history"
+ and channel == CHANNEL
+ and not self.bot_is_member
+ ):
+ return {"ok": False, "error": "not_in_channel"}
+ if api == "conversations.info":
+ return {"ok": True, "channel": self.channels[channel]}
+ if api == "conversations.history":
+ return self._history(channel, params)
+ if api == "conversations.replies":
+ parents = [m for m in self.messages[channel] if m["ts"] == params["ts"]]
+ if not parents:
+ return {"ok": False, "error": "thread_not_found"}
+ found = [*parents, *self.replies.get(params["ts"], [])]
+ limit = int(params["limit"])
+ return {
+ "ok": True,
+ "messages": found[:limit],
+ "has_more": len(found) > limit,
+ }
+ raise AssertionError(api)
+
+ def _history(self, channel: str, params: dict[str, Any]) -> dict[str, Any]:
+ # Newest first; ``latest`` is exclusive by default.
+ found = [
+ message
+ for message in reversed(self.messages[channel])
+ if float(message["ts"]) < float(params.get("latest") or "inf")
+ ]
+ limit = int(params["limit"])
+ return {"ok": True, "messages": found[:limit], "has_more": len(found) > limit}
+
+ def post(self, ts: str, text: str, *, thread: str | None = None) -> None:
+ message = {"ts": ts, "user": "U0OTHER0001", "text": text}
+ if thread is None:
+ self.messages[CHANNEL].append(message)
+ else:
+ self.replies[thread].append({**message, "thread_ts": thread})
+ # Slack counts a parent's replies as they are posted.
+ (parent,) = [m for m in self.messages[CHANNEL] if m["ts"] == thread]
+ parent["reply_count"] += 1
+
+
+def _tool(name: str) -> FakeTool:
+ function: Callable[..., str] = getattr(slack, name)
+
+ def respond(args: dict[str, Any]) -> Any:
+ return {
+ "content": [{"type": "text", "text": function(**args)}],
+ "structured_content": None,
+ "is_error": False,
+ }
+
+ return FakeTool(name, respond=respond)
+
+
+# Every tool the server registers, so the session lists what production does
+# without a channel access policy.
+_TOOLS = (
+ "slack_join_channel",
+ "slack_list_channels",
+ "slack_post_message",
+ "slack_get_channel_history",
+ "slack_get_thread_replies",
+ "slack_get_channel_info",
+ "slack_list_direct_messages",
+ "slack_search_messages",
+ "slack_add_reaction",
+ "slack_remove_reaction",
+ "slack_upload_file",
+)
+
+
+def workspace(monkeypatch: pytest.MonkeyPatch) -> Workspace:
+ workspace = Workspace()
+ monkeypatch.setenv("SLACK_ACCESS_TOKEN", "test-token")
+ monkeypatch.delenv(slack._CHANNEL_ACCESS_POLICY_ENV_VAR, raising=False)
+ monkeypatch.setattr(slack.requests, "request", workspace.request)
+ return workspace
+
+
+def _adapter(fixtures: dict[str, Any] = FIXTURES) -> FakeAdapter:
+ tools = [_tool(name) for name in _TOOLS]
+ return FakeAdapter(targets("slack"), {"slack": listed(*tools)}, fixtures=fixtures)
+
+
+async def _run(adapter: FakeAdapter) -> RunResult:
+ return await run_checks(adapter, check_timeout=5.0, total_timeout=60.0)
+
+
+def _check_rows(result: RunResult) -> dict[str, tuple[Any, ...]]:
+ # The session row carries no check id.
+ return {row[1] or "session": row[2:] for row in rows(result) if row[0] == "slack"}
+
+
+_CHECKS = (
+ "slack.l3_probe",
+ "slack.find_seed_channel",
+ "slack.read_seed_history",
+ "slack.read_seed_thread",
+ "slack.read_seed_channel",
+)
+_PASSED = {
+ "session": ("pass", None, None),
+ **{check: ("pass", None, None) for check in _CHECKS},
+}
+
+
+# The declaration
+
+
+def test_slack_checks_the_tools_that_carry_its_calls() -> None:
+ checks = CONNECTOR_CHECKS["slack"]
+
+ assert [check.id for check in checks.reads] == list(_CHECKS)
+ assert checks.writes == ()
+ assert checks.sweep is None
+ assert checks.checked_tools() == {
+ "slack_list_channels",
+ "slack_get_channel_history",
+ "slack_get_thread_replies",
+ "slack_get_channel_info",
+ }
+ assert sorted(exemption.tool for exemption in checks.exemptions) == [
+ "slack_add_reaction",
+ "slack_join_channel",
+ "slack_list_direct_messages",
+ "slack_post_message",
+ "slack_remove_reaction",
+ "slack_search_messages",
+ "slack_upload_file",
+ ]
+
+
+def test_each_exemption_names_the_issue_that_decides_it() -> None:
+ reasons = {e.tool: e.reason for e in CONNECTOR_CHECKS["slack"].exemptions}
+
+ for tool in ("slack_join_channel", "slack_add_reaction", "slack_remove_reaction"):
+ assert reasons[tool].endswith("decided in #3743"), tool
+ for tool in ("slack_search_messages", "slack_list_direct_messages"):
+ assert reasons[tool].endswith("decided in #3571"), tool
+
+
+# A run against the seeded workspace
+
+
+async def test_every_check_passes_on_the_seeded_workspace_and_writes_nothing(
+ workspace: Workspace,
+) -> None:
+ result = await _run(_adapter())
+
+ assert _check_rows(result) == _PASSED
+ assert result.complete
+ assert {method for method, _ in workspace.calls} == {"GET"}
+
+
+async def test_a_run_calls_each_throttled_method_once(workspace: Workspace) -> None:
+ await _run(_adapter())
+
+ # Slack allows a new non-Marketplace installation one call a minute to each.
+ called = Counter(api for _, api in workspace.calls)
+ assert called["conversations.history"] == 1
+ assert called["conversations.replies"] == 1
+
+
+async def test_every_check_passes_under_the_bot_grants_read_policy(
+ workspace: Workspace, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # What a Toby run starts Slack with: a read-only policy that admits the
+ # public seed channel (xorbitsai/xagent-saas#2553).
+ monkeypatch.setenv(
+ slack._CHANNEL_ACCESS_POLICY_ENV_VAR,
+ json.dumps(
+ {
+ "version": 2,
+ "channel_ids": [CHANNEL],
+ "expires_at": time.time() + 300,
+ "capabilities": ["read"],
+ }
+ ),
+ )
+
+ result = await _run(_adapter())
+
+ assert _check_rows(result) == _PASSED
+
+
+async def test_later_messages_in_the_seed_channel_break_no_check(
+ workspace: Workspace,
+) -> None:
+ # Another bot joins after the seed, someone posts, and the thread grows.
+ workspace.post("1762909200.000100", "<@U0BOT000002> has joined the channel")
+ workspace.post("1762909300.000100", "an unrelated message")
+ workspace.post("1762909400.000100", "a later reply", thread=THREAD)
+
+ result = await _run(_adapter())
+
+ assert _check_rows(result) == _PASSED
+
+
+async def test_without_the_seeds_fixtures_only_the_probe_runs(
+ workspace: Workspace,
+) -> None:
+ result = await _run(_adapter({}))
+
+ checks = _check_rows(result)
+ assert checks.pop("slack.l3_probe") == ("pass", None, None)
+ assert checks.pop("session") == ("pass", None, None)
+ assert {outcome[:2] for outcome in checks.values()} == {("skip", "fixture_missing")}
+
+
+async def test_a_revoked_grant_fails_every_check(workspace: Workspace) -> None:
+ workspace.revoked = True
+
+ result = await _run(_adapter())
+
+ checks = _check_rows(result)
+ assert checks.pop("session") == ("pass", None, None)
+ assert set(checks) == set(_CHECKS)
+ assert {outcome for outcome, _, _ in checks.values()} == {"fail"}
+
+
+async def test_a_bot_outside_the_seed_channel_fails_the_history_read(
+ workspace: Workspace,
+) -> None:
+ workspace.bot_is_member = False
+
+ result = await _run(_adapter())
+
+ outcome, _, _ = _check_rows(result)["slack.read_seed_history"]
+ assert outcome == "fail"
+
+
+ ("key", "value", "check"),
+ [
+ ("slack.seed.channel_id", OTHER_CHANNEL, "slack.find_seed_channel"),
+ ("slack.seed.channel_id", OTHER_CHANNEL, "slack.read_seed_channel"),
+ ("slack.seed.thread_ts", "1762905599.000100", "slack.read_seed_history"),
+ ("slack.seed.thread_ts", "1762905599.000100", "slack.read_seed_thread"),
+ ("slack.seed.reply_ts", "1762905602.000300", "slack.read_seed_thread"),
+ ],
+)
+async def test_a_seeded_object_that_drifted_fails_its_read(
+ workspace: Workspace, key: str, value: str, check: str
+) -> None:
+ result = await _run(_adapter({**FIXTURES, key: value}))
+
+ outcome, failure_class, _ = _check_rows(result)[check]
+ assert (outcome, failure_class) == ("fail", "assertion_failed")
+
+
+ ("key", "check"),
+ [
+ ("slack.seed.thread_ts", "slack.read_seed_history"),
+ ("slack.seed.reply_ts", "slack.read_seed_thread"),
+ ],
+)
+async def test_a_ts_fixture_recorded_as_a_number_fails_its_read(
+ workspace: Workspace, key: str, check: str
+) -> None:
+ # A registry that records a ts unquoted loads it as a float. Here it only
+ # reaches an Equals, which never matches the string the tools return. As
+ # a tool argument (thread_ts, latest) FastMCP refuses it before the call,
+ # which this fake, calling the functions directly, does not model.
+ result = await _run(_adapter({**FIXTURES, key: float(FIXTURES[key])}))
+
+ outcome, failure_class, _ = _check_rows(result)[check]
+ assert (outcome, failure_class) == ("fail", "assertion_failed")
+
+
+ ("drift", "check"),
+ [
+ (
+ lambda w: w.channels[CHANNEL].update(name="renamed"),
+ "slack.find_seed_channel",
+ ),
+ (
+ lambda w: w.channels[CHANNEL]["topic"].update(value="changed"),
+ "slack.read_seed_channel",
+ ),
+ (
+ lambda w: w.messages[CHANNEL][-1].update(text="edited"),
+ "slack.read_seed_history",
+ ),
+ (
+ lambda w: w.messages[CHANNEL][-1].pop("reply_count"),
+ "slack.read_seed_history",
+ ),
+ (
+ lambda w: w.replies[THREAD][0].update(text="edited"),
+ "slack.read_seed_thread",
+ ),
+ ],
+)
+async def test_seeded_content_that_changed_fails_its_read(
+ workspace: Workspace, drift: Callable[[Workspace], None], check: str
+) -> None:
+ drift(workspace)
+
+ result = await _run(_adapter())
+
+ outcome, failure_class, _ = _check_rows(result)[check]
+ assert (outcome, failure_class) == ("fail", "assertion_failed")
+
+
+async def test_a_seed_channel_whose_name_matches_twice_fails_the_lookup(
+ workspace: Workspace,
+) -> None:
+ workspace.channels["C0SEED00002"] = Workspace._channel(
+ "C0SEED00002", f"{CHANNEL_NAME}-copy", ""
+ )
+
+ result = await _run(_adapter())
+
+ outcome, failure_class, _ = _check_rows(result)["slack.find_seed_channel"]
+ assert (outcome, failure_class) == ("fail", "assertion_failed")
diff --git a/tests/web/connector_checks/test_tool_coverage.py b/tests/web/connector_checks/test_tool_coverage.py
index a3d3dde49..ef47eb081 100644
--- a/tests/web/connector_checks/test_tool_coverage.py
+++ b/tests/web/connector_checks/test_tool_coverage.py
@@ -57,7 +57,6 @@ _PINNED_LEGACY = frozenset(
"rocketlane",
"salesforce",
"shopify",
- "slack",
"stripe",
"whatsapp",
"xero",