Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

E2E scenarios for recent behaviours, and delete a room with a package attached

Petr Bauch6cb3eb8803a6
diff --git a/console/tooling-e2e/README.md b/console/tooling-e2e/README.md
index 5c947cf7f..7759fc615 100644
--- a/console/tooling-e2e/README.md
+++ b/console/tooling-e2e/README.md
@@ -138,6 +138,11 @@ passed or not. A teardown that leaves something behind fails the scenario.
| `operations-front-door` | The operations registry over HTTP: it lists the operations, `list_rooms` returns the room, `read_context` returns a message posted in it. | — |
| `mattermost-roundtrip` | A room on the Mattermost bridge: an addressed post from the human reaches the agent, and the agent's reply appears as its bot. | Mattermost |
| `agent-identity` | Renaming an agent from the gateway and with `!rename-agent` renames its Mattermost bot in place, and mentions follow the new name; `!delete-agent`, confirmed, deletes it. | Mattermost |
+| `room-group-paths` | The same subgroup name under two parents: a bare name both share is refused naming each path, the path picks one, and a second group of that name under one parent is refused whatever its case. | — |
+| `attachment-names` | A file whose name is not latin-1 is posted, listed by `read_context` and downloaded under its own name (`filename*` in `Content-Disposition`). | — |
+| `client-mistakes` | An outsider inviting itself into a room or reading it gets a 403, and a history timestamp that is not ISO 8601 a 400 naming the parameter; never a 500. | — |
+| `room-capture` | A room built from a document keeps the package it names, and the room captured back as a document builds a second room with the same package; both rooms delete. | — |
+| `follow-ups` | Follow the conversation: an untagged message continuing an exchange the agent just answered, or replying in its thread, reaches it; one after another agent spoke, or after the window, does not; an agent's message follows no one. | Mattermost; ephemeral only, where the stack turns the flag on with a 20-second window |
If `mattermost-roundtrip` passes and a provider-lane scenario fails, the fault
is in the agent session, not in Switch, the bridge or the harness.
@@ -601,6 +606,8 @@ than being half-used.
| `question` | Two turns: the agent asks, the human answers in the channel, the agent uses the answer. | The bot offers `red`/`green`/`blue`, then replies `green` after the answer. |
| `approval` | A tool call the session is not pre-authorised to make. | The bot posts an approval card, answering it with `<handle> 1` allows the call, the bot replies `done`. |
| `interrupt` | `!interrupt @agent-name` cancels a running local shell command after its start marker is verified. | No completion marker appears after the command’s 90-second lifetime (observed for 95 seconds). Requires `SWITCH_E2E_AGENT_DIR` on this machine. |
+| `card-answer-refused` | With the agent open only to its owner, a person who is not the owner answers its approval card, then the owner does. | The Switch admin bot tells the other person in the card's thread that the answer did not land and the command does not run; after the owner's answer the bot replies `done`. Ephemeral and local only: the other person signs up on the spot. |
+| `template-agent` | A template creates a managed agent on the controller and a Mattermost room with it in it. | The run reports the agent on the controller and listening, and it answers `SWITCH_E2E_OK_TPL` in the new channel. Requires an agent controller. |
`question` is the one a one-shot reply cannot fake: it only passes if the session
is still alive and still connected when the second message arrives.
diff --git a/console/tooling-e2e/scripts/ephemeral-stack.sh b/console/tooling-e2e/scripts/ephemeral-stack.sh
index 1d4f04b24..dd9425b4e 100755
--- a/console/tooling-e2e/scripts/ephemeral-stack.sh
+++ b/console/tooling-e2e/scripts/ephemeral-stack.sh
@@ -40,6 +40,7 @@ LANE="${SWITCH_E2E_LANE:-fast}"
# The agentic lane runs on the provider lane's setup: a real session on an
# agent controller.
NEEDS_CONTROLLER=0
+FOLLOW_WINDOW_SECONDS=20
case "$LANE" in provider|agentic) NEEDS_CONTROLLER=1 ;; esac
KEEP=0
VITEST_ARGS=()
@@ -109,6 +110,13 @@ if [ "$NEEDS_CONTROLLER" = 1 ]; then
sed -i.bak -e "s|^FEATURE_FLAGS_ENABLED=.*|FEATURE_FLAGS_ENABLED=agent_management|" \
-e "s|^CONTROLLER_TOKEN_SECRET=.*|CONTROLLER_TOKEN_SECRET=$(openssl rand -hex 32)|" "$ENV_FILE"
rm -f "$ENV_FILE.bak"
+else
+ # Follow-ups, with a window short enough for a scenario to wait out. Not on
+ # the controller lanes: there an untagged answer to a request card, or an
+ # agentic scenario's unaddressed setup message, would become a follow-up.
+ sed -i.bak -e "s|^FEATURE_FLAGS_ENABLED=.*|FEATURE_FLAGS_ENABLED=follow_the_conversation|" "$ENV_FILE"
+ rm -f "$ENV_FILE.bak"
+ printf 'FOLLOW_THE_CONVERSATION_WINDOW_SECONDS=%s\n' "$FOLLOW_WINDOW_SECONDS" >>"$ENV_FILE"
fi
chmod 600 "$ENV_FILE"
set -a; source "$ENV_FILE"; set +a
@@ -255,6 +263,8 @@ status=0
export SWITCH_E2E_SERVER_LOG="$OUT/switch-core.log"
# Screenshots of failed pages, and agentic transcripts and verdicts.
export SWITCH_E2E_OUTPUT_DIR="$OUT"
+ # The follow-up window the server was started with, for the scenario that waits it out.
+ [ "$NEEDS_CONTROLLER" = 0 ] && export SWITCH_E2E_FOLLOW_WINDOW_SECONDS="$FOLLOW_WINDOW_SECONDS"
SWITCH_E2E_LANE="$LANE" \
SWITCH_E2E_ENVIRONMENT=ephemeral \
SWITCH_E2E_STRICT=1 \
diff --git a/console/tooling-e2e/src/clients/mattermost-client.ts b/console/tooling-e2e/src/clients/mattermost-client.ts
index cfe067e6d..8b0813acb 100644
--- a/console/tooling-e2e/src/clients/mattermost-client.ts
+++ b/console/tooling-e2e/src/clients/mattermost-client.ts
@@ -102,6 +102,45 @@ export class MattermostClient {
return (text === '' ? undefined : JSON.parse(text)) as T;
}
+ /**
+ * Sign a new person up, as anyone may on a server with open sign-up, and
+ * log them in. `POST /api/v4/users`, then `POST /api/v4/users/login`, whose
+ * `Token` header is the session token the returned client posts with.
+ */
+ static async signUp(
+ env: MattermostEnv,
+ environment: Environment,
+ person: { username: string; email: string; password: string }
+ ): Promise<{ client: MattermostClient; user: MattermostUser; token: string }> {
+ const base = env.url.endsWith('/') ? env.url.slice(0, -1) : env.url;
+ const send = async (path: string, body: unknown): Promise<Response> => {
+ assertAllowed(environment, 'POST', `/api/v4${path}`);
+ const response = await fetch(`${base}/api/v4${path}`, {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
+ body: JSON.stringify(body),
+ signal: AbortSignal.timeout(30_000),
+ });
+ if (!response.ok) {
+ throw new MattermostHttpError('POST', path, response.status, await response.text());
+ }
+ return response;
+ };
+ const user = (await (await send('/users', person)).json()) as MattermostUser;
+ const login = await send('/users/login', {
+ login_id: person.username,
+ password: person.password,
+ });
+ const token = login.headers.get('Token');
+ if (token === null) throw new Error(`Mattermost logged ${person.username} in without a token.`);
+ return { client: new MattermostClient({ ...env, token }, environment), user, token };
+ }
+
+ /** `POST /api/v4/teams/{id}/members` — join a team that is open to anyone, as the token's user. */
+ async joinTeam(teamId: string, userId: string): Promise<void> {
+ await this.request('POST', `/teams/${teamId}/members`, { team_id: teamId, user_id: userId });
+ }
+
/** `GET /api/v4/users/me` — also the reachability + token-validity probe. */
async me(): Promise<MattermostUser> {
return this.request<MattermostUser>('GET', '/users/me');
diff --git a/console/tooling-e2e/src/clients/switch-client.ts b/console/tooling-e2e/src/clients/switch-client.ts
index 9e64ba2f8..7e30d3250 100644
--- a/console/tooling-e2e/src/clients/switch-client.ts
+++ b/console/tooling-e2e/src/clients/switch-client.ts
@@ -134,6 +134,59 @@ export class SwitchHttpError extends Error {
}
}
+/** An agent's addressing policy: anyone no rule matches is refused. */
+export interface AddressingPolicy {
+ rules: {
+ rooms: "*" | string[];
+ room_groups: "*" | string[];
+ users: "*" | string[];
+ agents: "*" | string[];
+ owner: boolean;
+ owner_agents: boolean;
+ platform: boolean;
+ }[];
+}
+
+/** Only the agent's owner, and the platform identities its owner claimed. */
+export const OWNER_ONLY: AddressingPolicy = {
+ rules: [
+ {
+ rooms: "*",
+ room_groups: "*",
+ users: [],
+ agents: [],
+ owner: true,
+ owner_agents: false,
+ platform: false,
+ },
+ ],
+};
+
+/** What a template run made. */
+export interface TemplateRun {
+ created_agents: {
+ slot: string;
+ agent_id: string;
+ name: string;
+ machine: { id: string; name: string };
+ listening: boolean;
+ }[];
+ room: { room_id: string; room_name: string; failed_attachments: unknown[] };
+}
+
+/** A template run the server refused, having created nothing. */
+export class TemplateRefusal extends Error {
+ readonly problems: { slot: string; code: string; message: string }[];
+ constructor(
+ cause: SwitchHttpError,
+ problems: { slot: string; code: string; message: string }[],
+ ) {
+ super(cause.message, { cause });
+ this.name = "TemplateRefusal";
+ this.problems = problems;
+ }
+}
+
/**
* The reason code of a management API refusal (`{"error": {"code": …}}`), or
* null when the error is not one.
@@ -148,6 +201,16 @@ export function managementErrorCode(error: unknown): string | null {
}
}
+/** A FastAPI error body's `detail` when it is a string, else the body as it came. */
+export function errorDetail(body: string): string {
+ try {
+ const parsed = JSON.parse(body) as { detail?: unknown };
+ return typeof parsed.detail === "string" ? parsed.detail : body;
+ } catch {
+ return body;
+ }
+}
+
function truncate(text: string, max: number): string {
return text.length > max ? `${text.slice(0, max)}…` : text;
}
@@ -530,6 +593,54 @@ export class SwitchClient {
await this.gateway("DELETE", `/rooms/${roomId}`);
}
+ /** `DELETE /gateway/room-groups/{id}` — a group's rooms and subgroups move up a level, so delete those first. */
+ async deleteRoomGroup(groupId: string): Promise<void> {
+ await this.gateway("DELETE", `/room-groups/${groupId}`);
+ }
+
+ /** `POST /gateway/packages` — a private package owned by the signed-in admin, with nothing in it. */
+ async createPackage(params: {
+ name: string;
+ description: string;
+ instructions: string;
+ }): Promise<{ id: string; name: string }> {
+ return this.gateway<{ id: string; name: string }>("POST", "/packages", params);
+ }
+
+ async deletePackage(packageId: string): Promise<void> {
+ await this.gateway("DELETE", `/packages/${packageId}`);
+ }
+
+ /** `GET /gateway/rooms/{id}/packages` — the packages attached to a room. */
+ async roomPackages(roomId: string): Promise<{ id: string; name: string }[]> {
+ return this.gateway<{ id: string; name: string }[]>(
+ "GET",
+ `/rooms/${roomId}/packages`,
+ );
+ }
+
+ /**
+ * `GET /gateway/rooms/{id}/yaml` — the room captured as the document
+ * `/rooms/from-yaml` takes, so the room can be built again from it.
+ */
+ async exportRoomYaml(roomId: string): Promise<string> {
+ assertAllowed(this.environment, "GET", `/gateway/rooms/${roomId}/yaml`);
+ const response = await fetch(`${this.apiUrl}/gateway/rooms/${roomId}/yaml`, {
+ headers: {
+ Accept: "application/x-yaml",
+ Cookie: await this.gatewayLogin(),
+ ...this.requestId(),
+ },
+ redirect: "manual",
+ signal: AbortSignal.timeout(30_000),
+ });
+ const text = await response.text();
+ if (!response.ok) {
+ throw new SwitchHttpError("GET", `/gateway/rooms/${roomId}/yaml`, response.status, text);
+ }
+ return text;
+ }
+
/**
* `DELETE /gateway/agents/by-name/{name}` — admin-authenticated teardown that,
* unlike `DELETE /agents/{id}`, does not need the agent's own API key. Used by
@@ -578,11 +689,49 @@ export class SwitchClient {
* agents and an unclaimed platform user, so its agents are opened explicitly.
*/
async openAddressing(agentId: string): Promise<void> {
+ await this.setAddressingPolicy(agentId, null);
+ }
+
+ /** `PUT /gateway/agents/{id}/addressing-policy` — who may address the agent; null lets anyone. */
+ async setAddressingPolicy(
+ agentId: string,
+ policy: AddressingPolicy | null,
+ ): Promise<void> {
await this.gateway("PUT", `/agents/${agentId}/addressing-policy`, {
- policy: null,
+ policy,
});
}
+ /**
+ * `POST /gateway/rooms/from-template-document` — run a template as the
+ * signed-in admin: create its agents, wait for each to listen, then build
+ * its room. A refusal creates nothing and throws a {@link TemplateRefusal}
+ * naming every problem.
+ */
+ async runTemplateDocument(params: {
+ template: string;
+ templateName: string;
+ }): Promise<TemplateRun> {
+ try {
+ return await this.gateway<TemplateRun>(
+ "POST",
+ "/rooms/from-template-document",
+ { template: params.template, template_name: params.templateName },
+ );
+ } catch (error) {
+ if (error instanceof SwitchHttpError && error.status === 409) {
+ const body = JSON.parse(error.body) as {
+ code?: string;
+ problems?: { slot: string; code: string; message: string }[];
+ };
+ if (body.code === "agents_not_created") {
+ throw new TemplateRefusal(error, body.problems ?? []);
+ }
+ }
+ throw error;
+ }
+ }
+
/**
* The status Switch reports for one agent in one room. `live` means a session
* is attending the room; `dormant` means an auto-session connector is watching
@@ -990,6 +1139,86 @@ export class SwitchClient {
return body.event_id;
}
+ /**
+ * `POST /agents/{id}/rooms/{room_id}/media` — post one file to a room as the
+ * agent (multipart, part `file`). Returns the media URI it was stored under.
+ */
+ async uploadMedia(
+ agent: RegisteredAgent,
+ roomId: string,
+ file: { name: string; bytes: Uint8Array<ArrayBuffer>; contentType: string },
+ ): Promise<{ eventId: string; mxc: string }> {
+ const path = `/agents/${agent.id}/rooms/${roomId}/media`;
+ assertAllowed(this.environment, "POST", path);
+ const form = new FormData();
+ form.append(
+ "file",
+ new Blob([file.bytes], { type: file.contentType }),
+ file.name,
+ );
+ const response = await fetch(`${this.apiUrl}${path}`, {
+ method: "POST",
+ headers: {
+ Accept: "application/json",
+ Authorization: `Bearer ${agent.apiKey}`,
+ ...this.requestId(),
+ },
+ body: form,
+ redirect: "manual",
+ signal: AbortSignal.timeout(30_000),
+ });
+ const text = await response.text();
+ if (!response.ok) throw new SwitchHttpError("POST", path, response.status, text);
+ const body = JSON.parse(text) as { event_id: string; mxc: string };
+ return { eventId: body.event_id, mxc: body.mxc };
+ }
+
+ /**
+ * `GET /agents/{id}/rooms/{room_id}/media?mxc=` — an attachment's bytes, and
+ * the headers that name it, as the agent's session host downloads it.
+ */
+ async downloadMedia(
+ agent: RegisteredAgent,
+ roomId: string,
+ mxc: string,
+ ): Promise<{ bytes: Uint8Array; contentType: string; contentDisposition: string }> {
+ const path = `/agents/${agent.id}/rooms/${roomId}/media?mxc=${encodeURIComponent(mxc)}`;
+ assertAllowed(this.environment, "GET", path);
+ const response = await fetch(`${this.apiUrl}${path}`, {
+ headers: { Authorization: `Bearer ${agent.apiKey}`, ...this.requestId() },
+ redirect: "manual",
+ signal: AbortSignal.timeout(30_000),
+ });
+ if (!response.ok) {
+ throw new SwitchHttpError("GET", path, response.status, await response.text());
+ }
+ return {
+ bytes: new Uint8Array(await response.arrayBuffer()),
+ contentType: response.headers.get("content-type") ?? "",
+ contentDisposition: response.headers.get("content-disposition") ?? "",
+ };
+ }
+
+ /**
+ * Any agent-bridge request as the agent, answered with its status and body
+ * whatever the status: for the checks that a client's mistake is answered
+ * with a 4xx that says what was wrong.
+ */
+ async agentAttempt(
+ agent: RegisteredAgent,
+ method: "GET" | "POST",
+ path: string,
+ body: unknown,
+ ): Promise<{ status: number; detail: string }> {
+ try {
+ await this.request(method, path, { token: agent.apiKey, body: body ?? undefined });
+ return { status: 200, detail: "" };
+ } catch (error) {
+ if (!(error instanceof SwitchHttpError)) throw error;
+ return { status: error.status, detail: errorDetail(error.body) };
+ }
+ }
+
// ── agent bridge: the operations front door ───────────────────────────────
/**
diff --git a/console/tooling-e2e/src/core/scenario.ts b/console/tooling-e2e/src/core/scenario.ts
index e33ee7fa0..08ad53670 100644
--- a/console/tooling-e2e/src/core/scenario.ts
+++ b/console/tooling-e2e/src/core/scenario.ts
@@ -39,8 +39,14 @@ export interface ScenarioHeader {
* `agent-dir`: the provider session's working directory is on this machine.
* `gateway-ui`: the address of a gateway UI served against this Switch server.
* `agentic-models`: the driver and judge models, and the spending limit.
+ * `controller`: an agent controller that runs the agents Switch manages.
*/
-export type Prerequisite = 'mattermost' | 'agent-dir' | 'gateway-ui' | 'agentic-models';
+export type Prerequisite =
+ | 'mattermost'
+ | 'agent-dir'
+ | 'gateway-ui'
+ | 'agentic-models'
+ | 'controller';
const OPERATION = /^[a-z][a-z0-9_]*$/;
const ROUTE = /^(GET|POST|PUT|PATCH|DELETE) \/\S*$/;
@@ -136,6 +142,8 @@ function satisfied(prerequisite: Prerequisite, env: HarnessEnv): boolean {
env.agentic.judgeModel !== null &&
env.agentic.budgetUsd !== null
);
+ case 'controller':
+ return env.provider.controllerId !== null;
}
}
@@ -149,5 +157,7 @@ function describePrerequisite(prerequisite: Prerequisite): string {
return 'a gateway UI (SWITCH_GATEWAY_UI_URL)';
case 'agentic-models':
return 'the driver and judge models and a spending limit (SWITCH_E2E_DRIVER_MODEL, SWITCH_E2E_JUDGE_MODEL, SWITCH_E2E_AGENTIC_BUDGET_USD)';
+ case 'controller':
+ return 'an agent controller (SWITCH_E2E_CONTROLLER_ID)';
}
}
diff --git a/console/tooling-e2e/src/fast/attachments.ts b/console/tooling-e2e/src/fast/attachments.ts
new file mode 100644
index 000000000..0c5b2a8fa
--- /dev/null
+++ b/console/tooling-e2e/src/fast/attachments.ts
@@ -0,0 +1,67 @@
+import { expect } from 'vitest';
+import { sleep } from '../clients/switch-client.ts';
+import { WRITABLE, type FastScenario } from './scenario.ts';
+
+// An em dash and the narrow no-break space a macOS screenshot name carries:
+// neither is latin-1, the only charset an HTTP header carries as is.
+const FILE_NAME = 'Device profile — SSH host notes.txt';
+const HISTORY_DEADLINE_MS = 30_000;
+
+/**
+ * An attachment whose name is not latin-1 is posted, listed and downloaded
+ * under its own name. Its download put the name straight into
+ * `Content-Disposition`, which failed with a 500, so the agent never got the
+ * file.
+ */
+export const attachmentNames: FastScenario = {
+ header: {
+ id: 'attachment-names',
+ covers: [
+ 'POST /agents/{agent_id}/rooms/{room_id}/media',
+ 'GET /agents/{agent_id}/rooms/{room_id}/media',
+ 'read_context',
+ ],
+ lane: 'fast',
+ environments: WRITABLE,
+ owner: 'platform',
+ requires: [],
+ },
+ async run({ fixtures, names, switchClient }) {
+ const sender = await fixtures.agent('up');
+ const reader = await fixtures.agent('down');
+ const room = await fixtures.internalRoom('room', [sender, reader]);
+
+ const bytes = new TextEncoder().encode(`${names.marker('file')}\nnaïve café ✓\n`);
+ const posted = await switchClient.uploadMedia(sender, room.id, {
+ name: FILE_NAME,
+ bytes,
+ contentType: 'text/plain',
+ });
+
+ interface Entry {
+ attachments: { filename: string; mxc: string }[];
+ }
+ let listed: { filename: string; mxc: string } | undefined;
+ const until = Date.now() + HISTORY_DEADLINE_MS;
+ while (listed === undefined && Date.now() < until) {
+ const context = await switchClient.callOperation<{
+ threads: { root: Entry; replies: Entry[] }[];
+ }>(reader, 'read_context', { room_id: room.id, limit: 20 });
+ listed = context.threads
+ .flatMap((thread) => [thread.root, ...thread.replies])
+ .flatMap((entry) => entry.attachments)
+ .find((attachment) => attachment.mxc === posted.mxc);
+ if (listed === undefined) await sleep(1_000);
+ }
+ expect(listed, `the room's history never listed ${posted.mxc}`).toBeDefined();
+ expect(listed!.filename).toBe(FILE_NAME);
+
+ const download = await switchClient.downloadMedia(reader, room.id, posted.mxc);
+ expect(Buffer.from(download.bytes).equals(Buffer.from(bytes))).toBe(true);
+ expect(download.contentType).toMatch(/^text\/plain/);
+ // RFC 6266: the name itself, percent-encoded as UTF-8, in `filename*`.
+ expect(download.contentDisposition).toContain(
+ `filename*=UTF-8''${encodeURIComponent(FILE_NAME)}`
+ );
+ },
+};
diff --git a/console/tooling-e2e/src/fast/client-mistakes.ts b/console/tooling-e2e/src/fast/client-mistakes.ts
new file mode 100644
index 000000000..028ef1494
--- /dev/null
+++ b/console/tooling-e2e/src/fast/client-mistakes.ts
@@ -0,0 +1,63 @@
+import { expect } from 'vitest';
+import { WRITABLE, type FastScenario } from './scenario.ts';
+
+/**
+ * An agent's mistake is answered with a 4xx that says what was wrong, never a
+ * 500: an HTTP agent reads a 500 as the server broken and may retry a request
+ * that can never succeed. The server-error check fails the scenario too, if
+ * any of these is logged as an error.
+ */
+export const clientMistakes: FastScenario = {
+ header: {
+ id: 'client-mistakes',
+ covers: [
+ 'POST /agents/{agent_id}/moderation/rooms/{room_id}/invite',
+ 'GET /agents/{agent_id}/moderation/rooms/{room_id}',
+ 'GET /agents/{agent_id}/rooms/{room_id}/history',
+ ],
+ lane: 'fast',
+ environments: WRITABLE,
+ owner: 'platform',
+ requires: [],
+ },
+ async run({ fixtures, switchClient }) {
+ const member = await fixtures.agent('in');
+ const outsider = await fixtures.agent('out');
+ const room = await fixtures.internalRoom('room', [member]);
+
+ const invite = await switchClient.agentAttempt(
+ outsider,
+ 'POST',
+ `/agents/${outsider.id}/moderation/rooms/${room.id}/invite`,
+ { agent_name: outsider.name }
+ );
+ expect(invite, 'an outsider inviting itself into a room').toEqual({
+ status: 403,
+ detail: "Not authorized to change this room's membership",
+ });
+
+ const lookup = await switchClient.agentAttempt(
+ outsider,
+ 'GET',
+ `/agents/${outsider.id}/moderation/rooms/${room.id}`,
+ null
+ );
+ expect(lookup, 'an outsider reading a room').toEqual({
+ status: 403,
+ detail: 'Agent is not a member of this room',
+ });
+
+ for (const parameter of ['since', 'before']) {
+ const history = await switchClient.agentAttempt(
+ member,
+ 'GET',
+ `/agents/${member.id}/rooms/${room.id}/history?${parameter}=yesterday`,
+ null
+ );
+ expect(history, `history with ${parameter}=yesterday`).toEqual({
+ status: 400,
+ detail: `'${parameter}' is not an ISO 8601 timestamp: 'yesterday'`,
+ });
+ }
+ },
+};
diff --git a/console/tooling-e2e/src/fast/follow-ups.ts b/console/tooling-e2e/src/fast/follow-ups.ts
new file mode 100644
index 000000000..1c06f6c0a
--- /dev/null
+++ b/console/tooling-e2e/src/fast/follow-ups.ts
@@ -0,0 +1,178 @@
+import { expect } from 'vitest';
+import type { MattermostClient, MattermostPost } from '../clients/mattermost-client.ts';
+import {
+ eventText,
+ sleep,
+ type AgentEvent,
+ type RegisteredAgent,
+ type SwitchClient,
+} from '../clients/switch-client.ts';
+import type { BridgedRoom } from '../fixtures/fixtures.ts';
+import type { FastScenario } from './scenario.ts';
+
+const FLAG = 'follow_the_conversation';
+const DEADLINE_MS = 60_000;
+
+/** The window the server was started with: set by the ephemeral stack, which starts it. */
+function followWindowMs(): number {
+ const raw = process.env.SWITCH_E2E_FOLLOW_WINDOW_SECONDS;
+ const seconds = Number(raw);
+ if (raw === undefined || !Number.isFinite(seconds) || seconds <= 0) {
+ throw new Error(
+ `SWITCH_E2E_FOLLOW_WINDOW_SECONDS must name the server's FOLLOW_THE_CONVERSATION_WINDOW_SECONDS (got '${raw}').`
+ );
+ }
+ return seconds * 1000;
+}
+
+/**
+ * Follow the conversation: a person's untagged message reaches the agent they
+ * are talking to. It does when it continues an exchange the agent has just
+ * answered, or replies in a thread the agent spoke in last. It does not when
+ * another agent spoke in between, or when the exchange has gone quiet for longer
+ * than the window. A message from an agent never follows anyone.
+ *
+ * Each "does not" is shown by an `@agent` message posted after it: the agent's
+ * notifications arrive in order, so if the untagged one had reached it, it
+ * would have arrived first.
+ *
+ * Ephemeral only: the flag is server-wide, and the window has to be short
+ * enough to wait out.
+ */
+export const followUps: FastScenario = {
+ header: {
+ id: 'follow-ups',
+ covers: ['GET /agents/{agent_id}/notifications', 'POST /agents/{agent_id}/message'],
+ lane: 'fast',
+ environments: ['ephemeral'],
+ owner: 'platform',
+ requires: ['mattermost'],
+ },
+ async run({ fixtures, names, switchClient, mattermost, log }) {
+ const flags = await switchClient.enabledFeatureFlags();
+ if (!flags.has(FLAG)) {
+ throw new Error(
+ `The server does not have '${FLAG}' on; the ephemeral stack starts it with it.`
+ );
+ }
+ const windowMs = followWindowMs();
+ const a = await fixtures.agent('fa');
+ const b = await fixtures.agent('fb');
+ const bridged = await fixtures.mattermostRoom('ftc', [a, b]);
+ const talk = new Exchange(switchClient, mattermost!, bridged);
+
+ // A channel follow-up: the person goes on, untagged, after the agent answered.
+ const q1 = names.marker('q1');
+ await talk.human(`@${a.name} first question ${q1}`, null);
+ await talk.notified(a, q1, 'a mention', []);
+ const answer = await talk.answer(a, `first answer ${names.marker('a1')}`);
+ const q2 = names.marker('q2');
+ await talk.human(`and a second one ${q2}`, null);
+ const followed = await talk.notified(a, q2, 'an untagged channel follow-up', []);
+ expect(followed.payload?.addressed).toBe(true);
+ expect(followed.payload?.thread_id ?? null).toBeNull();
+
+ // A thread follow-up: a reply under the agent's answer, in a thread it spoke in last.
+ const q3 = names.marker('q3');
+ await talk.human(`why so ${q3}`, answer.post.id);
+ const threaded = await talk.notified(a, q3, 'an untagged reply in its thread', []);
+ expect(threaded.payload?.thread_id).toBe(answer.eventId);
+
+ // Another agent speaks: its message follows no one, and ends the person's exchange.
+ const fyi = names.marker('fyi');
+ await talk.answer(b, `for your information ${fyi}`);
+ const q4 = names.marker('q4');
+ await talk.human(`and a third one ${q4}`, null);
+ const q5 = names.marker('q5');
+ await talk.human(`@${a.name} back to you ${q5}`, null);
+ await talk.notified(a, q5, 'a mention', [fyi, q4]);
+ const history = await switchClient.roomHistory(a, bridged.room.id, 20);
+ expect(history.map((event) => event.body).some((body) => body.includes(fyi))).toBe(true);
+
+ // The exchange goes quiet for longer than the window.
+ await talk.answer(a, `answered ${names.marker('a5')}`);
+ log(`waiting out the ${windowMs / 1000}s follow-up window`);
+ await sleep(windowMs + 5_000);
+ const q6 = names.marker('q6');
+ await talk.human(`one more thing ${q6}`, null);
+ const q7 = names.marker('q7');
+ await talk.human(`@${a.name} last one ${q7}`, null);
+ await talk.notified(a, q7, 'a mention', [q6]);
+
+ // b was never addressed, and is not followed: nothing above reached it.
+ const strays = (await switchClient.pollNotifications(b, 2)).filter((event) =>
+ [q1, q2, q3, q4, q5, q6, q7].some((marker) => eventText(event).includes(marker))
+ );
+ expect(strays.map(eventText), 'messages that reached the agent nobody talked to').toEqual([]);
+ },
+};
+
+/** One person and two agents talking in a bridged room. */
+class Exchange {
+ private readonly switchClient: SwitchClient;
+ private readonly mattermost: MattermostClient;
+ private readonly bridged: BridgedRoom;
+
+ constructor(switchClient: SwitchClient, mattermost: MattermostClient, bridged: BridgedRoom) {
+ this.switchClient = switchClient;
+ this.mattermost = mattermost;
+ this.bridged = bridged;
+ }
+
+ /** Post as the person: top-level, or under `rootId` in a thread. */
+ async human(message: string, rootId: string | null): Promise<void> {
+ await this.mattermost.post({
+ channelId: this.bridged.channel.id,
+ message,
+ ...(rootId === null ? {} : { rootId }),
+ });
+ }
+
+ /** Post top-level as the agent, and wait for the post to reach the channel. */
+ async answer(
+ agent: RegisteredAgent,
+ text: string
+ ): Promise<{ eventId: string; post: MattermostPost }> {
+ const since = Date.now() - 1;
+ const eventId = await this.switchClient.sendMessage(agent, this.bridged.room.id, text);
+ const { match, transcript } = await this.mattermost.waitForPost({
+ channelId: this.bridged.channel.id,
+ fromUserId: this.bridged.bots.get(agent.id)!.id,
+ predicate: (post) => post.message.includes(text),
+ sinceMs: since,
+ deadlineMs: DEADLINE_MS,
+ pollIntervalMs: 500,
+ });
+ if (match === null) {
+ throw new Error(
+ `${agent.name}'s '${text}' never reached the channel; it said: ` +
+ transcript.map((post) => post.message).join(' | ')
+ );
+ }
+ return { eventId, post: match };
+ }
+
+ /**
+ * Wait for `marker` to reach the agent, and fail if any of `notBefore` did
+ * first.
+ */
+ async notified(
+ agent: RegisteredAgent,
+ marker: string,
+ what: string,
+ notBefore: string[]
+ ): Promise<AgentEvent> {
+ const { match, seen } = await this.switchClient.waitForNotification(
+ agent,
+ (event) => eventText(event).includes(marker),
+ DEADLINE_MS
+ );
+ expect(
+ match,
+ `${what} (${marker}) never reached ${agent.name}; saw ${seen.map(eventText).join(' | ')}`
+ ).not.toBeNull();
+ const early = seen.filter((event) => notBefore.some((text) => eventText(event).includes(text)));
+ expect(early.map(eventText), `reached ${agent.name} though it should not have`).toEqual([]);
+ return match!;
+ }
+}
diff --git a/console/tooling-e2e/src/fast/index.ts b/console/tooling-e2e/src/fast/index.ts
index 676054e08..acab1dd96 100644
--- a/console/tooling-e2e/src/fast/index.ts
+++ b/console/tooling-e2e/src/fast/index.ts
@@ -1,7 +1,12 @@
import { agentIdentity } from "./agent-identity.ts";
+import { attachmentNames } from "./attachments.ts";
+import { clientMistakes } from "./client-mistakes.ts";
+import { followUps } from "./follow-ups.ts";
import { addressedDelivery, unaddressedHistory } from "./agent-messaging.ts";
import { mattermostRoundtrip } from "./mattermost-roundtrip.ts";
import { operationsFrontDoor } from "./operations.ts";
+import { roomCapture } from "./room-capture.ts";
+import { roomGroupPaths } from "./room-groups.ts";
import type { FastScenario } from "./scenario.ts";
/** Every fast-lane scenario. A scenario not listed here does not run. */
@@ -11,4 +16,9 @@ export const FAST_SCENARIOS: readonly FastScenario[] = [
operationsFrontDoor,
mattermostRoundtrip,
agentIdentity,
+ roomGroupPaths,
+ attachmentNames,
+ clientMistakes,
+ roomCapture,
+ followUps,
];
diff --git a/console/tooling-e2e/src/fast/room-capture.ts b/console/tooling-e2e/src/fast/room-capture.ts
new file mode 100644
index 000000000..c715908a2
--- /dev/null
+++ b/console/tooling-e2e/src/fast/room-capture.ts
@@ -0,0 +1,67 @@
+import { expect } from 'vitest';
+import { parse, stringify } from 'yaml';
+import { WRITABLE, type FastScenario } from './scenario.ts';
+
+/**
+ * A room built from a document keeps the packages the document names, and
+ * the room captured back as a document builds a second room with the same
+ * packages: what a template made from a live room relies on.
+ */
+export const roomCapture: FastScenario = {
+ header: {
+ id: 'room-capture',
+ covers: [
+ 'POST /gateway/packages',
+ 'DELETE /gateway/packages/{package_id}',
+ 'GET /gateway/rooms/{room_id}/packages',
+ 'GET /gateway/rooms/{room_id}/yaml',
+ 'POST /gateway/rooms/from-yaml',
+ ],
+ lane: 'fast',
+ environments: WRITABLE,
+ owner: 'platform',
+ requires: [],
+ },
+ async run({ fixtures, names, switchClient }) {
+ const agent = await fixtures.agent('cap');
+ const about = `Switch end-to-end harness (run ${names.runId}). Safe to delete.`;
+ const pkg = await switchClient.createPackage({
+ name: names.name('pkg'),
+ description: about,
+ instructions: names.marker('pkg'),
+ });
+ fixtures.defer(`package ${pkg.name} (${pkg.id})`, () => switchClient.deletePackage(pkg.id));
+
+ const build = async (document: string) => {
+ const room = await switchClient.createRoomFromYaml(document);
+ fixtures.defer(`room ${room.name} (${room.id})`, () => switchClient.deleteRoom(room.id));
+ return room;
+ };
+
+ const original = await build(
+ stringify({
+ room: {
+ name: names.name('orig'),
+ description: about,
+ agents: [agent.name],
+ packages: [{ name: pkg.name }],
+ },
+ })
+ );
+ expect(await switchClient.roomPackages(original.id)).toMatchObject([
+ { id: pkg.id, name: pkg.name },
+ ]);
+
+ const captured = parse(await switchClient.exportRoomYaml(original.id)) as {
+ room: Record<string, unknown>;
+ };
+ expect(captured.room.packages).toEqual([{ id: pkg.id }]);
+ expect(captured.room.agents).toEqual([agent.name]);
+
+ const copy = await build(stringify({ room: { ...captured.room, name: names.name('copy') } }));
+ expect(copy.id).not.toBe(original.id);
+ expect(await switchClient.roomPackages(copy.id)).toMatchObject([
+ { id: pkg.id, name: pkg.name },
+ ]);
+ },
+};
diff --git a/console/tooling-e2e/src/fast/room-groups.ts b/console/tooling-e2e/src/fast/room-groups.ts
new file mode 100644
index 000000000..20d00217a
--- /dev/null
+++ b/console/tooling-e2e/src/fast/room-groups.ts
@@ -0,0 +1,102 @@
+import { expect } from 'vitest';
+import { SwitchHttpError, errorDetail } from '../clients/switch-client.ts';
+import { WRITABLE, type FastScenario } from './scenario.ts';
+
+interface RoomGroup {
+ id: string;
+ name: string;
+ parent_group_id: string | null;
+ path: string;
+ member_rooms: { id: string; name: string }[];
+}
+
+/**
+ * Room groups named by path: the same subgroup name under two parents is
+ * allowed, a bare name that two groups share is refused naming the path of
+ * each, the path picks one, and a second group of that name under the same
+ * parent is refused whatever its case.
+ */
+export const roomGroupPaths: FastScenario = {
+ header: {
+ id: 'room-group-paths',
+ covers: [
+ 'create_room_group',
+ 'get_room_group_detail',
+ 'create_room',
+ 'DELETE /gateway/room-groups/{group_id}',
+ ],
+ lane: 'fast',
+ environments: WRITABLE,
+ owner: 'platform',
+ requires: [],
+ },
+ async run({ fixtures, names, switchClient, log }) {
+ const agent = await fixtures.agent('grp');
+ const call = <T>(operation: string, args: Record<string, unknown>): Promise<T> =>
+ switchClient.callOperation<T>(agent, operation, args);
+ const group = async (name: string, parent: string | null): Promise<RoomGroup> => {
+ const made = await call<RoomGroup>('create_room_group', {
+ name,
+ description: `Switch end-to-end harness (run ${names.runId}). Safe to delete.`,
+ ...(parent === null ? {} : { parent_group_name: parent }),
+ });
+ fixtures.defer(`room group ${made.path} (${made.id})`, () =>
+ switchClient.deleteRoomGroup(made.id)
+ );
+ return made;
+ };
+ const refusal = async (operation: string, args: Record<string, unknown>): Promise<string> => {
+ try {
+ await call(operation, args);
+ } catch (error) {
+ if (error instanceof SwitchHttpError && error.status === 400)
+ return errorDetail(error.body);
+ throw error;
+ }
+ throw new Error(`${operation} ${JSON.stringify(args)} was accepted`);
+ };
+
+ const parents = [await group(names.name('ga'), null), await group(names.name('gb'), null)];
+ // One name, used under both parents: what made bare names ambiguous.
+ const child = names.name('inc');
+ const children = [];
+ for (const parent of parents) {
+ const made = await group(child, parent.name);
+ expect(made.parent_group_id).toBe(parent.id);
+ // Shown spaced ("a / b"); a reference may be written either way.
+ expect(made.path.split('/').map((segment) => segment.trim())).toEqual([parent.name, child]);
+ children.push(made);
+ }
+ const [first, second] = children as [RoomGroup, RoomGroup];
+ const [firstPath, secondPath] = parents.map((parent) => `${parent.name}/${child}`) as [
+ string,
+ string,
+ ];
+
+ const duplicate = await refusal('create_room_group', {
+ name: child.toUpperCase(),
+ parent_group_name: parents[0]!.name,
+ });
+ expect(duplicate).toContain('already exists');
+
+ const room = names.name('room');
+ const roomArgs = {
+ name: room,
+ description: `Switch end-to-end harness (run ${names.runId}). Safe to delete.`,
+ agent_names: [agent.name],
+ internal_only: true,
+ };
+ const ambiguous = await refusal('create_room', { ...roomArgs, group_name: child });
+ expect(ambiguous).toContain(`Ambiguous room group '${child}'`);
+ expect(ambiguous).toContain(firstPath);
+ expect(ambiguous).toContain(secondPath);
+ log(`a bare '${child}' was refused: ${ambiguous}`);
+
+ const made = await call<{ id: string }>('create_room', { ...roomArgs, group_name: secondPath });
+ fixtures.defer(`room ${room} (${made.id})`, () => switchClient.deleteRoom(made.id));
+ const detail = await call<RoomGroup>('get_room_group_detail', { group_id: second.id });
+ expect(detail.member_rooms).toEqual([{ id: made.id, name: room }]);
+ const other = await call<RoomGroup>('get_room_group_detail', { group_id: first.id });
+ expect(other.member_rooms).toEqual([]);
+ },
+};
diff --git a/console/tooling-e2e/src/provider/harness.ts b/console/tooling-e2e/src/provider/harness.ts
index 0e5e81593..72d8494d4 100644
--- a/console/tooling-e2e/src/provider/harness.ts
+++ b/console/tooling-e2e/src/provider/harness.ts
@@ -74,17 +74,17 @@ const ROLE_BY_AGENT_TYPE: Record<HarnessAgentType, string> = {
// sentence: told merely to answer briefly, the agent declined the `interrupt`
// scenario's task on the grounds that 200 messages would spam the room — a
// refusal the scenario reads as a session that never started.
-const AGENT_INSTRUCTIONS =
+export const AGENT_INSTRUCTIONS =
'You are a Switch end-to-end test agent. Do exactly what you are asked in the ' +
'room, and post your answer back to the room you were addressed in. Never ' +
'decline or shorten a task on the grounds that it is repetitive or long: the ' +
'requests you get here are deliberate tests of exactly that.';
/** Cheap on purpose: a run spends a real subscription. */
-const CLAUDE_MODEL = 'claude-sonnet-5';
+export const CLAUDE_MODEL = 'claude-sonnet-5';
/** The management API's provider id for each harness agent type. */
-const PROVIDER_BY_AGENT_TYPE: Record<HarnessAgentType, string> = {
+export const PROVIDER_BY_AGENT_TYPE: Record<HarnessAgentType, string> = {
opencode: 'opencode',
'claude-code': 'claude',
codex: 'codex',
@@ -93,10 +93,10 @@ const PROVIDER_BY_AGENT_TYPE: Record<HarnessAgentType, string> = {
};
/** How long to keep asking a controller to take the agent while it comes up. */
-const PLACEMENT_DEADLINE_MS = 120_000;
+export const PLACEMENT_DEADLINE_MS = 120_000;
/** The placement refusals that clear once the controller is up and signed in. */
-const NOT_READY_CODES = new Set([
+export const NOT_READY_CODES = new Set([
'controller_offline',
'provider_not_installed',
'provider_login_missing',
diff --git a/console/tooling-e2e/src/provider/provider.e2e.test.ts b/console/tooling-e2e/src/provider/provider.e2e.test.ts
index 8e22bb8f7..dc2489a57 100644
--- a/console/tooling-e2e/src/provider/provider.e2e.test.ts
+++ b/console/tooling-e2e/src/provider/provider.e2e.test.ts
@@ -97,10 +97,11 @@ describe('provider lane', () => {
}
const h = harness!;
const log = (message: string): void => console.log(`[${scenario.header.id}] ${message}`);
- const observer = watch.begin(h.switch, scopes.scope().tag, scenario.header.id);
+ const names = scopes.scope();
+ const observer = watch.begin(h.switch, names.tag, scenario.header.id);
let result: ScenarioResult;
try {
- result = await scenario.run(h);
+ result = await scenario.run(h, names);
} finally {
observer.stop();
}
diff --git a/console/tooling-e2e/src/provider/scenarios.ts b/console/tooling-e2e/src/provider/scenarios.ts
index fa73f4e96..4dfb0a2a0 100644
--- a/console/tooling-e2e/src/provider/scenarios.ts
+++ b/console/tooling-e2e/src/provider/scenarios.ts
@@ -1,8 +1,20 @@
+import { randomBytes } from 'node:crypto';
import { readFile } from 'node:fs/promises';
import path from 'node:path';
-import { sleep, type MattermostPost } from '../clients/mattermost-client.ts';
+import { stringify } from 'yaml';
+import { MattermostClient, sleep, type MattermostPost } from '../clients/mattermost-client.ts';
+import { OWNER_ONLY, TemplateRefusal, type TemplateRun } from '../clients/switch-client.ts';
+import type { NameScope } from '../core/run.ts';
import type { ScenarioHeader } from '../core/scenario.ts';
-import type { Harness } from './harness.ts';
+import { IssuedSecrets } from '../core/secrets.ts';
+import {
+ AGENT_INSTRUCTIONS,
+ CLAUDE_MODEL,
+ NOT_READY_CODES,
+ PLACEMENT_DEADLINE_MS,
+ PROVIDER_BY_AGENT_TYPE,
+ type Harness,
+} from './harness.ts';
/**
* The scenarios. Each one drives the full path
@@ -26,7 +38,8 @@ export interface ScenarioResult {
durationMs: number;
}
-export type Scenario = (harness: Harness) => Promise<ScenarioResult>;
+/** `names` names whatever a scenario makes beyond the shared agent and channel. */
+export type Scenario = (harness: Harness, names: NameScope) => Promise<ScenarioResult>;
/** How long a scenario waits for the agent to say something. */
const REPLY_DEADLINE_MS = Number(process.env.SWITCH_E2E_REPLY_TIMEOUT_MS ?? 5 * 60_000);
@@ -204,8 +217,10 @@ async function ask(harness: Harness, text: string): Promise<number> {
/** Wait for a bot post matching `predicate`, or throw with the transcript. */
async function expectBotPost(
- harness: Harness,
+ mattermost: MattermostClient,
params: {
+ /** The channel to watch, and the bot whose posts count. */
+ where: { channelId: string; botId: string };
since: number;
predicate: (post: MattermostPost) => boolean;
describe: string;
@@ -213,9 +228,9 @@ async function expectBotPost(
record: (posts: MattermostPost[]) => void;
}
): Promise<MattermostPost> {
- const { match, transcript } = await harness.mattermost.waitForPost({
- channelId: harness.channel.id,
- fromUserId: harness.bot.id,
+ const { match, transcript } = await mattermost.waitForPost({
+ channelId: params.where.channelId,
+ fromUserId: params.where.botId,
// Neither a no-session notice nor Switch's activity commentary is ever a
// match, however permissive the caller's predicate — both are Switch
// talking on the bot's account, not the agent.
@@ -234,6 +249,11 @@ async function expectBotPost(
return match;
}
+/** The shared channel, and the shared agent's bot. */
+function sharedRoom(harness: Harness): { channelId: string; botId: string } {
+ return { channelId: harness.channel.id, botId: harness.bot.id };
+}
+
function summarise(posts: MattermostPost[]): string {
if (posts.length === 0) return '(nothing at all — no session ever replied)';
return posts.map((post) => `«${post.message.replace(/\s+/g, ' ').slice(0, 160)}»`).join(' | ');
@@ -255,7 +275,8 @@ function lower(post: MattermostPost): string {
export const greet: Scenario = (harness) =>
scenario('greet', async (record) => {
const since = await ask(harness, `reply with exactly ${OK_MARKER} and nothing else`);
- const match = await expectBotPost(harness, {
+ const match = await expectBotPost(harness.mattermost, {
+ where: sharedRoom(harness),
since,
record,
predicate: (post) => post.message.includes(OK_MARKER),
@@ -291,7 +312,8 @@ export const question: Scenario = (harness) =>
`exactly the options ${CHOICES.join(', ')}. Then reply with the single word I chose.`
);
- const asked = await expectBotPost(harness, {
+ const asked = await expectBotPost(harness.mattermost, {
+ where: sharedRoom(harness),
since,
record,
// A native question is relayed as a request card numbering the options;
@@ -312,7 +334,8 @@ export const question: Scenario = (harness) =>
const answeredAt = prose
? await ask(harness, CHOSEN)
: await answerCard(harness, asked, chosenOption(asked));
- const answer = await expectBotPost(harness, {
+ const answer = await expectBotPost(harness.mattermost, {
+ where: sharedRoom(harness),
since: answeredAt,
record,
predicate: (post) =>
@@ -348,7 +371,8 @@ export const approval: Scenario = (harness) =>
`run the shell command \`echo ${OK_MARKER} > ${file}\` and then reply with the single word done`
);
- const prompt = await expectBotPost(harness, {
+ const prompt = await expectBotPost(harness.mattermost, {
+ where: sharedRoom(harness),
since,
record,
predicate: (post) =>
@@ -358,7 +382,8 @@ export const approval: Scenario = (harness) =>
});
const approvedAt = await answerCard(harness, prompt, 1);
- const done = await expectBotPost(harness, {
+ const done = await expectBotPost(harness.mattermost, {
+ where: sharedRoom(harness),
since: approvedAt,
record,
predicate: (post) => lower(post).includes('done'),
@@ -398,7 +423,8 @@ export const interrupt: Scenario = (harness) =>
harness,
`Run this exact shell command for a bounded cancellation test: \`${command}\`. Wait for completion before replying. Do not create these files with another tool.`
);
- const permission = await expectBotPost(harness, {
+ const permission = await expectBotPost(harness.mattermost, {
+ where: sharedRoom(harness),
since,
record,
predicate: (post) =>
@@ -432,6 +458,228 @@ export const interrupt: Scenario = (harness) =>
return 'command start confirmed; completion marker absent after 95s';
});
+// ── card answered by someone who may not ─────────────────────────────────────
+
+/** The bot Switch posts its own notices as, on Mattermost. */
+const SWITCH_ADMIN = 'switch-admin';
+
+/**
+ * Answering a request card takes the same permission as addressing the agent.
+ * With the agent open only to its owner, a person who is not the owner answers
+ * the card and is told the answer did not land, and the command does not run;
+ * the owner answers it and the command runs.
+ *
+ * The second person signs up on the spot, so this runs only where Mattermost
+ * lets anyone sign up: the throwaway stack and a local one. On both, the person
+ * the harness posts as is the identity the admin who owns the agent claimed.
+ */
+export const cardAnswerRefused: Scenario = (harness, names) =>
+ scenario('card-answer-refused', async (record) => {
+ const mattermostEnv = harness.env.mattermost;
+ if (mattermostEnv === null) throw new Error('This scenario needs Mattermost.');
+ const secrets = new IssuedSecrets(harness.env.secretsFile);
+ const username = names.name('other');
+ const password = `E2e-${randomBytes(18).toString('base64url')}`;
+ secrets.record(password);
+ const other = await MattermostClient.signUp(mattermostEnv, harness.env.environment, {
+ username,
+ email: `${username}@example.com`,
+ password,
+ });
+ secrets.record(other.token);
+ const team = await harness.mattermost.findTeam(mattermostEnv.team);
+ await other.client.joinTeam(team.id, other.user.id);
+ await harness.mattermost.addUserToChannel(harness.channel.id, other.user.id);
+ const admin = await harness.mattermost.findBotUser(SWITCH_ADMIN);
+ if (admin === null) throw new Error(`Mattermost has no '${SWITCH_ADMIN}' account.`);
+
+ await harness.switch.setAddressingPolicy(harness.agent.id, OWNER_ONLY);
+ try {
+ const file = `refused-${harness.runId}.txt`;
+ const since = await ask(
+ harness,
+ `run the shell command \`echo ${OK_MARKER} > ${file}\` and then reply with the single word done`
+ );
+ const card = await expectBotPost(harness.mattermost, {
+ where: sharedRoom(harness),
+ since,
+ record,
+ predicate: (post) =>
+ requestHandle(post.message) !== null &&
+ APPROVAL_HINTS.some((hint) => lower(post).includes(hint)),
+ describe: 'an approval card for the shell command',
+ });
+ const handle = requestHandle(card.message)!;
+ const thread = card.root_id === '' ? card.id : card.root_id;
+
+ const refusedAt = Date.now() - 1;
+ await other.client.post({ channelId: harness.channel.id, message: `${handle} 1` });
+ const { match: notice, transcript } = await harness.mattermost.waitForPost({
+ channelId: harness.channel.id,
+ fromUserId: admin.id,
+ predicate: (post) =>
+ post.root_id === thread &&
+ post.message.includes(`answer to ${handle} did not land`) &&
+ post.message.includes('may not address this agent'),
+ sinceMs: refusedAt,
+ deadlineMs: 60_000,
+ });
+ record(transcript);
+ if (notice === null) {
+ throw new Error(
+ `${username}'s answer to ${handle} was not refused in the card's thread. Channel said: ${summarise(transcript)}`
+ );
+ }
+ const ranEarly = (
+ await harness.mattermost.postsFrom(harness.channel.id, harness.bot.id, refusedAt)
+ ).filter(
+ (post) =>
+ !isRuntimeStatusPost(post) && !isNoSessionNotice(post) && lower(post).includes('done')
+ );
+ record(ranEarly);
+ if (ranEarly.length > 0) {
+ throw new Error(`The command ran on the answer of someone who may not address the agent.`);
+ }
+
+ const approvedAt = await answerCard(harness, card, 1);
+ const done = await expectBotPost(harness.mattermost, {
+ where: sharedRoom(harness),
+ since: approvedAt,
+ record,
+ predicate: (post) => lower(post).includes('done'),
+ describe: "a 'done' reply after the owner approved the command",
+ });
+ return `refused in ${notice.id}, approved by the owner, completed in ${done.id}`;
+ } finally {
+ await harness.switch.openAddressing(harness.agent.id);
+ }
+ });
+
+// ── template ─────────────────────────────────────────────────────────────────
+
+/**
+ * Run a template, retrying while the controller is not ready to take its agent:
+ * a refused run creates nothing, so running it again is safe.
+ */
+async function runTemplate(harness: Harness, template: string): Promise<TemplateRun> {
+ const until = Date.now() + PLACEMENT_DEADLINE_MS;
+ for (;;) {
+ try {
+ return await harness.switch.runTemplateDocument({ template, templateName: 'e2e-template' });
+ } catch (error) {
+ const waiting =
+ error instanceof TemplateRefusal &&
+ error.problems.length > 0 &&
+ error.problems.every((problem) => NOT_READY_CODES.has(problem.code));
+ if (!waiting || Date.now() > until) throw error;
+ await sleep(3_000);
+ }
+ }
+}
+
+/**
+ * A template creates a managed agent on the controller and a room on the
+ * Mattermost bridge with it in it, and the agent answers there: what a person
+ * using a template from Switch Console gets.
+ */
+export const templateAgent: Scenario = (harness, names) =>
+ scenario('template-agent', async (record) => {
+ const controllerId = harness.env.provider.controllerId;
+ if (controllerId === null) throw new Error('This scenario needs an agent controller.');
+ const provider = PROVIDER_BY_AGENT_TYPE[harness.env.provider.agentType];
+ const about = `Switch end-to-end harness (run ${harness.runId}). Safe to delete.`;
+ const agentName = names.name('tpl');
+ const bridge = await harness.switch.defaultBridge('mattermost');
+ const template = stringify({
+ version: 1,
+ agent: {
+ name: agentName,
+ description: about,
+ instructions: AGENT_INSTRUCTIONS,
+ provider,
+ machine: controllerId,
+ // The machine's own workspace for the agent, when the run names none.
+ directory: harness.env.provider.agentRepoDir ?? '{$agents_dir}/{agent}',
+ ...(provider === 'claude' ? { model: CLAUDE_MODEL } : {}),
+ auto_mode: false,
+ auto_approve: false,
+ addressing: 'anyone',
+ },
+ room: {
+ name: names.name('troom'),
+ description: about,
+ bridge: bridge.displayName,
+ agents: ['{agent}'],
+ users: [harness.human.username],
+ },
+ });
+
+ const run = await runTemplate(harness, template);
+ const undo: { label: string; run: () => Promise<void> }[] = [];
+ for (const made of run.created_agents) {
+ undo.push({
+ label: `agent ${made.name} (${made.agent_id})`,
+ run: () => harness.switch.deleteAgentByName(made.name),
+ });
+ }
+ const room = await harness.switch.getRoom(run.room.room_id);
+ const channelId = room.externalChannelId;
+ undo.push({
+ label: `room ${room.name} (${room.id})`,
+ run: async () => {
+ await harness.switch.deleteRoom(room.id);
+ if (channelId === null) return;
+ const left = await harness.mattermost.getChannel(channelId);
+ if (!left.delete_at) await harness.mattermost.archiveChannel(channelId);
+ },
+ });
+
+ let failed = false;
+ try {
+ if (run.created_agents.length !== 1) {
+ throw new Error(`The template made ${run.created_agents.length} agents, not one.`);
+ }
+ const made = run.created_agents[0]!;
+ if (made.name !== agentName || made.machine.id !== controllerId || !made.listening) {
+ throw new Error(`The template's agent is not the one asked for: ${JSON.stringify(made)}`);
+ }
+ if (channelId === null)
+ throw new Error(`The template's room ${room.id} has no Mattermost channel.`);
+ const bot = await harness.mattermost.waitForBotUser(agentName, 60_000);
+ const marker = `${OK_MARKER}_TPL`;
+ const since = Date.now() - 1;
+ await harness.mattermost.post({
+ channelId,
+ message: `@${agentName} reply with exactly ${marker} and nothing else`,
+ });
+ const reply = await expectBotPost(harness.mattermost, {
+ where: { channelId, botId: bot.id },
+ since,
+ record,
+ predicate: (post) => post.message.includes(marker),
+ describe: `a reply containing ${marker} from the template's agent`,
+ });
+ return `${agentName} on ${made.machine.name} replied in post ${reply.id}`;
+ } catch (error) {
+ failed = true;
+ throw error;
+ } finally {
+ const left: string[] = [];
+ for (const step of undo.reverse()) {
+ try {
+ await step.run();
+ } catch (error) {
+ left.push(`${step.label}: ${error instanceof Error ? error.message : String(error)}`);
+ }
+ }
+ if (left.length > 0) {
+ const message = `could not remove what the template made — delete it by hand: ${left.join('; ')}`;
+ if (failed) console.warn(message);
+ else throw new Error(message);
+ }
+ }
+ });
+
/**
* A provider-lane scenario: its header, and the function that drives it. All of
* them share one agent and one channel, set up once per run.
@@ -472,6 +720,26 @@ export const SCENARIOS: readonly ProviderScenario[] = [
header: providerHeader('interrupt', ['connect_to_room', 'post_message'], ['agent-dir']),
run: interrupt,
},
+ {
+ // A second person signs up, which only a throwaway or local Mattermost lets anyone do.
+ header: {
+ ...providerHeader(
+ 'card-answer-refused',
+ ['PUT /gateway/agents/{agent_id}/addressing-policy'],
+ []
+ ),
+ environments: ['ephemeral', 'local'],
+ },
+ run: cardAnswerRefused,
+ },
+ {
+ header: providerHeader(
+ 'template-agent',
+ ['POST /gateway/rooms/from-template-document'],
+ ['controller']
+ ),
+ run: templateAgent,
+ },
];
/** Fixed-width summary of a run, printed at the end of the suite. */
diff --git a/console/tooling-e2e/surface/exemptions.json b/console/tooling-e2e/surface/exemptions.json
index b49e321e7..55cac117e 100644
--- a/console/tooling-e2e/surface/exemptions.json
+++ b/console/tooling-e2e/surface/exemptions.json
@@ -42,10 +42,8 @@
"attach_reference_to_room",
"create_agent",
"create_reference",
- "create_room",
"create_room_document",
"create_room_from_yaml",
- "create_room_group",
"define_role",
"delete_role",
"delete_room_document",
@@ -56,7 +54,6 @@
"get_my_context",
"get_role_detail",
"get_room_detail",
- "get_room_group_detail",
"get_template",
"get_template_guide",
"invite_agent_to_room",
@@ -105,14 +102,12 @@
"DELETE /gateway/management/controllers/{controller_id}",
"DELETE /gateway/management/controllers/{controller_id}/provider-logins/{provider}",
"DELETE /gateway/messaging-apps/installs/{install_id}",
- "DELETE /gateway/packages/{package_id}",
"DELETE /gateway/packages/{package_id}/documents/{document_id}",
"DELETE /gateway/packages/{package_id}/references/{reference_id}",
"DELETE /gateway/provider-connections/github",
"DELETE /gateway/provider-connections/github/flows/{flow_id}",
"DELETE /gateway/references/reference-types/{type}",
"DELETE /gateway/references/{reference_id}",
- "DELETE /gateway/room-groups/{group_id}",
"DELETE /gateway/rooms/{room_id}",
"DELETE /gateway/rooms/{room_id}/agents/{agent_id}",
"DELETE /gateway/rooms/{room_id}/documents/{document_id}",
@@ -192,10 +187,8 @@
"GET /gateway/rooms/{room_id}/documents/{document_id}",
"GET /gateway/rooms/{room_id}/linked-rooms",
"GET /gateway/rooms/{room_id}/linked-rooms/inbound",
- "GET /gateway/rooms/{room_id}/packages",
"GET /gateway/rooms/{room_id}/references",
"GET /gateway/rooms/{room_id}/roles",
- "GET /gateway/rooms/{room_id}/yaml",
"GET /gateway/service-connections",
"GET /gateway/service-connections/{service}/agents",
"GET /gateway/service-connections/{service}/flows/{flow_id}",
@@ -261,7 +254,6 @@
"POST /gateway/management/operations",
"POST /gateway/messaging-apps/{platform}/claim",
"POST /gateway/messaging-apps/{platform}/install",
- "POST /gateway/packages",
"POST /gateway/packages/{package_id}/documents/{document_id}",
"POST /gateway/packages/{package_id}/references/{reference_id}",
"POST /gateway/provider-connections/github/flows",
@@ -272,7 +264,6 @@
"POST /gateway/rooms/bulk-archive",
"POST /gateway/rooms/bulk-delete",
"POST /gateway/rooms/from-template",
- "POST /gateway/rooms/from-template-document",
"POST /gateway/rooms/{room_id}/agents",
"POST /gateway/rooms/{room_id}/archive",
"POST /gateway/rooms/{room_id}/documents/{document_id}",
@@ -294,7 +285,6 @@
"POST /gateway/tenants/{tenant_id}/join-domains",
"POST /gateway/tenants/{tenant_id}/switch",
"POST /gateway/users",
- "PUT /gateway/agents/{agent_id}/addressing-policy",
"PUT /gateway/agents/{agent_id}/can-manage-agents",
"PUT /gateway/agents/{agent_id}/description",
"PUT /gateway/agents/{agent_id}/display-name",
diff --git a/core/switch_core/db/stores/room_store.py b/core/switch_core/db/stores/room_store.py
index a8968a92f..1730754d2 100644
--- a/core/switch_core/db/stores/room_store.py
+++ b/core/switch_core/db/stores/room_store.py
@@ -15,6 +15,10 @@ from switch_core.db.models import (
RoomGroup,
require_tenant_id,
room_agents,
+ room_documents,
+ room_packages,
+ room_references,
+ room_skills,
)
from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
@@ -248,9 +252,14 @@ class RoomStore:
async def delete(self, session: AsyncSession, room_id: str) -> None:
await session.execute(delete(ClientRoom).where(ClientRoom.room_id == room_id))
- await session.execute(
- delete(room_agents).where(room_agents.c.room_id == room_id)
- )
+ for attached in (
+ room_agents,
+ room_references,
+ room_documents,
+ room_packages,
+ room_skills,
+ ):
+ await session.execute(delete(attached).where(attached.c.room_id == room_id))
room = await session.get(Room, room_id)
if room:
await session.delete(room)
diff --git a/core/tests/switch_core/db/stores/test_room_store.py b/core/tests/switch_core/db/stores/test_room_store.py
index df845f221..f51c71211 100644
--- a/core/tests/switch_core/db/stores/test_room_store.py
+++ b/core/tests/switch_core/db/stores/test_room_store.py
@@ -5,8 +5,17 @@ import uuid
import pytest
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-from switch_core.db.models import Room, Tenant, TenantNotBoundError
+from switch_core.db.models import (
+ Package,
+ Reference,
+ Room,
+ Tenant,
+ TenantNotBoundError,
+ User,
+)
from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.package_store import PackageStore
+from switch_core.db.stores.reference_store import ReferenceStore
from switch_core.db.stores.room_store import RoomStore
@@ -111,3 +120,52 @@ class TestGetByTransportRoomIdRequiresATenant:
async with session_factory() as verify:
with pytest.raises(TenantNotBoundError):
await store.get_by_transport_room_id(verify, shared_transport_room_id)
+
+
+class TestDelete:
+ async def test_deletes_a_room_with_a_package_and_a_reference_attached(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ packages, references = PackageStore(), ReferenceStore()
+ async with session_factory() as session:
+ owner = User(name="owner", email="[email protected]", role="admin")
+ session.add(owner)
+ await session.flush()
+ room = await _make_room(session, "!attached:switch.local")
+ package = await packages.create(
+ session,
+ Package(
+ owner_id=owner.id,
+ read_visibility="private",
+ write_visibility="private",
+ name="pkg",
+ description="d",
+ instructions="i",
+ ),
+ )
+ reference = await references.create(
+ session,
+ Reference(
+ owner_id=owner.id,
+ read_visibility="private",
+ write_visibility="private",
+ type="url",
+ name="ref",
+ description="d",
+ instructions="i",
+ value={"url": "https://example.com"},
+ ),
+ )
+ await packages.attach_to_room(session, room.id, package.id)
+ await references.attach_to_room(session, room.id, reference.id)
+ await session.commit()
+ room_id, package_id, reference_id = room.id, package.id, reference.id
+
+ async with session_factory() as session:
+ await RoomStore().delete(session, room_id)
+ await session.commit()
+
+ async with session_factory() as session:
+ assert await session.get(Room, room_id) is None
+ assert await packages.get(session, package_id) is not None
+ assert await session.get(Reference, reference_id) is not None