Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

E2E provider lane on a headless agent controller

Petr Baucheea8ed01a9a0
diff --git a/console/tooling-e2e/README.md b/console/tooling-e2e/README.md
index 6f36ec74e..b8f620dc8 100644
--- a/console/tooling-e2e/README.md
+++ b/console/tooling-e2e/README.md
@@ -155,15 +155,61 @@ beside this file. Never commit one. `just e2e-fast` sets all of them itself.
| `SWITCH_E2E_SESSION_TIMEOUT_MS` | provider lane | How long setup waits for a live session (default 2 min). |
| `SWITCH_E2E_AGENT_TYPE` | provider lane | `opencode` (default), `claude-code`, `codex`, `antigravity` or `cursor` — which provider's runtime is under test. |
| `SWITCH_E2E_QUESTION_MODE` | provider lane | `numbered` (default) or `prose`; see the `question` row below. |
+| `SWITCH_E2E_CONTROLLER_ID` | provider lane | An agent controller to run the agent on. `just e2e-provider` sets it. Unset, a Switch Console session is expected instead. |
+| `SWITCH_E2E_PROVIDER_ENV_FILE` | `just e2e-provider` | A file of provider keys, handed to the controller as `--env-file`. Leave it unset to use the CLI's own sign-in. |
Mattermost is all or nothing: none of its three variables means "no Mattermost
here"; some of them is reported as a mistake.
# The provider lane
-## What must be running
+The provider lane needs Mattermost and something that runs a real session for
+the agent it registers. There are two ways to provide one.
-The provider lane needs Mattermost and a **Switch Console session for the agent
+## On a headless agent controller (`just e2e-provider`)
+
+The unattended way, and the one CI will use. `just e2e-provider` does
+everything `just e2e-fast` does, and also:
+
+1. turns on `agent_management` and sets `CONTROLLER_TOKEN_SECRET` for the stack;
+2. builds `console/packages` and starts the agent controller
+ (`console/packages/agent-controller`) on the host. Its credential comes from
+ `POST /gateway/management/controllers` and goes in on stdin, never to disk;
+3. runs the provider lane with `SWITCH_E2E_CONTROLLER_ID` set. Setup then
+ registers the agent with a registration key the admin mints for the purpose
+ and deletes straight away, because the management API only places agents
+ their caller owns, and the deployment-wide key registers agents to a
+ bootstrap account. Setup places the agent on the controller with
+ `auto_approve: false`, retrying while the controller reports itself not
+ ready, and writes no Switch Console files.
+
+```bash
+SWITCH_E2E_AGENT_TYPE=claude-code just e2e-provider
+SWITCH_E2E_PROVIDER_ENV_FILE=~/e2e-keys.env just e2e-provider # an API key instead of a signed-in CLI
+```
+
+The provider CLI (`claude`, `opencode`, …) must be installed and signed in for
+the user running it, or its key must be in the file `SWITCH_E2E_PROVIDER_ENV_FILE`
+names, which the controller gets as `--env-file`. Sessions only inherit the
+variables on the allow-list in
+`console/packages/agent-providers/src/host/agent-env.ts`. `node
+console/packages/agent-controller/dist/cli.mjs doctor` shows what the controller
+sees. A run spends real model calls.
+
+Two things differ from a Console session:
+- **The agent's own key stops working.** While a controller acts for an agent,
+ the server refuses requests made with the agent's own key. So the
+ bridge-readiness probe reads the room through a silent second agent, the
+ *witness*, and teardown deletes agents by name as the admin.
+- **`live` comes early.** Switch reports the agent `live` as soon as the
+ controller is connected, before any session exists. The first scenario
+ therefore absorbs the session's start-up time.
+
+The controller's log goes to `build/e2e/ephemeral/controller.log`.
+
+## With a Switch Console session
+
+The provider lane can also run against a **Switch Console session for the agent
it just registered**. Setup polls the agent's status in the room and refuses to run the
scenarios until Switch reports it `live` (a session is attending the room) or
`dormant` (an auto-session watcher is running and will spawn one on demand) —
@@ -243,7 +289,7 @@ agent, and deletes it during teardown — so a manifest never outlives its room.
| --- | --- | --- |
| `greet` | An addressed message reaches a session and its answer comes back. | The bot posts `SWITCH_E2E_OK`. |
| `question` | Two turns: the agent asks, the human answers in the channel, the agent uses the answer. | The bot offers `red`/`green`/`blue`, then replies `green` after the answer. |
-| `approval` | A tool call the session is not pre-authorised to make. | The bot surfaces an approval prompt, `@agent 1` allows it, the bot replies `done`. |
+| `approval` | A tool call the session is not pre-authorised to make. | The bot posts an approval card, answering it with `<handle> 1` allows the call, the bot replies `done`. |
| `interrupt` | `!interrupt @agent-name` cancels a running local shell command after its start marker is verified. | No completion marker appears after the command’s 90-second lifetime (observed for 95 seconds). Requires `SWITCH_E2E_AGENT_DIR` on this machine. |
`question` is the one a one-shot reply cannot fake: it only passes if the session
@@ -251,9 +297,10 @@ is still alive and still connected when the second message arrives.
It has two modes, because the relay it exercises needs a native ask-the-user
tool and not every provider offers one. In `numbered` mode the agent must use
-that tool, the console relays it into the room as a numbered list, and the
-answer goes back as `@agent 2` — the numbering is what proves the relay path was
-taken. In `prose` mode the agent asks in an ordinary message and is answered
+that tool, and Switch relays it into the room as a request card numbering the
+options (`… request \`A1\` … 2. green …`). The answer goes back as `A1 2`, a
+plain message naming the card's handle, and the card is what proves the relay
+path was taken. In `prose` mode the agent asks in an ordinary message and is answered
with `@agent green`; the two-turn round trip is still proved, the relay is not.
Every agent type defaults to `numbered` — both providers offer the tool — and
`prose` is an explicit override for a provider or CLI version that does not.
diff --git a/console/tooling-e2e/scripts/ephemeral-stack.sh b/console/tooling-e2e/scripts/ephemeral-stack.sh
index b75681c2c..4b839fca5 100755
--- a/console/tooling-e2e/scripts/ephemeral-stack.sh
+++ b/console/tooling-e2e/scripts/ephemeral-stack.sh
@@ -4,9 +4,16 @@
# host, then the chosen lane, then teardown.
#
# just e2e-fast the fast lane (no provider, no Console)
+# just e2e-provider the provider lane, on a headless agent controller
# just e2e-fast --keep leave the stack running afterwards
# just e2e-fast -- <vitest args> pass arguments through to vitest
#
+# The provider lane also builds console/packages and starts an agent controller
+# on the host, which runs the agent's sessions with a real provider CLI. The CLI
+# must be installed and signed in for the user running this, or its key given in
+# the file SWITCH_E2E_PROVIDER_ENV_FILE names (passed to the controller as
+# --env-file). SWITCH_E2E_AGENT_TYPE picks the provider (default opencode).
+#
# Secrets are generated per run into a temporary env file. The repository's
# own .env is never read or written, so this cannot rotate a developer's live
# secrets or pick up a stale value from them.
@@ -44,6 +51,12 @@ rm -rf "$OUT"; mkdir -p "$OUT"
ENV_FILE="$OUT/stack.env"
cp "$REPO/.env.example" "$ENV_FILE"
"$REPO/deploy/local/generate-env-secrets.sh" "$ENV_FILE" e2e
+if [ "$LANE" = provider ]; then
+ # Agent management, and the key controllers' access tokens are signed with.
+ sed -i.bak -e "s|^FEATURE_FLAGS_ENABLED=.*|FEATURE_FLAGS_ENABLED=agent_management|" \
+ -e "s|^CONTROLLER_TOKEN_SECRET=.*|CONTROLLER_TOKEN_SECRET=$(openssl rand -hex 32)|" "$ENV_FILE"
+ rm -f "$ENV_FILE.bak"
+fi
chmod 600 "$ENV_FILE"
set -a; source "$ENV_FILE"; set +a
@@ -51,16 +64,18 @@ cd "$REPO"
COMPOSE=(docker compose -p switch-e2e-ephemeral -f deploy/local/docker-compose.yml
--env-file "$ENV_FILE" --project-directory .)
SERVER_PID=""
+CONTROLLER_PID=""
cleanup() {
local status=$?
if [ "$KEEP" = 0 ]; then
+ [ -n "$CONTROLLER_PID" ] && kill "$CONTROLLER_PID" 2>/dev/null && wait "$CONTROLLER_PID" 2>/dev/null
[ -n "$SERVER_PID" ] && kill "$SERVER_PID" 2>/dev/null && wait "$SERVER_PID" 2>/dev/null
"${COMPOSE[@]}" down -v >/dev/null 2>&1 || true
rm -f "$ENV_FILE"
else
- echo "Left running: switch-core :8000 (pid $SERVER_PID), Mattermost :8065."
+ echo "Left running: switch-core :8000 (pid $SERVER_PID), Mattermost :8065${CONTROLLER_PID:+, agent controller (pid $CONTROLLER_PID)}."
echo "Secrets for this stack: $ENV_FILE. Tear down with:"
- echo " kill $SERVER_PID; ${COMPOSE[*]} down -v"
+ echo " kill $SERVER_PID $CONTROLLER_PID; ${COMPOSE[*]} down -v"
fi
exit "$status"
}
@@ -101,10 +116,49 @@ MATTERMOST_TOKEN="$(curl -sf -D - -o /dev/null -H 'Content-Type: application/jso
http://localhost:8065/api/v4/users/login | tr -d '\r' | awk -F': ' 'tolower($1)=="token" {print $2}')"
[ -n "$MATTERMOST_TOKEN" ] || { echo "could not log in to Mattermost as $MATTERMOST_USER" >&2; exit 1; }
+PROVIDER_ENV=()
+if [ "$LANE" = provider ]; then
+ echo "Building the agent controller…"
+ (cd "$REPO/console" && pnpm -r --filter './packages/**' run build) >"$OUT/console-build.log" 2>&1 \
+ || { echo "console package build failed; see $OUT/console-build.log" >&2; exit 1; }
+
+ JAR="$OUT/gateway.cookies"
+ curl -sf -c "$JAR" -o /dev/null -H 'Content-Type: application/json' \
+ -d "$(jq -n --arg e "$GATEWAY_ADMIN_EMAIL" --arg p "$GATEWAY_ADMIN_PASSWORD" '{email: $e, password: $p}')" \
+ http://localhost:8000/gateway/auth/login \
+ || { echo "could not sign in to the gateway as $GATEWAY_ADMIN_EMAIL" >&2; exit 1; }
+ CONTROLLER="$(curl -sf -b "$JAR" -H 'Content-Type: application/json' \
+ -d "$(jq -n --arg os "$(uname -s | tr '[:upper:]' '[:lower:]')" --arg arch "$(uname -m)" \
+ --arg osv "$(uname -r)" \
+ '{name: "e2e-controller", kind: "console", version: "e2e",
+ platform: {os: $os, arch: $arch, os_version: $osv}}')" \
+ http://localhost:8000/gateway/management/controllers)" \
+ || { echo "could not create an agent controller (is agent_management on?)" >&2; exit 1; }
+ rm -f "$JAR"
+ CONTROLLER_ID="$(jq -r .controller_id <<<"$CONTROLLER")"
+
+ echo "Starting the agent controller…"
+ CONTROLLER_ARGS=(run --data-dir "$OUT/controller" --controller-id "$CONTROLLER_ID"
+ --server http://localhost:8000 --credential-stdin)
+ if [ -n "${SWITCH_E2E_PROVIDER_ENV_FILE:-}" ]; then
+ CONTROLLER_ARGS+=(--env-file "$SWITCH_E2E_PROVIDER_ENV_FILE")
+ fi
+ # The credential goes in on stdin and is never written to disk.
+ jq -r .credential <<<"$CONTROLLER" | SWITCH_CONTROLLER_LOG_LEVEL=info \
+ node "$REPO/console/packages/agent-controller/dist/cli.mjs" "${CONTROLLER_ARGS[@]}" \
+ >"$OUT/controller.log" 2>&1 &
+ CONTROLLER_PID=$!
+ unset CONTROLLER
+
+ mkdir -p "$OUT/agent-dir"
+ PROVIDER_ENV=(SWITCH_E2E_CONTROLLER_ID="$CONTROLLER_ID" SWITCH_E2E_AGENT_DIR="$OUT/agent-dir")
+fi
+
echo "Running the $LANE lane…"
status=0
(
cd "$HARNESS"
+ export ${PROVIDER_ENV[@]+"${PROVIDER_ENV[@]}"}
SWITCH_E2E_LANE="$LANE" \
SWITCH_E2E_ENVIRONMENT=ephemeral \
SWITCH_E2E_STRICT=1 \
@@ -121,5 +175,6 @@ status=0
if [ "$status" != 0 ]; then
echo "The $LANE lane failed. switch-core log: $OUT/switch-core.log" >&2
+ [ "$LANE" = provider ] && echo "Agent controller log: $OUT/controller.log" >&2
fi
exit "$status"
diff --git a/console/tooling-e2e/src/clients/switch-client.ts b/console/tooling-e2e/src/clients/switch-client.ts
index 391b5bc9d..bee901704 100644
--- a/console/tooling-e2e/src/clients/switch-client.ts
+++ b/console/tooling-e2e/src/clients/switch-client.ts
@@ -38,6 +38,15 @@ export interface RoomDetail extends RoomSummary {
agentIds: string[];
}
+/** The management API's agent definition (v1), as much of it as the harness sets. */
+export interface ManagedAgentDefinition {
+ provider: string;
+ model: string | null;
+ instructions: string;
+ auto_approve: boolean;
+ directory: string | null;
+}
+
export interface AgentSummary {
id: string;
name: string;
@@ -113,7 +122,8 @@ interface SwitchRequest {
export class SwitchClient {
private readonly apiUrl: string;
- private readonly registrationToken: string;
+ /** The deployment-wide registration key; agents it registers belong to a bootstrap account. */
+ readonly registrationToken: string;
private readonly adminEmail: string;
private readonly adminPassword: string;
private readonly environment: Environment;
@@ -263,6 +273,42 @@ export class SwitchClient {
await this.gateway('DELETE', `/agents/by-name/${encodeURIComponent(name)}`);
}
+ /**
+ * `POST /gateway/api-keys` — a registration key belonging to the signed-in
+ * admin. Agents registered with it are owned by the admin, which the
+ * management API requires of an agent it is asked to place on a controller;
+ * agents registered with the deployment-wide key belong to a bootstrap
+ * account instead.
+ */
+ async createRegistrationKey(label: string): Promise<{ id: string; key: string }> {
+ const body = await this.gateway<{ id: string; key: string }>('POST', '/api-keys', { label });
+ return { id: body.id, key: body.key };
+ }
+
+ async deleteApiKey(keyId: string): Promise<void> {
+ await this.gateway('DELETE', `/api-keys/${keyId}`);
+ }
+
+ /**
+ * `PUT /gateway/management/agents/{id}` — place an agent the admin owns on a
+ * controller and set it running. The controller then runs its sessions.
+ *
+ * Answers 409 until the controller has connected and reported the provider
+ * installed and signed in; the caller retries on {@link SwitchHttpError}
+ * status 409 while the controller comes up.
+ */
+ async placeAgent(params: {
+ agentId: string;
+ controllerId: string;
+ definition: ManagedAgentDefinition;
+ }): Promise<void> {
+ await this.gateway('PUT', `/management/agents/${params.agentId}`, {
+ controller_id: params.controllerId,
+ desired_state: 'running',
+ definition: params.definition,
+ });
+ }
+
/**
* `PUT /gateway/agents/{id}/addressing-policy` with `{policy: null}` — clear
* the agent's scoped addressing policy, so anyone may address it.
@@ -362,6 +408,8 @@ export class SwitchClient {
* be added to a channel.
*/
async registerKnownAgent(params: {
+ /** {@link registrationToken}, or a personal key from {@link createRegistrationKey}. */
+ registrationToken: string;
agentType: 'opencode' | 'codex' | 'claude-code' | 'antigravity' | 'cursor';
name: string;
description: string;
@@ -371,7 +419,7 @@ export class SwitchClient {
'POST',
'/agents/register-known',
{
- token: this.registrationToken,
+ token: params.registrationToken,
body: {
agent_type: params.agentType,
name: params.name,
diff --git a/console/tooling-e2e/src/core/env.ts b/console/tooling-e2e/src/core/env.ts
index 1a280d130..8a869d1a2 100644
--- a/console/tooling-e2e/src/core/env.ts
+++ b/console/tooling-e2e/src/core/env.ts
@@ -76,6 +76,12 @@ export interface ProviderEnv {
agentType: HarnessAgentType;
/** How `question` expects the clarifying question to arrive. */
questionMode: QuestionMode;
+ /**
+ * A headless agent controller to run the agent on. When set, setup places
+ * the agent on it through the management API and writes no Switch Console
+ * files; when not, a Console session for the agent is expected instead.
+ */
+ controllerId: string | null;
}
export interface HarnessEnv {
@@ -242,6 +248,7 @@ export function loadEnv(
manifestPath: get('SWITCH_E2E_MANIFEST') ?? null,
agentType: agentTypeFrom(get('SWITCH_E2E_AGENT_TYPE')),
questionMode: questionModeFrom(get('SWITCH_E2E_QUESTION_MODE')),
+ controllerId: get('SWITCH_E2E_CONTROLLER_ID') ?? null,
},
};
}
diff --git a/console/tooling-e2e/src/fixtures/fixtures.ts b/console/tooling-e2e/src/fixtures/fixtures.ts
index 691886517..d00c96560 100644
--- a/console/tooling-e2e/src/fixtures/fixtures.ts
+++ b/console/tooling-e2e/src/fixtures/fixtures.ts
@@ -53,6 +53,7 @@ export class Fixtures {
): Promise<RegisteredAgent> {
const name = this.names.name(role);
const agent = await this.switchClient.registerKnownAgent({
+ registrationToken: this.switchClient.registrationToken,
agentType: options.agentType ?? 'opencode',
name,
description: `Switch end-to-end harness agent (run ${this.names.runId}). Safe to delete.`,
diff --git a/console/tooling-e2e/src/provider/harness.ts b/console/tooling-e2e/src/provider/harness.ts
index 17269c4bf..38dc491ef 100644
--- a/console/tooling-e2e/src/provider/harness.ts
+++ b/console/tooling-e2e/src/provider/harness.ts
@@ -6,6 +6,7 @@ import type {
} from '../clients/mattermost-client.ts';
import {
sleep,
+ SwitchHttpError,
type RegisteredAgent,
type RoomDetail,
type SwitchClient,
@@ -31,6 +32,13 @@ export interface Harness {
switch: SwitchClient;
mattermost: MattermostClient;
agent: RegisteredAgent;
+ /**
+ * A silent second agent in the room, registered with the deployment key and
+ * run by nobody. The bridge-readiness probe reads the room through it,
+ * because an agent run by a controller cannot use its own key: the server
+ * refuses it while the controller acts for the agent.
+ */
+ witness: RegisteredAgent;
/** The agent's Mattermost bot account — `username === agent.name`. */
bot: MattermostUser;
channel: MattermostChannel;
@@ -55,6 +63,102 @@ const ROLE_BY_AGENT_TYPE: Record<HarnessAgentType, string> = {
codex: 'codex',
};
+// "Do exactly what you are asked" is load-bearing, and so is the second
+// sentence: told merely to answer briefly, the agent declined the `interrupt`
+// scenario's task on the grounds that 200 messages would spam the room — a
+// refusal the scenario reads as a session that never started.
+const AGENT_INSTRUCTIONS =
+ 'You are a Switch end-to-end test agent. Do exactly what you are asked in the ' +
+ 'room, and post your answer back to the room you were addressed in. Never ' +
+ 'decline or shorten a task on the grounds that it is repetitive or long: the ' +
+ 'requests you get here are deliberate tests of exactly that.';
+
+/** Cheap on purpose: a run spends a real subscription. */
+const CLAUDE_MODEL = 'claude-sonnet-5';
+
+/** The management API's provider id for each harness agent type. */
+const PROVIDER_BY_AGENT_TYPE: Record<HarnessAgentType, string> = {
+ opencode: 'opencode',
+ 'claude-code': 'claude',
+ codex: 'codex',
+ antigravity: 'antigravity',
+ cursor: 'cursor',
+};
+
+/** How long to keep asking a controller to take the agent while it comes up. */
+const PLACEMENT_DEADLINE_MS = 120_000;
+
+/**
+ * Register the agent. For a controller it is registered with a key the admin
+ * mints for the purpose and deletes straight after, because the management API
+ * only places agents their caller owns, and the deployment-wide key registers
+ * them to a bootstrap account.
+ */
+async function registerAgent(
+ switchClient: SwitchClient,
+ env: HarnessEnv,
+ name: string,
+ runId: string,
+ forController: boolean
+): Promise<RegisteredAgent> {
+ // `ClaudeCodeOptions` and `OpencodeOptions` agree on the two keys used here
+ // (`auto_session`, `repo_dir`), so the payload is the same for both types.
+ const params = {
+ agentType: env.provider.agentType,
+ name,
+ description: `Switch end-to-end harness agent (run ${runId}). Safe to delete.`,
+ options: {
+ auto_session: true,
+ ...(env.provider.agentRepoDir ? { repo_dir: env.provider.agentRepoDir } : {}),
+ },
+ };
+ if (!forController) {
+ return switchClient.registerKnownAgent({
+ ...params,
+ registrationToken: switchClient.registrationToken,
+ });
+ }
+ const key = await switchClient.createRegistrationKey(`e2e harness run ${runId}`);
+ try {
+ return await switchClient.registerKnownAgent({ ...params, registrationToken: key.key });
+ } finally {
+ await switchClient.deleteApiKey(key.id);
+ }
+}
+
+/**
+ * Hand the agent to the controller, retrying while the server answers 409:
+ * the controller has not connected yet, or has not yet reported the provider
+ * installed and signed in. Any other refusal is final.
+ */
+async function placeOnController(
+ switchClient: SwitchClient,
+ env: HarnessEnv,
+ agent: RegisteredAgent,
+ controllerId: string
+): Promise<void> {
+ const provider = PROVIDER_BY_AGENT_TYPE[env.provider.agentType];
+ const definition = {
+ provider,
+ model: provider === 'claude' ? CLAUDE_MODEL : null,
+ instructions: AGENT_INSTRUCTIONS,
+ // The `approval` scenario needs the session to ask before running a tool.
+ auto_approve: false,
+ directory: env.provider.agentRepoDir,
+ };
+ const until = Date.now() + PLACEMENT_DEADLINE_MS;
+ for (;;) {
+ try {
+ await switchClient.placeAgent({ agentId: agent.id, controllerId, definition });
+ return;
+ } catch (error) {
+ const notReady = error instanceof SwitchHttpError && error.status === 409;
+ if (!notReady || Date.now() > until) throw error;
+ await sleep(3_000);
+ }
+ }
+}
+
/**
* Register a throwaway agent, create a room on the Mattermost bridge for it,
* and wait for the bridge to carry a message.
@@ -89,31 +193,31 @@ export async function setupHarness(target: LaneTarget): Promise<Harness> {
// and clearly when MATTERMOST_TEAM names a team that does not exist.
await mattermost.findTeam(env.mattermost.team);
- // `ClaudeCodeOptions` and `OpencodeOptions` agree on the two keys used here
- // (`auto_session`, `repo_dir`), so the payload is the same for both types.
- const agent = await switchClient.registerKnownAgent({
- agentType: env.provider.agentType,
- name: agentName,
- description: `Switch Console end-to-end harness agent (run ${runId}). Safe to delete.`,
- options: {
- auto_session: true,
- ...(env.provider.agentRepoDir ? { repo_dir: env.provider.agentRepoDir } : {}),
- },
- });
+ const controllerId = env.provider.controllerId;
+ const agent = await registerAgent(switchClient, env, agentName, runId, controllerId !== null);
// Owner-only by default, and the human the harness posts as is an unclaimed
// Mattermost identity, so its messages would be refused rather than delivered.
await switchClient.openAddressing(agent.id);
- // Registration is what mints the bot; wait for it so a later `@name` mention
- // has something to resolve to.
+ const witness = await switchClient.registerKnownAgent({
+ registrationToken: switchClient.registrationToken,
+ agentType: 'opencode',
+ name: names.name('witness'),
+ description: `Switch end-to-end harness bridge witness (run ${runId}). Safe to delete.`,
+ options: { auto_session: false },
+ });
+
+ // Registration is what mints the bots; wait for them so the room has them
+ // as members and a later `@name` mention has something to resolve to.
const bot = await mattermost.waitForBotUser(agent.name, 60_000);
+ await mattermost.waitForBotUser(witness.name, 60_000);
const room = await switchClient.createRoom({
name: names.name('room'),
- description: `Switch Console end-to-end harness (run ${runId}). Safe to delete.`,
+ description: `Switch end-to-end harness (run ${runId}). Safe to delete.`,
bridgeId,
- agentNames: [agent.name],
+ agentNames: [agent.name, witness.name],
userNames: [human.username],
});
@@ -130,6 +234,7 @@ export async function setupHarness(target: LaneTarget): Promise<Harness> {
switch: switchClient,
mattermost,
agent,
+ witness,
bot,
channel,
room,
@@ -137,10 +242,12 @@ export async function setupHarness(target: LaneTarget): Promise<Harness> {
runId,
};
- // With SWITCH_E2E_AGENT_DIR set, provision the agent's identity into that
- // working directory so a session started there IS this agent. Without it the
- // operator has to add the agent in Switch Console by hand before the run.
- if (env.provider.agentRepoDir) {
+ if (controllerId !== null) {
+ await placeOnController(switchClient, env, agent, controllerId);
+ } else if (env.provider.agentRepoDir) {
+ // Provision the agent's identity into that working directory so a Console
+ // session started there IS this agent. Without it the operator has to add
+ // the agent in Switch Console by hand before the run.
harness.credentialsPath = await writeAgentCredentials({
workingDir: env.provider.agentRepoDir,
agent,
@@ -153,24 +260,14 @@ export async function setupHarness(target: LaneTarget): Promise<Harness> {
workingDir: env.provider.agentRepoDir,
agentName: agent.name,
description: `Switch Console end-to-end harness agent (run ${runId}).`,
- // "Do exactly what you are asked" is load-bearing, and so is the second
- // sentence: told merely to answer briefly, the agent declined the
- // `interrupt` scenario's task on the grounds that 200 messages would
- // spam the room — a refusal the scenario reads as a session that never
- // started.
- instructions:
- 'You are a Switch end-to-end test agent. Do exactly what you are asked in the ' +
- 'room, and post your answer back to the room you were addressed in. Never ' +
- 'decline or shorten a task on the grounds that it is repetitive or long: the ' +
- 'requests you get here are deliberate tests of exactly that.',
- // Cheap on purpose: this spends a real subscription on every run.
- model: 'claude-sonnet-5',
+ instructions: AGENT_INSTRUCTIONS,
+ model: CLAUDE_MODEL,
effort: 'low',
});
}
}
- await waitForBridgeReady({ switchClient, mattermost, agent, room, channel, runId });
+ await waitForBridgeReady({ switchClient, mattermost, agent: witness, room, channel, runId });
await writeManifest(env.provider.manifestPath, harness);
return harness;
}
@@ -188,6 +285,7 @@ export async function setupHarness(target: LaneTarget): Promise<Harness> {
export interface HarnessManifest {
runId: string;
agent: RegisteredAgent;
+ witness: RegisteredAgent;
roomId: string;
channelId: string;
}
@@ -197,6 +295,7 @@ async function writeManifest(path: string | null, harness: Harness): Promise<voi
const manifest: HarnessManifest = {
runId: harness.runId,
agent: harness.agent,
+ witness: harness.witness,
roomId: harness.room.id,
channelId: harness.channel.id,
};
@@ -228,6 +327,7 @@ async function reuseHarness(
switch: switchClient,
mattermost,
agent: manifest.agent,
+ witness: manifest.witness,
bot: await mattermost.waitForBotUser(manifest.agent.name, 30_000),
channel: await mattermost.getChannel(manifest.channelId),
room: await switchClient.getRoom(manifest.roomId),
@@ -238,7 +338,7 @@ async function reuseHarness(
await waitForBridgeReady({
switchClient,
mattermost,
- agent: harness.agent,
+ agent: harness.witness,
room: harness.room,
channel: harness.channel,
runId: harness.runId,
@@ -317,13 +417,17 @@ export async function teardownHarness(
);
}
- try {
- await harness.switch.deleteAgent(harness.agent);
- } catch (error) {
- log(
- `teardown: failed to delete agent ${harness.agent.name} (${harness.agent.id}) — ` +
- `delete it by hand: ${error instanceof Error ? error.message : String(error)}`
- );
+ // By name, as the admin: an agent run by a controller cannot delete itself
+ // with its own key, which the server refuses while the controller acts for it.
+ for (const agent of [harness.agent, harness.witness]) {
+ try {
+ await harness.switch.deleteAgentByName(agent.name);
+ } catch (error) {
+ log(
+ `teardown: failed to delete agent ${agent.name} (${agent.id}) — ` +
+ `delete it by hand: ${error instanceof Error ? error.message : String(error)}`
+ );
+ }
}
// Deleting the room usually takes the channel with it; archive is a no-op
diff --git a/console/tooling-e2e/src/provider/scenarios.test.ts b/console/tooling-e2e/src/provider/scenarios.test.ts
index ede5d897c..c51b020aa 100644
--- a/console/tooling-e2e/src/provider/scenarios.test.ts
+++ b/console/tooling-e2e/src/provider/scenarios.test.ts
@@ -1,5 +1,5 @@
import { describe, expect, it } from 'vitest';
-import { isStartMarker } from './scenarios.ts';
+import { isStartMarker, requestHandle } from './scenarios.ts';
describe('interruption admission', () => {
it('requires the command marker, not an acknowledgement or refusal', () => {
@@ -10,3 +10,13 @@ describe('interruption admission', () => {
expect(isStartMarker('I will write SWITCH_INTERRUPT_STARTED')).toBe(false);
});
});
+
+describe('request cards', () => {
+ it('reads the handle a card answers to', () => {
+ expect(requestHandle('@user **Questions** · request `A1` Question from the agent 1. red')).toBe(
+ 'A1'
+ );
+ expect(requestHandle('**Permission** · request `R42` Run `echo hi`?')).toBe('R42');
+ expect(requestHandle('SWITCH_E2E_OK')).toBeNull();
+ });
+});
diff --git a/console/tooling-e2e/src/provider/scenarios.ts b/console/tooling-e2e/src/provider/scenarios.ts
index eeb6b4d5f..fba4d761f 100644
--- a/console/tooling-e2e/src/provider/scenarios.ts
+++ b/console/tooling-e2e/src/provider/scenarios.ts
@@ -132,6 +132,24 @@ async function scenario(
}
}
+/**
+ * The handle a request card answers to: `A1` in "request `A1`". Switch renders
+ * every relayed question and permission prompt as such a card, and an answer
+ * is the handle followed by the option's number, typed as a plain message.
+ */
+export function requestHandle(text: string): string | null {
+ return /request `([A-Za-z]+\d+)`/.exec(text)?.[1] ?? null;
+}
+
+/** Answer a request card, and return the moment just before the answer was posted. */
+async function answerCard(harness: Harness, card: MattermostPost, option: number): Promise<number> {
+ const handle = requestHandle(card.message);
+ if (handle === null) throw new Error(`Post ${card.id} is not a request card: «${card.message}»`);
+ const sinceMs = Date.now() - 1;
+ await harness.mattermost.post({ channelId: harness.channel.id, message: `${handle} ${option}` });
+ return sinceMs;
+}
+
/** Post `@agent <text>` and return the moment just before it was posted. */
async function ask(harness: Harness, text: string): Promise<number> {
const sinceMs = Date.now() - 1;
@@ -234,27 +252,31 @@ export const question: Scenario = (harness) =>
const asked = await expectBotPost(harness, {
since,
record,
- // The console relays a native question into the room as a numbered list;
- // a plain-prose question also lists the three options, so the numbering is
+ // A native question is relayed as a request card numbering the options;
+ // a plain-prose question also lists the three options, so the card is
// what proves the relay path was taken — and is exactly what cannot be
// required of a provider that offers SDK sessions no such tool.
predicate: (post) =>
CHOICES.every((choice) => lower(post).includes(choice)) &&
- (prose || /\b2\.\s/.test(post.message)),
+ (prose || (requestHandle(post.message) !== null && /\b2\.\s/.test(post.message))),
describe: prose
? `a question listing the options ${CHOICES.join('/')}`
: `a relayed question numbering the options ${CHOICES.join('/')}`,
});
- // By number where the relay is what reads the answer; by word where the
- // agent itself is. The relay's acknowledgement names the choice too, so the
- // agent's own reply must be told apart from it either way.
- const answeredAt = await ask(harness, prose ? CHOSEN : '2');
+ // On the card where the relay is what reads the answer; by word where the
+ // agent itself is. The card and the relay's acknowledgement name the choice
+ // too, so the agent's own reply must be told apart from them either way.
+ const answeredAt = prose
+ ? await ask(harness, CHOSEN)
+ : await answerCard(harness, asked, CHOICES.indexOf(CHOSEN) + 1);
const answer = await expectBotPost(harness, {
since: answeredAt,
record,
predicate: (post) =>
- lower(post).includes(CHOSEN) && !post.message.trimStart().startsWith('✅'),
+ lower(post).includes(CHOSEN) &&
+ requestHandle(post.message) === null &&
+ !post.message.trimStart().startsWith('✅'),
describe: `a reply naming my choice '${CHOSEN}'`,
});
@@ -288,12 +310,12 @@ export const approval: Scenario = (harness) =>
since,
record,
predicate: (post) =>
- APPROVAL_HINTS.some((hint) => lower(post).includes(hint)) ||
- /(^|\n)\s*1[.)]/.test(post.message),
- describe: 'an approval prompt for the shell command',
+ requestHandle(post.message) !== null &&
+ APPROVAL_HINTS.some((hint) => lower(post).includes(hint)),
+ describe: 'an approval card for the shell command',
});
- const approvedAt = await ask(harness, '1');
+ const approvedAt = await answerCard(harness, prompt, 1);
const done = await expectBotPost(harness, {
since: approvedAt,
record,
@@ -334,13 +356,15 @@ export const interrupt: Scenario = (harness) =>
harness,
`Run this exact shell command for a bounded cancellation test: \`${command}\`. Wait for completion before replying. Do not create these files with another tool.`
);
- await expectBotPost(harness, {
+ const permission = await expectBotPost(harness, {
since,
record,
- predicate: (post) => APPROVAL_HINTS.some((hint) => lower(post).includes(hint)),
- describe: 'permission to run the cancellation command',
+ predicate: (post) =>
+ requestHandle(post.message) !== null &&
+ APPROVAL_HINTS.some((hint) => lower(post).includes(hint)),
+ describe: 'an approval card for the cancellation command',
});
- await ask(harness, '1');
+ await answerCard(harness, permission, 1);
const deadline = Date.now() + 60_000;
while (!isStartMarker((await markerContents(startedFile)) ?? '')) {
if (Date.now() > deadline)
diff --git a/justfile b/justfile
index 725d9b6e6..cdb5b148a 100644
--- a/justfile
+++ b/justfile
@@ -168,6 +168,14 @@ e2e-follow-ups *args:
e2e-fast *args:
console/tooling-e2e/scripts/ephemeral-stack.sh {{ args }}
+# ── End-to-end harness, provider lane, on a throwaway stack ────────────────────
+# As e2e-fast, plus a headless agent controller that runs the agent's sessions
+# with a real provider CLI (installed and signed in, or its key in the file
+# SWITCH_E2E_PROVIDER_ENV_FILE names). SWITCH_E2E_AGENT_TYPE picks the provider.
+# Spends real model calls.
+e2e-provider *args:
+ SWITCH_E2E_LANE=provider console/tooling-e2e/scripts/ephemeral-stack.sh {{ args }}
+
# ── Run integration tests (real Postgres via testcontainers) ───────────────────
# DOCKER_HOST is auto-resolved from the active docker context in conftest, so this
# works under Docker Desktop / OrbStack / colima without extra setup.