Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

E2E harness foundation: fast and provider lanes in one package

Petr Bauch128540862dee
diff --git a/console/tooling-e2e/.env.example b/console/tooling-e2e/.env.example
index b64d7c188..96b3a7008 100644
--- a/console/tooling-e2e/.env.example
+++ b/console/tooling-e2e/.env.example
@@ -1,13 +1,21 @@
# Copy to .env (gitignored) and fill in. Never commit real values.
-SWITCH_API_URL=http://127.0.0.1:54111
+# `just e2e-fast` needs none of this: it sets everything for its own stack.
+
+# fast | provider. Unset, every end-to-end scenario skips.
+SWITCH_E2E_LANE=fast
+# ephemeral | local | dev | staging | prod. prod is read-only for the harness.
+SWITCH_E2E_ENVIRONMENT=local
+
+SWITCH_API_URL=http://127.0.0.1:8000
SWITCH_GATEWAY_ADMIN_PASSWORD=
SWITCH_AGENT_REGISTRATION_TOKEN=
-MATTERMOST_URL=http://127.0.0.1:54112
+# All three or none. Without them, scenarios that need Mattermost skip.
+MATTERMOST_URL=http://127.0.0.1:8065
MATTERMOST_TOKEN=
MATTERMOST_TEAM=switch
-# Optional. Working directory of the session under test — the harness writes the
-# agent's credentials to <dir>/.switch/agents/<agent-name>.json when set.
+# Provider lane only. Working directory of the session under test — the harness
+# writes the agent's credentials to <dir>/.switch/agents/<agent-name>.json when set.
# SWITCH_E2E_AGENT_DIR=
diff --git a/console/tooling-e2e/README.md b/console/tooling-e2e/README.md
index 77dd3fcde..069675ac5 100644
--- a/console/tooling-e2e/README.md
+++ b/console/tooling-e2e/README.md
@@ -1,80 +1,167 @@
# `@switch-console/tooling-e2e`
-An end-to-end harness that sends **real messages through a real Switch server and
-a real Mattermost bridge** to exercise Switch Console's provider-backed sessions.
-
-It plays the human: it registers a throwaway agent, creates a room (and with it a
-Mattermost channel), posts `@agent …` into that channel, and waits for the
-agent's bot to post back. Nothing here talks to Switch Console directly — if a
-scenario passes, an operator doing the same thing by hand would have seen the
-same result.
-
-Every artefact it creates is prefixed `e2e-` and removed on teardown.
+End-to-end tests for Switch: **real messages through a real Switch server**, and
+through a real Mattermost bridge where a scenario crosses one. One package, two
+runner modes:
+
+- **The fast lane** needs no provider, no Switch Console and no LLM key. The
+ harness plays every participant over plain HTTP: it registers throwaway
+ agents, creates rooms, posts as agents and as a Mattermost human, and asserts
+ on what arrives. It carries no secrets a fork lacks, so it can gate pull
+ requests.
+- **The provider lane** needs a real agent session (Switch Console or its
+ sidecar) attending the room, and checks what that session does: answering,
+ asking a question, asking for approval, being interrupted.
+
+Nothing here talks to Switch Console directly. If a scenario passes, an operator
+doing the same thing by hand would have seen the same result.
## Running it
+The fast lane against a throwaway stack built from this checkout (fresh
+Postgres and Mattermost, migrations, switch-core on the host), torn down
+afterwards. Needs Docker, `uv`, `jq`, and your local stack stopped:
+
```bash
-# from console/tooling-e2e
-SWITCH_E2E=1 ../node_modules/.bin/vitest run # everything
-SWITCH_E2E=1 ../node_modules/.bin/vitest run src/loopback.integration.test.ts
-SWITCH_E2E=1 ../node_modules/.bin/vitest run src/run.integration.test.ts
+just e2e-fast # from the repo root
+just e2e-fast --keep # leave the stack running afterwards
+just e2e-fast -- -t mattermost # arguments after -- go to vitest
```
-This package is not yet listed in `console/pnpm-workspace.yaml`, so it has no
-`node_modules` of its own and runs against the workspace root's hoisted `vitest`
-and `typescript`. Adding `- 'tooling-e2e'` to that file's `packages:` list makes
-the usual forms work instead:
+Its log, JUnit report and the stack's per-run secrets go to `build/e2e/ephemeral/`.
+The repository's own `.env` is never read or written.
+
+Against a stack you already have, set the configuration below and run vitest
+from this directory:
```bash
-pnpm --filter @switch-console/tooling-e2e test
-pnpm --filter @switch-console/tooling-e2e cleanup
+SWITCH_E2E_LANE=fast pnpm test
+SWITCH_E2E_LANE=provider pnpm test
```
-Housekeeping after an interrupted run — deletes every `e2e-*` agent and room on
+With `SWITCH_E2E_LANE` unset every end-to-end scenario is skipped, which is
+what `pnpm -r test` and PR CI see. The unit tests (configuration, the read-only
+guard, naming, scenario headers) run either way.
+
+Housekeeping after an interrupted run deletes every `e2e-*` agent and room on
the Switch server and archives every `e2e-*` Mattermost channel, and nothing
-else:
+else. It is also what a scheduled wipe of a deployed environment runs:
```bash
-SWITCH_E2E=1 node --experimental-strip-types src/cleanup.ts
+SWITCH_E2E_LANE=fast pnpm cleanup
+```
+
+Typecheck: `pnpm typecheck`.
+
+## Layout
+
+| Path | What it holds |
+| --- | --- |
+| `src/core/` | Configuration, run naming, the read-only guard, scenario headers and admission, lane resolution |
+| `src/clients/` | The Switch and Mattermost HTTP clients |
+| `src/fixtures/` | Throwaway agents and rooms for a scenario, the bridge-readiness probe, teardown |
+| `src/fast/` | Fast-lane scenarios, one file per area, listed in `index.ts` |
+| `src/provider/` | The provider lane: Console credential files, the shared agent and channel, the scenarios, `seed.ts` |
+| `scripts/ephemeral-stack.sh` | The throwaway-stack runner behind `just e2e-fast` |
+
+## Scenarios and their headers
+
+Every scenario, in either lane, carries a header:
+
+```ts
+{
+ id: 'addressed-delivery', // kebab-case, unique across both lanes
+ covers: ['POST /agents/{agent_id}/message', 'list_rooms'],
+ needsProvider: false, // true puts it in the provider lane
+ environments: ['ephemeral', 'local', 'dev', 'staging'],
+ owner: 'platform', // a team or role, never a person
+ requires: ['mattermost'], // what must be configured
+}
```
-Typecheck: `../node_modules/.bin/tsc --noEmit -p tsconfig.json`.
+`covers` names agent operations from the registry
+(`core/switch_core/bridges/agent/operations/definitions.py`) and HTTP routes
+as `METHOD /path/{param}`, with the server's own parameter names. It is what a
+coverage gate reads, without running anything. Headers are validated by a unit
+test, so a malformed one fails `pnpm test`.
+
+A scenario runs when its lane is the one asked for, the target environment is
+in `environments`, and its `requires` are configured. Otherwise it shows as
+**skipped, with the reason**, never as passed. Under `SWITCH_E2E_STRICT=1` (how
+`just e2e-fast` and CI run) a missing prerequisite, missing configuration or an
+unreachable server **fails** the run instead, so a job that ran nothing cannot
+go green.
+
+A scenario that changes server-global state (a feature flag, say) lists only
+`ephemeral`. One that writes anything never lists `prod`.
+
+### Fast lane
+
+Plain code. A scenario gets fixtures that create agents and rooms under names
+unique to the run and the scenario, and removes them afterwards whether it
+passed or not. A teardown that leaves something behind fails the scenario.
+
+| Scenario | What it drives | Needs |
+| --- | --- | --- |
+| `addressed-delivery` | One agent addresses another in a room with no bridge; the recipient's notification carries the room, the sender and the addressed flag. | — |
+| `unaddressed-history` | Five unaddressed messages arrive in the recipient's room history in the order sent, and none of them as a notification. | — |
+| `operations-front-door` | The operations registry over HTTP: it lists the operations, `list_rooms` returns the room, `read_context` returns a message posted in it. | — |
+| `mattermost-roundtrip` | A room on the Mattermost bridge: an addressed post from the human reaches the agent, and the agent's reply appears as its bot. | Mattermost |
+
+If `mattermost-roundtrip` passes and a provider-lane scenario fails, the fault
+is in the agent session, not in Switch, the bridge or the harness.
+
+## Naming, isolation and the read-only guard
+
+Everything the harness creates is named `e2e-<runId><scope>-<role>`: the run,
+a letter per scenario, then what the thing is for (`e2e-k3x9q2a-snd`). Names go
+through one helper, so no scenario builds a raw name, and the helper refuses a
+name too long for a Mattermost bot username (22 characters) rather than letting
+bot creation fail quietly on the bridge.
+
+The harness's agents are created owner-only, like every agent, and the
+fixtures open their addressing through the gateway: their senders are other
+throwaway agents and an unclaimed Mattermost user, whose messages would
+otherwise be refused rather than delivered.
+
+With `SWITCH_E2E_ENVIRONMENT=prod` the clients refuse every request but reads
+and logging in before it leaves the process, so a wrong base URL fails a test
+rather than writing to production.
## Configuration
Values come from the process environment, falling back to a **gitignored `.env`**
-beside this file. Never commit one.
+beside this file. Never commit one. `just e2e-fast` sets all of them itself.
| Variable | Required | Meaning |
| --- | --- | --- |
-| `SWITCH_E2E` | yes | Must be `1`. Without it the whole suite skips. |
+| `SWITCH_E2E_LANE` | to run anything | `fast` or `provider`. Unset, every end-to-end scenario skips. |
+| `SWITCH_E2E_ENVIRONMENT` | yes | `ephemeral`, `local`, `dev`, `staging` or `prod`. Decides which scenarios run, and makes `prod` read-only. |
| `SWITCH_API_URL` | yes | Switch origin — serves both `/agents/…` and `/gateway/…`. |
| `SWITCH_GATEWAY_ADMIN_EMAIL` | yes | Gateway admin account. |
| `SWITCH_GATEWAY_ADMIN_PASSWORD` | yes | Its password. |
-| `SWITCH_AGENT_REGISTRATION_TOKEN` | yes | Registration API key; mints the throwaway agent. |
-| `MATTERMOST_URL` | yes | Mattermost origin. |
-| `MATTERMOST_TOKEN` | yes | Personal access token of the **human** the harness posts as. |
-| `MATTERMOST_TEAM` | yes | Team **slug** (e.g. `switch`), not its display name. |
-| `SWITCH_E2E_AGENT_DIR` | no | Working directory of the session under test. When set, the agent's credentials are written into it (see below) and it is sent to Switch as the agent's `repo_dir`. |
-| `SWITCH_E2E_KEEP` | no | `1` leaves the agent, room and channel behind for inspection. |
-| `SWITCH_E2E_MANIFEST` | no | File the setup records its agent and room in, so a second process can reuse them instead of registering its own. See "Seeding first" below. |
-| `SWITCH_E2E_REPLY_TIMEOUT_MS` | no | How long a scenario waits for the agent (default 5 min). |
-| `SWITCH_E2E_SESSION_TIMEOUT_MS` | no | How long setup waits for a live session (default 2 min). |
-| `SWITCH_E2E_AGENT_TYPE` | no | `opencode` (default), `claude-code`, or `codex` — which provider's runtime is under test. Decides the agent name prefix, the Switch known-agent type, and (for `claude-code`) the extra files written into the working dir. |
-| `SWITCH_E2E_QUESTION_MODE` | no | `numbered` (default, every agent type) or `prose`; see the `question` row below. |
-
-The suite **skips, with the reason printed**, when `SWITCH_E2E` is unset, when
-configuration is missing (it names every missing variable at once), or when
-Switch or Mattermost is unreachable. A skip always means "this machine is not set
-up", never "the thing under test is broken".
+| `SWITCH_AGENT_REGISTRATION_TOKEN` | yes | Registration API key; mints the throwaway agents. |
+| `MATTERMOST_URL` | for Mattermost scenarios | Mattermost origin. |
+| `MATTERMOST_TOKEN` | for Mattermost scenarios | Token of the **human** the harness posts as (a personal access token, or a login session token). |
+| `MATTERMOST_TEAM` | for Mattermost scenarios | Team **slug** (e.g. `switch`), not its display name. |
+| `SWITCH_E2E_STRICT` | no | `1` turns every "cannot run" into a failure. CI sets it. |
+| `SWITCH_E2E_KEEP` | no | `1` leaves agents, rooms and channels behind for inspection. |
+| `SWITCH_E2E_AGENT_DIR` | provider lane | Working directory of the session under test. When set, the agent's credentials are written into it (see below) and it is sent to Switch as the agent's `repo_dir`. |
+| `SWITCH_E2E_MANIFEST` | provider lane | File the setup records its agent and room in, so a second process can reuse them. See "Seeding first". |
+| `SWITCH_E2E_REPLY_TIMEOUT_MS` | provider lane | How long a scenario waits for the agent (default 5 min). |
+| `SWITCH_E2E_SESSION_TIMEOUT_MS` | provider lane | How long setup waits for a live session (default 2 min). |
+| `SWITCH_E2E_AGENT_TYPE` | provider lane | `opencode` (default), `claude-code`, `codex`, `antigravity` or `cursor` — which provider's runtime is under test. |
+| `SWITCH_E2E_QUESTION_MODE` | provider lane | `numbered` (default) or `prose`; see the `question` row below. |
+
+Mattermost is all or nothing: none of its three variables means "no Mattermost
+here"; some of them is reported as a mistake.
+
+# The provider lane
## What must be running
-`loopback.integration.test.ts` needs nothing but the stack: it plays both the
-human and the agent, and proves the harness's own plumbing.
-
-`run.integration.test.ts` needs a **Switch Console session for the agent it just
-registered**. Setup polls the agent's status in the room and refuses to run the
+The provider lane needs Mattermost and a **Switch Console session for the agent
+it just registered**. Setup polls the agent's status in the room and refuses to run the
scenarios until Switch reports it `live` (a session is attending the room) or
`dormant` (an auto-session watcher is running and will spawn one on demand) —
otherwise you get four identical timeouts instead of one clear message.
@@ -116,19 +203,19 @@ database, and it reads that database at launch. So the agent has to exist before
the console starts, and the console has to be up before the first scenario
posts — three steps, in three processes.
-`src/seed.ts` is the first of them. It does exactly what `beforeAll` does —
+`src/provider/seed.ts` is the first of them. It does exactly what `beforeAll` does —
register the agent, create the room and channel, write the credentials — and
records the result in `SWITCH_E2E_MANIFEST`, above all the agent's API key,
which registration returns exactly once:
```bash
-SWITCH_E2E=1 SWITCH_E2E_MANIFEST=/tmp/e2e.json SWITCH_E2E_AGENT_DIR=/tmp/e2e-work \
- node --experimental-strip-types src/seed.ts
+SWITCH_E2E_LANE=provider SWITCH_E2E_MANIFEST=/tmp/e2e.json SWITCH_E2E_AGENT_DIR=/tmp/e2e-work \
+ node --experimental-strip-types src/provider/seed.ts
# add the agent to the console's database, pointed at that working directory,
# with provider_config {"version":"2","providerId":"opencode","values":{}};
# start Switch Console
-SWITCH_E2E=1 SWITCH_E2E_MANIFEST=/tmp/e2e.json SWITCH_E2E_AGENT_DIR=/tmp/e2e-work \
- ../node_modules/.bin/vitest run src/run.integration.test.ts
+SWITCH_E2E_LANE=provider SWITCH_E2E_MANIFEST=/tmp/e2e.json SWITCH_E2E_AGENT_DIR=/tmp/e2e-work \
+ pnpm test src/provider
```
For Claude Code, add `SWITCH_E2E_AGENT_TYPE=claude-code` to both commands and
@@ -147,7 +234,7 @@ the room before the greeting proceeds.
Setup finds the manifest, reuses what is in it rather than registering a second
agent, and deletes it during teardown — so a manifest never outlives its room.
-## The scenarios
+## The provider scenarios
| Scenario | What it drives | Passes when |
| --- | --- | --- |
@@ -202,6 +289,8 @@ POST /gateway/rooms {name, description, bridge_id, agent_n
Mattermost channel and adds the bot
DELETE /gateway/rooms/{id}
DELETE /gateway/agents/by-name/{name} (admin teardown, no agent key needed)
+PUT /gateway/agents/{id}/addressing-policy {"policy": null} -> anyone may address it
+POST /gateway/rooms with internal_only: true and no bridge_id -> a room with no channel
```
### Switch agent bridge
@@ -222,6 +311,10 @@ GET /agents/{id}/rooms/{room}/history?limit= -> {"events":[{sender,sender_n
POST /agents/{id}/message {"room_id":"…","content":"…"}
-> {"ok":true,"event_id":"$…"}
POST /agents/{id}/watch/heartbeat the auto_session "I am watching" beat
+GET /agents/{id}/ops -> {"operations": {<name>: {…}, …}} the registry, keyed by name
+POST /agents/{id}/ops/{operation} body = the operation's arguments
+ -> {"result": …}. Operations that act on "the room you are connected to"
+ need a live connection; those taking the room as an argument do not.
DELETE /agents/{id} agent's own key only, not the registration token
```
@@ -259,8 +352,8 @@ GET /api/v4/channels/{id}/posts?since=<ms> -> {order:[…newest first…],
`!interrupt` addresses nobody and the admin bot says so. `!interrupt-all-agents`
is the room-wide form. The same shape applies to `!reset` and `!compact`.
- Agent names must be valid Mattermost usernames — lowercase, 3–22 characters,
- `[a-z0-9._-]` — because they become bot usernames. `e2e-opencode-` plus a
- 6-character run id fits.
+ `[a-z0-9._-]` — because they become bot usernames. The naming helper
+ (`src/core/run.ts`) enforces it.
## Two traps this harness works around
@@ -287,7 +380,7 @@ turn.
Switch posts its "I'm not online in this room" onboarding notice **as the agent's
own bot account**, not as the Switch Admin bot. Any scenario that accepts "some
post from the bot" therefore passes with no session running at all. Those phrases
-are listed in `NO_SESSION_MARKERS` (`src/scenarios.ts`), never count as a match,
+are listed in `NO_SESSION_MARKERS` (`src/provider/scenarios.ts`), never count as a match,
and are reported as "no live Switch Console session" rather than as a timeout.
Keep that list in step with the wording in
`core/switch_core/gateway/known_agents.py` and
diff --git a/console/tooling-e2e/package.json b/console/tooling-e2e/package.json
index 0ed977d9f..c327f7f8b 100644
--- a/console/tooling-e2e/package.json
+++ b/console/tooling-e2e/package.json
@@ -3,7 +3,7 @@
"version": "0.0.0",
"private": true,
"type": "module",
- "description": "End-to-end harness that drives real messages through a Switch server and a Mattermost bridge to exercise Switch Console's provider-backed sessions.",
+ "description": "End-to-end harness for Switch: a provider-free fast lane over plain HTTP, and a provider lane that drives real agent sessions through Switch Console.",
"scripts": {
"test": "vitest run",
"test:e2e": "vitest run",
diff --git a/console/tooling-e2e/scripts/ephemeral-stack.sh b/console/tooling-e2e/scripts/ephemeral-stack.sh
new file mode 100755
index 000000000..384d1e75a
--- /dev/null
+++ b/console/tooling-e2e/scripts/ephemeral-stack.sh
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+# Run the harness against a fresh, throwaway Switch stack built from this
+# checkout: new Postgres and Mattermost volumes, migrations, switch-core on the
+# host, then the chosen lane, then teardown.
+#
+# just e2e-fast the fast lane (no provider, no Console)
+# just e2e-fast --keep leave the stack running afterwards
+# just e2e-fast -- <vitest args> pass arguments through to vitest
+#
+# Secrets are generated per run into a temporary env file. The repository's
+# own .env is never read or written, so this cannot rotate a developer's live
+# secrets or pick up a stale value from them.
+#
+# Ports 5432, 8000, 8065 and 8081 must be free: stop a local stack first.
+# Everything the run writes goes to build/e2e/ephemeral/ (or $E2E_OUT).
+set -euo pipefail
+
+REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+HARNESS="$REPO/console/tooling-e2e"
+OUT="${E2E_OUT:-$REPO/build/e2e/ephemeral}"
+LANE="${SWITCH_E2E_LANE:-fast}"
+KEEP=0
+VITEST_ARGS=()
+while [ $# -gt 0 ]; do
+ case "$1" in
+ --keep) KEEP=1 ;;
+ --) shift; VITEST_ARGS=("$@"); break ;;
+ *) echo "unknown option: $1" >&2; exit 2 ;;
+ esac
+ shift
+done
+
+for tool in docker uv openssl curl jq; do
+ command -v "$tool" >/dev/null || { echo "$tool is required but not on PATH." >&2; exit 1; }
+done
+for port in 5432 8000 8065 8081; do
+ if lsof -nP -iTCP:"$port" -sTCP:LISTEN >/dev/null 2>&1; then
+ echo "Port $port is in use. Stop your local stack first ('just down', and any 'just run')." >&2
+ exit 1
+ fi
+done
+
+rm -rf "$OUT"; mkdir -p "$OUT"
+ENV_FILE="$OUT/stack.env"
+cp "$REPO/.env.example" "$ENV_FILE"
+for key in DB_PASSWORD DB_OWNER_PASSWORD AGENT_REGISTRATION_TOKEN GATEWAY_ADMIN_PASSWORD \
+ MATTERMOST_ADMIN_PASSWORD MATTERMOST_USER_PASSWORD; do
+ sed -i.bak "s|^${key}=.*|${key}=$(openssl rand -hex 24)|" "$ENV_FILE"
+done
+sed -i.bak "s|^SECRET_KEYS=.*|SECRET_KEYS=e2e:$(openssl rand -hex 32)|" "$ENV_FILE"
+rm -f "$ENV_FILE.bak"
+chmod 600 "$ENV_FILE"
+set -a; source "$ENV_FILE"; set +a
+
+cd "$REPO"
+COMPOSE=(docker compose -p switch-e2e-ephemeral -f deploy/local/docker-compose.yml
+ --env-file "$ENV_FILE" --project-directory .)
+SERVER_PID=""
+cleanup() {
+ local status=$?
+ if [ "$KEEP" = 0 ]; then
+ [ -n "$SERVER_PID" ] && kill "$SERVER_PID" 2>/dev/null && wait "$SERVER_PID" 2>/dev/null
+ "${COMPOSE[@]}" down -v >/dev/null 2>&1 || true
+ rm -f "$ENV_FILE"
+ else
+ echo "Left running: switch-core :8000 (pid $SERVER_PID), Mattermost :8065."
+ echo "Secrets for this stack: $ENV_FILE. Tear down with:"
+ echo " kill $SERVER_PID; ${COMPOSE[*]} down -v"
+ fi
+ exit "$status"
+}
+trap cleanup EXIT
+
+echo "Starting a fresh stack (Postgres, Mattermost)…"
+"${COMPOSE[@]}" down -v >/dev/null 2>&1 || true
+"${COMPOSE[@]}" up -d --build >"$OUT/compose-up.log" 2>&1 \
+ || { echo "docker compose up failed; see $OUT/compose-up.log" >&2; exit 1; }
+
+for _ in $(seq 60); do
+ "${COMPOSE[@]}" exec -T postgres pg_isready -q >/dev/null 2>&1 && break; sleep 1
+done
+uv run --project core alembic -c core/alembic.ini upgrade head >"$OUT/migrate.log" 2>&1 \
+ || { echo "migrations failed; see $OUT/migrate.log" >&2; exit 1; }
+
+echo "Starting switch-core from this checkout…"
+GATEWAY_COOKIE_SECURE=false uv run --project core python -m switch_core.main \
+ >"$OUT/switch-core.log" 2>&1 &
+SERVER_PID=$!
+for _ in $(seq 120); do curl -sf localhost:8000/health >/dev/null && break; sleep 1; done
+curl -sf localhost:8000/health >/dev/null \
+ || { echo "switch-core did not come up; see $OUT/switch-core.log" >&2; exit 1; }
+
+echo "Waiting for the Mattermost bridge to be registered…"
+for _ in $(seq 150); do
+ "${COMPOSE[@]}" logs setup 2>&1 | grep -q "Setup complete" && break; sleep 2
+done
+"${COMPOSE[@]}" logs setup 2>&1 | grep -q "Setup complete" \
+ || { "${COMPOSE[@]}" logs setup >"$OUT/setup.log" 2>&1
+ echo "Mattermost setup never completed; see $OUT/setup.log" >&2; exit 1; }
+
+# The harness posts as the Mattermost human the setup container created. A
+# session token from logging in authenticates exactly as a personal access
+# token does, and dies with the stack.
+MATTERMOST_TOKEN="$(curl -sf -D - -o /dev/null -H 'Content-Type: application/json' \
+ -d "$(jq -n --arg u "$MATTERMOST_USER" --arg p "$MATTERMOST_USER_PASSWORD" '{login_id: $u, password: $p}')" \
+ http://localhost:8065/api/v4/users/login | tr -d '\r' | awk -F': ' 'tolower($1)=="token" {print $2}')"
+[ -n "$MATTERMOST_TOKEN" ] || { echo "could not log in to Mattermost as $MATTERMOST_USER" >&2; exit 1; }
+
+echo "Running the $LANE lane…"
+status=0
+(
+ cd "$HARNESS"
+ SWITCH_E2E_LANE="$LANE" \
+ SWITCH_E2E_ENVIRONMENT=ephemeral \
+ SWITCH_E2E_STRICT=1 \
+ SWITCH_API_URL=http://localhost:8000 \
+ SWITCH_GATEWAY_ADMIN_EMAIL="$GATEWAY_ADMIN_EMAIL" \
+ SWITCH_GATEWAY_ADMIN_PASSWORD="$GATEWAY_ADMIN_PASSWORD" \
+ SWITCH_AGENT_REGISTRATION_TOKEN="$AGENT_REGISTRATION_TOKEN" \
+ MATTERMOST_URL=http://localhost:8065 \
+ MATTERMOST_TOKEN="$MATTERMOST_TOKEN" \
+ MATTERMOST_TEAM="$MATTERMOST_TEAM_NAME" \
+ pnpm exec vitest run --reporter=verbose --reporter=junit \
+ --outputFile.junit="$OUT/junit.xml" ${VITEST_ARGS[@]+"${VITEST_ARGS[@]}"}
+) || status=$?
+
+if [ "$status" != 0 ]; then
+ echo "The $LANE lane failed. switch-core log: $OUT/switch-core.log" >&2
+fi
+exit "$status"
diff --git a/console/tooling-e2e/src/cleanup.ts b/console/tooling-e2e/src/cleanup.ts
index d574f8105..0948d054b 100644
--- a/console/tooling-e2e/src/cleanup.ts
+++ b/console/tooling-e2e/src/cleanup.ts
@@ -1,31 +1,23 @@
/**
- * Remove artefacts a crashed or interrupted run left behind: Switch agents named
- * `e2e-*`, Switch rooms named `e2e-*`, and Mattermost channels named `e2e-*`.
+ * Remove artefacts a crashed or interrupted run left behind: Switch agents,
+ * Switch rooms and Mattermost channels whose names start with `e2e-`.
*
- * Scoped strictly to the `e2e-` prefix — an operator's real agents (`switchdev`
- * and friends) and their rooms are never in scope, and the script prints
- * everything it is about to touch.
+ * Scoped strictly to that prefix — an operator's real agents and their rooms are
+ * never in scope, and the script prints everything it touches. It is also what
+ * a scheduled wipe of a deployed environment runs.
*
- * SWITCH_E2E=1 node --experimental-strip-types src/cleanup.ts
+ * SWITCH_E2E_LANE=fast node --experimental-strip-types src/cleanup.ts
*/
-import { loadEnv } from './env.ts';
-import { MattermostClient } from './mattermost-client.ts';
-import { SwitchClient } from './switch-client.ts';
-
-const PREFIX = 'e2e-';
+import { ARTIFACT_PREFIX } from './core/run.ts';
+import { message, resolveLane } from './core/suite.ts';
export async function cleanup(log: (message: string) => void = console.log): Promise<void> {
- const env = loadEnv();
- const switchClient = new SwitchClient({
- apiUrl: env.switchApiUrl,
- registrationToken: env.agentRegistrationToken,
- gatewayAdminEmail: env.gatewayAdminEmail,
- gatewayAdminPassword: env.gatewayAdminPassword,
- });
- const mattermost = new MattermostClient({ url: env.mattermostUrl, token: env.mattermostToken });
+ const lane = await resolveLane('fast');
+ if (!lane.target) throw new Error(`Cannot clean up: ${lane.skip}`);
+ const { env, switchClient, mattermost } = lane.target;
for (const room of await switchClient.listRooms()) {
- if (!room.name.startsWith(PREFIX)) continue;
+ if (!room.name.startsWith(ARTIFACT_PREFIX)) continue;
try {
await switchClient.deleteRoom(room.id);
log(`deleted room ${room.name} (${room.id})`);
@@ -35,7 +27,7 @@ export async function cleanup(log: (message: string) => void = console.log): Pro
}
for (const agent of await switchClient.listAgents()) {
- if (!agent.name.startsWith(PREFIX)) continue;
+ if (!agent.name.startsWith(ARTIFACT_PREFIX)) continue;
try {
await switchClient.deleteAgentByName(agent.name);
log(`deleted agent ${agent.name} (${agent.id})`);
@@ -44,9 +36,13 @@ export async function cleanup(log: (message: string) => void = console.log): Pro
}
}
- const team = await mattermost.findTeam(env.mattermostTeam);
+ if (!mattermost || !env.mattermost) {
+ log('no Mattermost configured — skipping channels');
+ return;
+ }
+ const team = await mattermost.findTeam(env.mattermost.team);
for (const channel of await mattermost.listTeamChannels(team.id)) {
- if (!channel.name.startsWith(PREFIX)) continue;
+ if (!channel.name.startsWith(ARTIFACT_PREFIX)) continue;
try {
await mattermost.archiveChannel(channel.id);
log(`archived channel ${channel.name} (${channel.id})`);
@@ -56,10 +52,6 @@ export async function cleanup(log: (message: string) => void = console.log): Pro
}
}
-function message(error: unknown): string {
- return error instanceof Error ? error.message : String(error);
-}
-
if (import.meta.url === `file://${process.argv[1]}`) {
cleanup().catch((error: unknown) => {
console.error(message(error));
diff --git a/console/tooling-e2e/src/mattermost-client.ts b/console/tooling-e2e/src/clients/mattermost-client.ts
similarity index 93%
rename from console/tooling-e2e/src/mattermost-client.ts
rename to console/tooling-e2e/src/clients/mattermost-client.ts
index 4db7ef0d9..a2c31f788 100644
--- a/console/tooling-e2e/src/mattermost-client.ts
+++ b/console/tooling-e2e/src/clients/mattermost-client.ts
@@ -1,3 +1,6 @@
+import type { Environment, MattermostEnv } from '../core/env.ts';
+import { assertAllowed } from '../core/guard.ts';
+
/**
* Mattermost REST client for the harness — the *human* half of the loop.
*
@@ -56,18 +59,15 @@ export class MattermostHttpError extends Error {
}
}
-/** Mattermost usernames: lowercase, 3-22 chars, `[a-z0-9._-]`, must start alphanumeric. */
-export function isValidMattermostUsername(name: string): boolean {
- return /^[a-z0-9][a-z0-9._-]{2,21}$/.test(name);
-}
-
export class MattermostClient {
private readonly baseUrl: string;
private readonly token: string;
+ private readonly environment: Environment;
- constructor(params: { url: string; token: string }) {
- this.baseUrl = params.url.endsWith('/') ? params.url.slice(0, -1) : params.url;
- this.token = params.token;
+ constructor(env: MattermostEnv, environment: Environment) {
+ this.baseUrl = env.url.endsWith('/') ? env.url.slice(0, -1) : env.url;
+ this.token = env.token;
+ this.environment = environment;
}
private async request<T>(
@@ -76,6 +76,7 @@ export class MattermostClient {
body?: unknown,
options: { allow404?: boolean } = {}
): Promise<T> {
+ assertAllowed(this.environment, method, `/api/v4${path}`);
const headers: Record<string, string> = {
Authorization: `Bearer ${this.token}`,
Accept: 'application/json',
@@ -137,10 +138,7 @@ export class MattermostClient {
/** `GET /api/v4/teams/{team_id}/channels` — the team's public channels. */
async listTeamChannels(teamId: string): Promise<MattermostChannel[]> {
- return this.request<MattermostChannel[]>(
- 'GET',
- `/teams/${teamId}/channels?per_page=200`
- );
+ return this.request<MattermostChannel[]>('GET', `/teams/${teamId}/channels?per_page=200`);
}
/** `GET /api/v4/channels/{id}`. */
@@ -267,11 +265,7 @@ export class MattermostClient {
* scenario, which asserts the *absence* of further output rather than the
* presence of a reply.
*/
- async postsFrom(
- channelId: string,
- userId: string,
- sinceMs: number
- ): Promise<MattermostPost[]> {
+ async postsFrom(channelId: string, userId: string, sinceMs: number): Promise<MattermostPost[]> {
const posts = await this.postsSince(channelId, sinceMs);
return posts.filter((post) => post.user_id === userId && !post.type?.startsWith('system_'));
}
diff --git a/console/tooling-e2e/src/switch-client.ts b/console/tooling-e2e/src/clients/switch-client.ts
similarity index 83%
rename from console/tooling-e2e/src/switch-client.ts
rename to console/tooling-e2e/src/clients/switch-client.ts
index bf4ddb12e..1b56c7709 100644
--- a/console/tooling-e2e/src/switch-client.ts
+++ b/console/tooling-e2e/src/clients/switch-client.ts
@@ -1,3 +1,6 @@
+import type { Environment, SwitchEnv } from '../core/env.ts';
+import { assertAllowed } from '../core/guard.ts';
+
/**
* Thin client for the two Switch HTTP surfaces the harness talks to, both served
* by the same origin (`SWITCH_API_URL`):
@@ -5,9 +8,11 @@
* - the **agent bridge** at `/agents/...`, authenticated with either the
* registration token (to mint an agent) or an agent's own API key (to act as
* that agent);
- * - the **gateway** at `/gateway/...`, authenticated with an admin bearer token
+ * - the **gateway** at `/gateway/...`, authenticated with the session cookie
* from `POST /gateway/auth/login`.
*
+ * Every request passes the read-only guard first (`core/guard.ts`).
+ *
* Only endpoints this harness has actually exercised against a live server are
* modelled here; see README.md for the verified request/response shapes.
*/
@@ -97,11 +102,12 @@ function truncate(text: string, max: number): string {
return text.length > max ? `${text.slice(0, max)}…` : text;
}
-export interface SwitchClientOptions {
- apiUrl: string;
- registrationToken: string;
- gatewayAdminEmail: string;
- gatewayAdminPassword: string;
+interface RequestInit {
+ token?: string;
+ cookie?: string;
+ headers?: Record<string, string>;
+ body?: unknown;
+ timeoutMs?: number;
}
export class SwitchClient {
@@ -109,33 +115,33 @@ export class SwitchClient {
private readonly registrationToken: string;
private readonly adminEmail: string;
private readonly adminPassword: string;
+ private readonly environment: Environment;
private gatewayCookie: string | null = null;
- constructor(options: SwitchClientOptions) {
- this.apiUrl = options.apiUrl;
- this.registrationToken = options.registrationToken;
- this.adminEmail = options.gatewayAdminEmail;
- this.adminPassword = options.gatewayAdminPassword;
+ constructor(env: SwitchEnv, environment: Environment) {
+ this.apiUrl = env.apiUrl;
+ this.registrationToken = env.agentRegistrationToken;
+ this.adminEmail = env.gatewayAdminEmail;
+ this.adminPassword = env.gatewayAdminPassword;
+ this.environment = environment;
}
// ── plumbing ──────────────────────────────────────────────────────────────
- private async request<T>(
- method: string,
- path: string,
- init: { token?: string; cookie?: string; body?: unknown; timeoutMs?: number } = {}
- ): Promise<T> {
+ private async request<T>(method: string, path: string, init: RequestInit = {}): Promise<T> {
return (await this.rawRequest(method, path, init)).parsed as T;
}
private async rawRequest(
method: string,
path: string,
- init: { token?: string; cookie?: string; body?: unknown; timeoutMs?: number }
+ init: RequestInit
): Promise<{ parsed: unknown; response: Response }> {
+ assertAllowed(this.environment, method, path);
const headers: Record<string, string> = { Accept: 'application/json' };
if (init.token) headers.Authorization = `Bearer ${init.token}`;
if (init.cookie) headers.Cookie = init.cookie;
+ Object.assign(headers, init.headers ?? {});
if (init.body !== undefined) headers['Content-Type'] = 'application/json';
const response = await fetch(`${this.apiUrl}${path}`, {
@@ -204,13 +210,7 @@ export class SwitchClient {
}
async getRoom(roomId: string): Promise<RoomDetail> {
- const row = await this.gateway<RawRoom>('GET', `/rooms/${roomId}`);
- return {
- ...toRoomSummary(row),
- externalChannelId: row.external_channel_id ?? null,
- matrixRoomId: row.matrix_room_id ?? null,
- agentIds: row.agent_ids ?? [],
- };
+ return toRoomDetail(await this.gateway<RawRoom>('GET', `/rooms/${roomId}`));
}
/**
@@ -227,29 +227,27 @@ export class SwitchClient {
*
* `userNames` are platform usernames added to the channel — the harness adds
* itself, since it has to be a channel member to post as the human.
+ *
+ * `bridgeId: null` makes an internal-only room: no platform channel, so the
+ * only participants are agents and the fast lane needs no bridge at all.
*/
async createRoom(params: {
name: string;
description: string;
- bridgeId: string;
+ bridgeId: string | null;
agentNames: string[];
- userNames?: string[];
+ userNames: string[];
}): Promise<RoomDetail> {
const row = await this.gateway<RawRoom>('POST', '/rooms', {
name: params.name,
description: params.description,
- bridge_id: params.bridgeId,
+ ...(params.bridgeId === null
+ ? { internal_only: true }
+ : { bridge_id: params.bridgeId, internal_only: false, channel_type: 'channel_public' }),
agent_names: params.agentNames,
- user_names: params.userNames ?? [],
- channel_type: 'channel_public',
- internal_only: false,
+ user_names: params.userNames,
});
- return {
- ...toRoomSummary(row),
- externalChannelId: row.external_channel_id ?? null,
- matrixRoomId: row.matrix_room_id ?? null,
- agentIds: row.agent_ids ?? [],
- };
+ return toRoomDetail(row);
}
async deleteRoom(roomId: string): Promise<void> {
@@ -265,6 +263,19 @@ export class SwitchClient {
await this.gateway('DELETE', `/agents/by-name/${encodeURIComponent(name)}`);
}
+ /**
+ * `PUT /gateway/agents/{id}/addressing-policy` with `{policy: null}` — clear
+ * the agent's scoped addressing policy, so anyone may address it.
+ *
+ * Agents are created owner-only: only their owner, or agents and identities
+ * the owner has claimed, may address them, and anything else is answered with
+ * a refusal instead of delivered. The harness's senders are other throwaway
+ * agents and an unclaimed platform user, so its agents are opened explicitly.
+ */
+ async openAddressing(agentId: string): Promise<void> {
+ await this.gateway('PUT', `/agents/${agentId}/addressing-policy`, { policy: null });
+ }
+
/**
* The status Switch reports for one agent in one room. `live` means a session
* is attending the room; `dormant` means an auto-session connector is watching
@@ -468,6 +479,40 @@ export class SwitchClient {
return body.event_id;
}
+ // ── agent bridge: the operations front door ───────────────────────────────
+
+ /**
+ * `GET /agents/{id}/ops` — every operation the registry defines, with its
+ * parameters, keyed by name. The same registry serves the MCP server, so what this lists is
+ * what an agent's session sees as tools.
+ */
+ async listOperations(agent: RegisteredAgent): Promise<string[]> {
+ const body = await this.request<{ operations: Record<string, unknown> }>(
+ 'GET',
+ `/agents/${agent.id}/ops`,
+ { token: agent.apiKey }
+ );
+ return Object.keys(body.operations);
+ }
+
+ /**
+ * `POST /agents/{id}/ops/{operation}` — run one registry operation as the
+ * agent, with `args` as its arguments, and return its `result`. Operations that act on "the room you
+ * are connected to" need a live connection and fail without one; the fast
+ * lane calls those that take the room as an argument.
+ */
+ async callOperation<T>(
+ agent: RegisteredAgent,
+ operation: string,
+ args: Record<string, unknown>
+ ): Promise<T> {
+ const body = await this.request<{ result: T }>('POST', `/agents/${agent.id}/ops/${operation}`, {
+ token: agent.apiKey,
+ body: args,
+ });
+ return body.result;
+ }
+
/**
* `POST /agents/{id}/watch/heartbeat` — the room-agnostic "an operator
* connector is watching this agent" beat. Switch Console pings it while its
@@ -494,6 +539,15 @@ interface RawRoom {
agent_statuses?: Record<string, unknown>;
}
+function toRoomDetail(row: RawRoom): RoomDetail {
+ return {
+ ...toRoomSummary(row),
+ externalChannelId: row.external_channel_id ?? null,
+ matrixRoomId: row.matrix_room_id ?? null,
+ agentIds: row.agent_ids ?? [],
+ };
+}
+
function toRoomSummary(row: RawRoom): RoomSummary {
return {
id: row.id,
diff --git a/console/tooling-e2e/src/core/core.test.ts b/console/tooling-e2e/src/core/core.test.ts
new file mode 100644
index 000000000..8d2561b45
--- /dev/null
+++ b/console/tooling-e2e/src/core/core.test.ts
@@ -0,0 +1,161 @@
+import { describe, expect, it } from 'vitest';
+import { FAST_SCENARIOS } from '../fast/index.ts';
+import { SCENARIOS as PROVIDER_SCENARIOS } from '../provider/scenarios.ts';
+import { laneFrom, loadEnv, MissingConfigError, type HarnessEnv } from './env.ts';
+import { assertAllowed, ReadOnlyEnvironmentError } from './guard.ts';
+import { isValidMattermostUsername, RunContext } from './run.ts';
+import {
+ admit,
+ assertValidHeaders,
+ headerProblems,
+ laneOf,
+ type ScenarioHeader,
+} from './scenario.ts';
+
+const BASE = {
+ SWITCH_API_URL: 'http://switch.invalid/',
+ SWITCH_GATEWAY_ADMIN_EMAIL: '[email protected]',
+ SWITCH_GATEWAY_ADMIN_PASSWORD: 'placeholder',
+ SWITCH_AGENT_REGISTRATION_TOKEN: 'placeholder',
+ SWITCH_E2E_ENVIRONMENT: 'ephemeral',
+};
+
+describe('configuration', () => {
+ it('needs no Mattermost, and strips a trailing slash', () => {
+ const env = loadEnv('fast', BASE, {});
+ expect(env.mattermost).toBeNull();
+ expect(env.switch.apiUrl).toBe('http://switch.invalid');
+ expect(env.strict).toBe(false);
+ });
+
+ it('names every missing variable at once', () => {
+ expect(() => loadEnv('fast', {}, {})).toThrow(MissingConfigError);
+ try {
+ loadEnv('fast', {}, {});
+ } catch (error) {
+ expect((error as MissingConfigError).missing).toEqual(Object.keys(BASE));
+ }
+ });
+
+ it('treats a partial Mattermost configuration as a mistake', () => {
+ expect(() => loadEnv('fast', { ...BASE, MATTERMOST_URL: 'http://mm.invalid' }, {})).toThrow(
+ /MATTERMOST_TOKEN, MATTERMOST_TEAM/
+ );
+ });
+
+ it('refuses an unknown environment or lane', () => {
+ expect(() => loadEnv('fast', { ...BASE, SWITCH_E2E_ENVIRONMENT: 'qa' }, {})).toThrow(/qa/);
+ expect(() => laneFrom({ SWITCH_E2E_LANE: 'slow' })).toThrow(/slow/);
+ expect(laneFrom({})).toBeNull();
+ });
+});
+
+describe('read-only guard', () => {
+ it('lets reads and logins through against prod, and nothing else', () => {
+ expect(() => assertAllowed('prod', 'GET', '/gateway/rooms')).not.toThrow();
+ expect(() => assertAllowed('prod', 'POST', '/gateway/auth/login')).not.toThrow();
+ expect(() => assertAllowed('prod', 'POST', '/gateway/rooms')).toThrow(ReadOnlyEnvironmentError);
+ expect(() => assertAllowed('prod', 'DELETE', '/agents/x')).toThrow(ReadOnlyEnvironmentError);
+ expect(() => assertAllowed('prod', 'POST', '/gateway/auth/login/../rooms')).toThrow();
+ });
+
+ it('allows writes everywhere else', () => {
+ for (const environment of ['ephemeral', 'local', 'dev', 'staging'] as const) {
+ expect(() => assertAllowed(environment, 'POST', '/gateway/rooms')).not.toThrow();
+ }
+ });
+});
+
+describe('run naming', () => {
+ it('gives each scope distinct, Mattermost-safe names', () => {
+ const run = new RunContext('abc123');
+ const first = run.scope();
+ const second = run.scope();
+ expect(first.name('snd')).toBe('e2e-abc123a-snd');
+ expect(second.name('snd')).toBe('e2e-abc123b-snd');
+ expect(run.owns('e2e-abc123b-snd')).toBe(true);
+ expect(run.owns('e2e-zzz999a-snd')).toBe(false);
+ });
+
+ it('refuses a name too long for a bot username, or one handed out twice', () => {
+ const scope = new RunContext('abc123').scope();
+ expect(() => scope.name('much-too-long')).toThrow(/Shorten/);
+ scope.name('room');
+ expect(() => scope.name('room')).toThrow(/already/);
+ });
+
+ it('fits every provider agent role', () => {
+ for (const role of ['opencode', 'antigrav', 'cursor', 'claude', 'codex']) {
+ expect(isValidMattermostUsername(`e2e-abc123a-${role}`)).toBe(true);
+ }
+ });
+});
+
+const header = (overrides: Partial<ScenarioHeader> = {}): ScenarioHeader => ({
+ id: 'some-scenario',
+ covers: ['list_rooms', 'POST /agents/{agent_id}/message'],
+ needsProvider: false,
+ environments: ['ephemeral', 'dev'],
+ owner: 'platform',
+ requires: [],
+ ...overrides,
+});
+
+const env = (overrides: Partial<HarnessEnv> = {}): HarnessEnv => ({
+ ...loadEnv('fast', BASE, {}),
+ ...overrides,
+});
+
+describe('scenario headers', () => {
+ it('accepts operation names and routes, and reports every problem', () => {
+ expect(headerProblems(header())).toEqual([]);
+ expect(
+ headerProblems(
+ header({ id: 'Bad Id', covers: ['post message'], environments: [], owner: '' })
+ )
+ ).toHaveLength(4);
+ });
+
+ it('keeps provider scenarios off prod', () => {
+ expect(headerProblems(header({ needsProvider: true, environments: ['prod'] }))).toHaveLength(1);
+ });
+
+ it('refuses a duplicated id', () => {
+ expect(() => assertValidHeaders([header(), header()])).toThrow(/used by another/);
+ });
+
+ it('holds for every scenario in the suite, with ids unique across lanes', () => {
+ const fast = FAST_SCENARIOS.map((scenario) => scenario.header);
+ const provider = PROVIDER_SCENARIOS.map((scenario) => scenario.header);
+ expect(() => assertValidHeaders([...fast, ...provider])).not.toThrow();
+ expect(fast.every((h) => laneOf(h) === 'fast')).toBe(true);
+ expect(provider.every((h) => laneOf(h) === 'provider')).toBe(true);
+ });
+});
+
+describe('admission', () => {
+ it('runs a scenario of this lane and environment with its prerequisites met', () => {
+ expect(admit(header(), env())).toEqual({ run: true });
+ });
+
+ it('passes over the other lane and unlisted environments without failing', () => {
+ expect(admit(header({ needsProvider: true }), env())).toMatchObject({
+ run: false,
+ failure: false,
+ });
+ expect(admit(header(), env({ environment: 'staging' }))).toMatchObject({
+ run: false,
+ failure: false,
+ });
+ });
+
+ it('skips on a missing prerequisite, and fails on one when strict', () => {
+ const needsMattermost = header({ requires: ['mattermost'] });
+ expect(admit(needsMattermost, env())).toMatchObject({ run: false, failure: false });
+ expect(admit(needsMattermost, env({ strict: true }))).toMatchObject({
+ run: false,
+ failure: true,
+ reason: expect.stringContaining('MATTERMOST_URL'),
+ });
+ });
+});
diff --git a/console/tooling-e2e/src/core/env.ts b/console/tooling-e2e/src/core/env.ts
new file mode 100644
index 000000000..b6346ffa1
--- /dev/null
+++ b/console/tooling-e2e/src/core/env.ts
@@ -0,0 +1,285 @@
+import { readFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+/**
+ * Configuration for the harness. Every value is a secret or a deployment
+ * address, so nothing is hard-coded: it comes from the process environment, or
+ * from a gitignored `.env` beside this package. Deployed environments' URLs are
+ * deliberately absent from source.
+ */
+
+/**
+ * Which runner mode a run is in.
+ *
+ * `fast` needs nothing but a Switch server (and Mattermost, for the scenarios
+ * that cross the bridge): no provider, no Console, no LLM key. It is the lane
+ * that can gate pull requests, because it carries no secrets a fork lacks.
+ * `provider` needs a real agent session attending the room, driven by Switch
+ * Console or its sidecar.
+ */
+export type Lane = 'fast' | 'provider';
+
+/**
+ * What the target is. Scenarios declare which they may run against.
+ *
+ * `ephemeral` is a stack created for this run and destroyed after it, so a
+ * scenario may change server-global state there (feature flags, for one).
+ * `local` is a developer's own long-lived stack. `prod` is read-only: the
+ * clients refuse every authenticated write against it.
+ */
+export type Environment = 'ephemeral' | 'local' | 'dev' | 'staging' | 'prod';
+
+/** The Switch known-agent type the provider lane registers its throwaway agent as. */
+export type HarnessAgentType = 'opencode' | 'claude-code' | 'codex' | 'antigravity' | 'cursor';
+
+/**
+ * How the `question` scenario expects the agent to ask.
+ *
+ * `numbered` is the real thing: the provider offers a native ask-the-user tool,
+ * the console relays it into the room as a numbered list, and the answer goes
+ * back by number. It is the default for every agent type. `prose` is the
+ * explicit fallback for a provider whose SDK sessions are offered no such tool:
+ * the agent asks in an ordinary message and is answered with the word. The
+ * round trip being tested (the session is still alive and still connected when
+ * the second message arrives) is the same; the relay path is not exercised.
+ */
+export type QuestionMode = 'numbered' | 'prose';
+
+export interface SwitchEnv {
+ apiUrl: string;
+ gatewayAdminEmail: string;
+ gatewayAdminPassword: string;
+ agentRegistrationToken: string;
+}
+
+export interface MattermostEnv {
+ url: string;
+ /** Token of the **human** the harness posts as. */
+ token: string;
+ /** Team slug (e.g. `switch`), not its display name. */
+ team: string;
+}
+
+/** Settings only the provider lane reads. */
+export interface ProviderEnv {
+ /** Working directory the console session under test runs from. */
+ agentRepoDir: string | null;
+ /**
+ * Where a setup records the agent and room it created, so a later process can
+ * reuse them instead of registering its own. Needed because a Switch Console
+ * session can only be pointed at an agent that already exists: seed first,
+ * start the console, then run the scenarios against the same agent.
+ */
+ manifestPath: string | null;
+ /** Which agent type to register, and so which provider's runtime is exercised. */
+ agentType: HarnessAgentType;
+ /** How `question` expects the clarifying question to arrive. */
+ questionMode: QuestionMode;
+}
+
+export interface HarnessEnv {
+ lane: Lane;
+ environment: Environment;
+ /**
+ * In strict mode a scenario whose prerequisites are missing fails instead of
+ * skipping. CI runs strict, so a misconfigured job goes red rather than
+ * reporting a suite that ran nothing as green.
+ */
+ strict: boolean;
+ /** Leave created agents, rooms and channels behind after the run (for debugging). */
+ keepArtifacts: boolean;
+ switch: SwitchEnv;
+ /** `null` when no Mattermost is configured; scenarios that need one then skip or fail. */
+ mattermost: MattermostEnv | null;
+ provider: ProviderEnv;
+}
+
+const PACKAGE_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..');
+
+/**
+ * Parse `KEY=VALUE` lines. Blank lines and `#` comments are skipped; a value may
+ * be wrapped in single or double quotes. Deliberately tiny — this is the only
+ * reason the package would otherwise need a dotenv dependency.
+ */
+export function parseDotEnv(raw: string): Record<string, string> {
+ const out: Record<string, string> = {};
+ for (const line of raw.split('\n')) {
+ const trimmed = line.trim();
+ if (trimmed === '' || trimmed.startsWith('#')) continue;
+ const eq = trimmed.indexOf('=');
+ if (eq <= 0) continue;
+ const key = trimmed.slice(0, eq).trim();
+ let value = trimmed.slice(eq + 1).trim();
+ if (
+ (value.startsWith('"') && value.endsWith('"')) ||
+ (value.startsWith("'") && value.endsWith("'"))
+ ) {
+ value = value.slice(1, -1);
+ }
+ out[key] = value;
+ }
+ return out;
+}
+
+/** Values from `.env` beside this package, or `{}` when there is no such file. */
+export function readDotEnvFile(file = path.join(PACKAGE_ROOT, '.env')): Record<string, string> {
+ let raw: string;
+ try {
+ raw = readFileSync(file, 'utf8');
+ } catch (error) {
+ if (error instanceof Error && 'code' in error && error.code === 'ENOENT') return {};
+ throw error;
+ }
+ return parseDotEnv(raw);
+}
+
+export class MissingConfigError extends Error {
+ readonly missing: string[];
+ constructor(missing: string[]) {
+ super(
+ `Missing required configuration: ${missing.join(', ')}. ` +
+ `Set them in the environment or in console/tooling-e2e/.env (see README.md).`
+ );
+ this.name = 'MissingConfigError';
+ this.missing = missing;
+ }
+}
+
+type Source = Record<string, string | undefined>;
+
+function reader(source: Source, file: Record<string, string>): (key: string) => string | undefined {
+ return (key) => {
+ const value = source[key] ?? file[key];
+ return value !== undefined && value.trim() !== '' ? value.trim() : undefined;
+ };
+}
+
+/**
+ * The lane this process was asked to run, or `null` when none was — in which
+ * case every end-to-end suite skips, which is what keeps `pnpm -r test` on a
+ * developer machine from reaching for a server.
+ */
+export function laneFrom(source: Source = process.env): Lane | null {
+ const raw = reader(source, {})('SWITCH_E2E_LANE');
+ if (raw === undefined) return null;
+ if (raw === 'fast' || raw === 'provider') return raw;
+ throw new Error(`SWITCH_E2E_LANE must be 'fast' or 'provider' — got '${raw}'.`);
+}
+
+/**
+ * Resolve the harness configuration from `process.env` layered over `.env`.
+ * Throws {@link MissingConfigError} naming every absent variable at once rather
+ * than failing on the first — a first run is usually missing several.
+ *
+ * Mattermost is all-or-nothing: none of its three variables means "no
+ * Mattermost here", some of them is a mistake and is reported as one.
+ */
+export function loadEnv(
+ lane: Lane,
+ source: Source = process.env,
+ file: Record<string, string> = readDotEnvFile()
+): HarnessEnv {
+ const get = reader(source, file);
+
+ const required = {
+ SWITCH_API_URL: get('SWITCH_API_URL'),
+ SWITCH_GATEWAY_ADMIN_EMAIL: get('SWITCH_GATEWAY_ADMIN_EMAIL'),
+ SWITCH_GATEWAY_ADMIN_PASSWORD: get('SWITCH_GATEWAY_ADMIN_PASSWORD'),
+ SWITCH_AGENT_REGISTRATION_TOKEN: get('SWITCH_AGENT_REGISTRATION_TOKEN'),
+ SWITCH_E2E_ENVIRONMENT: get('SWITCH_E2E_ENVIRONMENT'),
+ };
+ const mattermost = {
+ MATTERMOST_URL: get('MATTERMOST_URL'),
+ MATTERMOST_TOKEN: get('MATTERMOST_TOKEN'),
+ MATTERMOST_TEAM: get('MATTERMOST_TEAM'),
+ };
+ const mattermostGiven = Object.values(mattermost).some((value) => value !== undefined);
+
+ const missing = [
+ ...Object.entries(required),
+ ...(mattermostGiven ? Object.entries(mattermost) : []),
+ ]
+ .filter(([, value]) => value === undefined)
+ .map(([key]) => key);
+ if (missing.length > 0) throw new MissingConfigError(missing);
+
+ return {
+ lane,
+ environment: environmentFrom(required.SWITCH_E2E_ENVIRONMENT as string),
+ strict: get('SWITCH_E2E_STRICT') === '1',
+ keepArtifacts: get('SWITCH_E2E_KEEP') === '1',
+ switch: {
+ apiUrl: stripTrailingSlash(required.SWITCH_API_URL as string),
+ gatewayAdminEmail: required.SWITCH_GATEWAY_ADMIN_EMAIL as string,
+ gatewayAdminPassword: required.SWITCH_GATEWAY_ADMIN_PASSWORD as string,
+ agentRegistrationToken: required.SWITCH_AGENT_REGISTRATION_TOKEN as string,
+ },
+ mattermost: mattermostGiven
+ ? {
+ url: stripTrailingSlash(mattermost.MATTERMOST_URL as string),
+ token: mattermost.MATTERMOST_TOKEN as string,
+ team: mattermost.MATTERMOST_TEAM as string,
+ }
+ : null,
+ provider: {
+ agentRepoDir: get('SWITCH_E2E_AGENT_DIR') ?? null,
+ manifestPath: get('SWITCH_E2E_MANIFEST') ?? null,
+ agentType: agentTypeFrom(get('SWITCH_E2E_AGENT_TYPE')),
+ questionMode: questionModeFrom(get('SWITCH_E2E_QUESTION_MODE')),
+ },
+ };
+}
+
+const ENVIRONMENTS: readonly Environment[] = ['ephemeral', 'local', 'dev', 'staging', 'prod'];
+
+function environmentFrom(raw: string): Environment {
+ const found = ENVIRONMENTS.find((environment) => environment === raw);
+ if (!found) {
+ throw new Error(
+ `SWITCH_E2E_ENVIRONMENT must be one of ${ENVIRONMENTS.join(', ')} — got '${raw}'.`
+ );
+ }
+ return found;
+}
+
+const AGENT_TYPES: readonly HarnessAgentType[] = [
+ 'opencode',
+ 'claude-code',
+ 'codex',
+ 'antigravity',
+ 'cursor',
+];
+
+function agentTypeFrom(raw: string | undefined): HarnessAgentType {
+ if (raw === undefined) return 'opencode';
+ const found = AGENT_TYPES.find((type) => type === raw);
+ if (!found) {
+ throw new Error(
+ `SWITCH_E2E_AGENT_TYPE must be one of ${AGENT_TYPES.join(', ')} — got '${raw}'.`
+ );
+ }
+ return found;
+}
+
+function questionModeFrom(raw: string | undefined): QuestionMode {
+ if (raw === undefined || raw === 'numbered') return 'numbered';
+ if (raw === 'prose') return 'prose';
+ throw new Error(`SWITCH_E2E_QUESTION_MODE must be 'numbered' or 'prose' — got '${raw}'.`);
+}
+
+export function stripTrailingSlash(url: string): string {
+ return url.endsWith('/') ? url.slice(0, -1) : url;
+}
+
+/**
+ * A message that names the running config without disclosing any of it. Used in
+ * skip reasons and the result table, which are printed to a terminal and may be
+ * pasted into a ticket.
+ */
+export function describeEnv(env: HarnessEnv): string {
+ const mattermost = env.mattermost
+ ? `mattermost=${env.mattermost.url} team=${env.mattermost.team}`
+ : 'mattermost=none';
+ return `lane=${env.lane} environment=${env.environment} switch=${env.switch.apiUrl} ${mattermost}`;
+}
diff --git a/console/tooling-e2e/src/core/guard.ts b/console/tooling-e2e/src/core/guard.ts
new file mode 100644
index 000000000..9e00d0ed0
--- /dev/null
+++ b/console/tooling-e2e/src/core/guard.ts
@@ -0,0 +1,39 @@
+import type { Environment } from './env.ts';
+
+/**
+ * The read-only guard. Against `prod` the clients refuse every request that
+ * could change state, before it leaves the process, so a misconfigured base URL
+ * is a failed test rather than an incident.
+ *
+ * Logging in is the one exemption: it is a `POST` but changes nothing the
+ * product shows, and a read-only smoke still has to authenticate.
+ */
+
+export class ReadOnlyEnvironmentError extends Error {
+ constructor(method: string, path: string, environment: Environment) {
+ super(
+ `Refusing ${method} ${path}: the target environment is '${environment}', which is ` +
+ `read-only for the harness. Only GET requests and logging in are allowed there.`
+ );
+ this.name = 'ReadOnlyEnvironmentError';
+ }
+}
+
+const READ_METHODS = new Set(['GET', 'HEAD', 'OPTIONS']);
+
+/** Paths that are writes by method but not by effect. */
+const LOGIN_PATHS = new Set(['/gateway/auth/login', '/api/v4/users/login']);
+
+export function isReadOnly(environment: Environment): boolean {
+ return environment === 'prod';
+}
+
+/** Throw {@link ReadOnlyEnvironmentError} when `method path` is a write against a read-only environment. */
+export function assertAllowed(environment: Environment, method: string, path: string): void {
+ if (!isReadOnly(environment)) return;
+ const upper = method.toUpperCase();
+ if (READ_METHODS.has(upper)) return;
+ const bare = path.split('?', 1)[0] ?? path;
+ if (upper === 'POST' && LOGIN_PATHS.has(bare)) return;
+ throw new ReadOnlyEnvironmentError(upper, bare, environment);
+}
diff --git a/console/tooling-e2e/src/core/run.ts b/console/tooling-e2e/src/core/run.ts
new file mode 100644
index 000000000..cc796de85
--- /dev/null
+++ b/console/tooling-e2e/src/core/run.ts
@@ -0,0 +1,95 @@
+/**
+ * The run id, and every name built from it.
+ *
+ * On a deployed environment many runs share one tenant over time, so whatever a
+ * run creates has to be recognisable as that run's: list assertions filter on
+ * it, teardown deletes by it, and the scheduled wipe finds leftovers by the
+ * shared prefix. That only holds if no scenario ever builds a raw name itself,
+ * so every name goes through {@link NameScope.name}.
+ *
+ * Names are `e2e-<runId><scope>-<role>`: the run, then a letter per scenario so
+ * two scenarios can both ask for a `sender` without colliding, then what the
+ * thing is for.
+ */
+
+/** Everything the harness creates starts with this. The cleanup script deletes by it. */
+export const ARTIFACT_PREFIX = 'e2e-';
+
+/**
+ * Mattermost usernames: lowercase, 3-22 chars, `[a-z0-9._-]`, must start
+ * alphanumeric. Agent names become Mattermost bot usernames verbatim, so every
+ * agent name has to pass this, bridged or not.
+ */
+export function isValidMattermostUsername(name: string): boolean {
+ return /^[a-z0-9][a-z0-9._-]{2,21}$/.test(name);
+}
+
+/** Lowercase, filesystem- and Mattermost-username-safe run id. */
+export function shortId(): string {
+ return Math.random().toString(36).slice(2, 8).padEnd(6, '0');
+}
+
+const SCOPE_LETTERS = 'abcdefghijklmnopqrstuvwxyz';
+
+export class RunContext {
+ readonly runId: string;
+ private scopes = 0;
+
+ constructor(runId: string = shortId()) {
+ if (!/^[a-z0-9]{6}$/.test(runId)) {
+ throw new Error(`Run id '${runId}' must be six lowercase alphanumerics.`);
+ }
+ this.runId = runId;
+ }
+
+ /** A fresh set of names for one scenario (or one setup), distinct from every other scope's. */
+ scope(): NameScope {
+ const letter = SCOPE_LETTERS[this.scopes];
+ if (letter === undefined) {
+ throw new Error(`Run ${this.runId} ran out of name scopes (${SCOPE_LETTERS.length}).`);
+ }
+ this.scopes += 1;
+ return new NameScope(this.runId, letter);
+ }
+
+ /** Whether a name belongs to this run. */
+ owns(name: string): boolean {
+ return name.startsWith(`${ARTIFACT_PREFIX}${this.runId}`);
+ }
+}
+
+export class NameScope {
+ private readonly used = new Set<string>();
+
+ constructor(
+ readonly runId: string,
+ private readonly letter: string
+ ) {}
+
+ /**
+ * The name for an agent, room or channel.
+ *
+ * `role` is kept short because the result has to fit a Mattermost username;
+ * a name that does not is refused here rather than failing bot creation on
+ * the bridge, where it surfaces only as a room that is never provisioned.
+ * Asking for the same role twice is refused too: two artefacts sharing a name
+ * would make teardown and every by-name lookup ambiguous.
+ */
+ name(role: string): string {
+ const name = `${ARTIFACT_PREFIX}${this.runId}${this.letter}-${role}`;
+ if (!isValidMattermostUsername(name)) {
+ throw new Error(
+ `Derived name '${name}' is not a valid Mattermost username ` +
+ `(lowercase, 3-22 chars, [a-z0-9._-]). Shorten the role '${role}'.`
+ );
+ }
+ if (this.used.has(name)) throw new Error(`Name '${name}' was already handed out.`);
+ this.used.add(name);
+ return name;
+ }
+
+ /** A marker string unique to this scope, for grepping a transcript. */
+ marker(label: string): string {
+ return `SWITCH_E2E_${label.toUpperCase()}_${this.runId.toUpperCase()}${this.letter.toUpperCase()}`;
+ }
+}
diff --git a/console/tooling-e2e/src/core/scenario.ts b/console/tooling-e2e/src/core/scenario.ts
new file mode 100644
index 000000000..b7f5d1944
--- /dev/null
+++ b/console/tooling-e2e/src/core/scenario.ts
@@ -0,0 +1,128 @@
+import type { Environment, HarnessEnv, Lane } from './env.ts';
+
+/**
+ * The machine-readable header every scenario carries, in either lane.
+ *
+ * It is what lets tooling answer questions about the suite without running it:
+ * which lane a scenario belongs to (`needsProvider`), where it may run
+ * (`environments`), who to ask when it breaks (`owner`), and what it exercises
+ * (`covers`), which is what the coverage gate reads.
+ */
+export interface ScenarioHeader {
+ /** Kebab-case, unique across the suite. */
+ id: string;
+ /**
+ * What the scenario exercises: agent-operation names from the registry
+ * (`core/switch_core/bridges/agent/operations/definitions.py`, e.g.
+ * `list_rooms`) and HTTP routes as `METHOD /path/{param}` with the server's
+ * own parameter names (e.g. `POST /agents/{agent_id}/message`).
+ */
+ covers: readonly string[];
+ /** `true` puts the scenario in the provider lane: it needs a real agent session. */
+ needsProvider: boolean;
+ /**
+ * Where it may run. A scenario that changes server-global state (a feature
+ * flag) lists only `ephemeral`. One that writes anything never lists `prod`.
+ */
+ environments: readonly Environment[];
+ /** Who answers for it — a team or role, not a person. */
+ owner: string;
+ /** What has to be configured for it to mean anything. */
+ requires: readonly Prerequisite[];
+}
+
+/**
+ * `mattermost`: a Mattermost and a human token on it are configured.
+ * `agent-dir`: the provider session's working directory is on this machine.
+ */
+export type Prerequisite = 'mattermost' | 'agent-dir';
+
+const OPERATION = /^[a-z][a-z0-9_]*$/;
+const ROUTE = /^(GET|POST|PUT|PATCH|DELETE) \/\S*$/;
+const ID = /^[a-z0-9]+(-[a-z0-9]+)*$/;
+
+/** Every problem with a header, so a bad one is fixed in one pass. Empty when it is valid. */
+export function headerProblems(header: ScenarioHeader): string[] {
+ const problems: string[] = [];
+ if (!ID.test(header.id)) problems.push(`id '${header.id}' is not kebab-case`);
+ if (header.covers.length === 0) problems.push('covers is empty');
+ for (const entry of header.covers) {
+ if (!OPERATION.test(entry) && !ROUTE.test(entry)) {
+ problems.push(`covers entry '${entry}' is neither an operation name nor 'METHOD /path'`);
+ }
+ }
+ if (header.environments.length === 0) problems.push('environments is empty');
+ if (header.owner.trim() === '') problems.push('owner is empty');
+ if (header.needsProvider && header.environments.includes('prod')) {
+ problems.push('a provider scenario writes to the room, so it cannot list prod');
+ }
+ return problems;
+}
+
+/** Throw naming every problem in every header, plus any id used twice. */
+export function assertValidHeaders(headers: readonly ScenarioHeader[]): void {
+ const problems: string[] = [];
+ const seen = new Set<string>();
+ for (const header of headers) {
+ for (const problem of headerProblems(header)) problems.push(`${header.id}: ${problem}`);
+ if (seen.has(header.id)) problems.push(`${header.id}: id used by another scenario`);
+ seen.add(header.id);
+ }
+ if (problems.length > 0) {
+ throw new Error(`Invalid scenario headers:\n ${problems.join('\n ')}`);
+ }
+}
+
+export function laneOf(header: ScenarioHeader): Lane {
+ return header.needsProvider ? 'provider' : 'fast';
+}
+
+/**
+ * Whether a scenario runs here, and if not, whether that is a failure.
+ *
+ * A scenario from the other lane, or one that does not list this environment,
+ * is simply not this run's business. A missing prerequisite is different: the
+ * scenario belongs here and cannot run, which skips on a developer machine and
+ * fails in strict mode — CI must not report a scenario that never ran as green.
+ */
+export type Admission = { run: true } | { run: false; reason: string; failure: boolean };
+
+export function admit(header: ScenarioHeader, env: HarnessEnv): Admission {
+ if (laneOf(header) !== env.lane) {
+ return { run: false, reason: `belongs to the ${laneOf(header)} lane`, failure: false };
+ }
+ if (!header.environments.includes(env.environment)) {
+ return {
+ run: false,
+ reason: `does not run against '${env.environment}' (only ${header.environments.join(', ')})`,
+ failure: false,
+ };
+ }
+ const missing = header.requires.filter((prerequisite) => !satisfied(prerequisite, env));
+ if (missing.length > 0) {
+ return {
+ run: false,
+ reason: `needs ${missing.map(describePrerequisite).join(' and ')}`,
+ failure: env.strict,
+ };
+ }
+ return { run: true };
+}
+
+function satisfied(prerequisite: Prerequisite, env: HarnessEnv): boolean {
+ switch (prerequisite) {
+ case 'mattermost':
+ return env.mattermost !== null;
+ case 'agent-dir':
+ return env.provider.agentRepoDir !== null;
+ }
+}
+
+function describePrerequisite(prerequisite: Prerequisite): string {
+ switch (prerequisite) {
+ case 'mattermost':
+ return 'Mattermost (MATTERMOST_URL, MATTERMOST_TOKEN, MATTERMOST_TEAM)';
+ case 'agent-dir':
+ return 'the session working directory (SWITCH_E2E_AGENT_DIR)';
+ }
+}
diff --git a/console/tooling-e2e/src/core/suite.ts b/console/tooling-e2e/src/core/suite.ts
new file mode 100644
index 000000000..18c6ddf99
--- /dev/null
+++ b/console/tooling-e2e/src/core/suite.ts
@@ -0,0 +1,70 @@
+import { MattermostClient } from '../clients/mattermost-client.ts';
+import { SwitchClient } from '../clients/switch-client.ts';
+import { laneFrom, loadEnv, type HarnessEnv, type Lane } from './env.ts';
+
+/**
+ * Whether this process runs a lane's suite at all, and the clients it runs with.
+ */
+export interface LaneTarget {
+ env: HarnessEnv;
+ switchClient: SwitchClient;
+ mattermost: MattermostClient | null;
+}
+
+export type LaneResolution = { target: LaneTarget; skip: null } | { target: null; skip: string };
+
+/**
+ * Resolve a lane: configuration, then reachability.
+ *
+ * Not asked for this lane (`SWITCH_E2E_LANE` unset or naming the other one) is
+ * always a skip — that is `pnpm -r test` on a developer machine. Asked for it
+ * but unable to run (configuration missing, a server unreachable) skips with the
+ * reason printed, or **throws** under `SWITCH_E2E_STRICT=1`, so CI cannot pass a
+ * suite that never ran.
+ */
+export async function resolveLane(lane: Lane): Promise<LaneResolution> {
+ const asked = laneFrom();
+ if (asked !== lane) {
+ return {
+ target: null,
+ skip: asked === null ? 'SWITCH_E2E_LANE is not set' : `SWITCH_E2E_LANE is '${asked}'`,
+ };
+ }
+ const strict = process.env.SWITCH_E2E_STRICT === '1';
+ const unable = (reason: string): LaneResolution => {
+ if (strict) throw new Error(`The ${lane} lane cannot run (SWITCH_E2E_STRICT=1): ${reason}`);
+ return { target: null, skip: reason };
+ };
+
+ let env: HarnessEnv;
+ try {
+ env = loadEnv(lane);
+ } catch (error) {
+ return unable(message(error));
+ }
+
+ const switchClient = new SwitchClient(env.switch, env.environment);
+ try {
+ await switchClient.health();
+ } catch (error) {
+ return unable(`Switch server at ${env.switch.apiUrl} is unreachable: ${message(error)}`);
+ }
+
+ let mattermost: MattermostClient | null = null;
+ if (env.mattermost) {
+ mattermost = new MattermostClient(env.mattermost, env.environment);
+ try {
+ await mattermost.me();
+ } catch (error) {
+ return unable(
+ `Mattermost at ${env.mattermost.url} is unreachable or MATTERMOST_TOKEN is invalid: ${message(error)}`
+ );
+ }
+ }
+
+ return { target: { env, switchClient, mattermost }, skip: null };
+}
+
+export function message(error: unknown): string {
+ return error instanceof Error ? error.message : String(error);
+}
diff --git a/console/tooling-e2e/src/env.ts b/console/tooling-e2e/src/env.ts
deleted file mode 100644
index f24dad4de..000000000
--- a/console/tooling-e2e/src/env.ts
+++ /dev/null
@@ -1,177 +0,0 @@
-import { readFileSync } from 'node:fs';
-import path from 'node:path';
-import { fileURLToPath } from 'node:url';
-
-/**
- * Configuration for the harness. Every value is a secret or a deployment
- * address, so nothing is hard-coded: it comes from the process environment, or
- * from a gitignored `.env` beside this package.
- */
-/** The Switch known-agent type the harness registers its throwaway agent as. */
-export type HarnessAgentType = 'opencode' | 'claude-code' | 'codex' | 'antigravity' | 'cursor';
-
-/**
- * How the `question` scenario expects the agent to ask.
- *
- * `numbered` is the real thing: the provider offers a native ask-the-user tool,
- * the console relays it into the room as a numbered list, and the answer goes
- * back by number. It is the default for every agent type. `prose` is the
- * explicit fallback for a provider whose SDK sessions are offered no such tool:
- * the agent asks in an ordinary message and is answered with the word. The
- * round trip being tested (the session is still alive and still connected when
- * the second message arrives) is the same; the relay path is not exercised.
- */
-export type QuestionMode = 'numbered' | 'prose';
-
-export interface HarnessEnv {
- switchApiUrl: string;
- gatewayAdminEmail: string;
- gatewayAdminPassword: string;
- agentRegistrationToken: string;
- mattermostUrl: string;
- mattermostToken: string;
- mattermostTeam: string;
- /** Working directory the console session under test runs OpenCode from. */
- agentRepoDir: string | null;
- /**
- * Where a setup records the agent and room it created, so a later process can
- * reuse them instead of registering its own. Needed because a Switch Console
- * session can only be pointed at an agent that already exists: seed first,
- * start the console, then run the scenarios against the same agent.
- */
- manifestPath: string | null;
- /** Leave the channel and agent behind after the run (for debugging). */
- keepArtifacts: boolean;
- /** Which agent type to register, and so which provider's runtime is exercised. */
- agentType: HarnessAgentType;
- /** How `question` expects the clarifying question to arrive. */
- questionMode: QuestionMode;
-}
-
-const PACKAGE_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
-
-/**
- * Parse `KEY=VALUE` lines. Blank lines and `#` comments are skipped; a value may
- * be wrapped in single or double quotes. Deliberately tiny — this is the only
- * reason the package would otherwise need a dotenv dependency.
- */
-export function parseDotEnv(raw: string): Record<string, string> {
- const out: Record<string, string> = {};
- for (const line of raw.split('\n')) {
- const trimmed = line.trim();
- if (trimmed === '' || trimmed.startsWith('#')) continue;
- const eq = trimmed.indexOf('=');
- if (eq <= 0) continue;
- const key = trimmed.slice(0, eq).trim();
- let value = trimmed.slice(eq + 1).trim();
- if (
- (value.startsWith('"') && value.endsWith('"')) ||
- (value.startsWith("'") && value.endsWith("'"))
- ) {
- value = value.slice(1, -1);
- }
- out[key] = value;
- }
- return out;
-}
-
-/** Values from `.env` beside this package, or `{}` when there is no such file. */
-export function readDotEnvFile(file = path.join(PACKAGE_ROOT, '.env')): Record<string, string> {
- try {
- return parseDotEnv(readFileSync(file, 'utf8'));
- } catch {
- return {};
- }
-}
-
-export class MissingConfigError extends Error {
- readonly missing: string[];
- constructor(missing: string[]) {
- super(
- `Missing required configuration: ${missing.join(', ')}. ` +
- `Set them in the environment or in console/tooling-e2e/.env (see README.md).`
- );
- this.name = 'MissingConfigError';
- this.missing = missing;
- }
-}
-
-/**
- * Resolve the harness configuration from `process.env` layered over `.env`.
- * Throws {@link MissingConfigError} naming every absent variable at once rather
- * than failing on the first — a first run is usually missing several.
- */
-export function loadEnv(source: Record<string, string | undefined> = process.env): HarnessEnv {
- const file = readDotEnvFile();
- const get = (key: string): string | undefined => {
- const value = source[key] ?? file[key];
- return value !== undefined && value.trim() !== '' ? value.trim() : undefined;
- };
-
- const required = {
- SWITCH_API_URL: get('SWITCH_API_URL'),
- SWITCH_GATEWAY_ADMIN_EMAIL: get('SWITCH_GATEWAY_ADMIN_EMAIL'),
- SWITCH_GATEWAY_ADMIN_PASSWORD: get('SWITCH_GATEWAY_ADMIN_PASSWORD'),
- SWITCH_AGENT_REGISTRATION_TOKEN: get('SWITCH_AGENT_REGISTRATION_TOKEN'),
- MATTERMOST_URL: get('MATTERMOST_URL'),
- MATTERMOST_TOKEN: get('MATTERMOST_TOKEN'),
- MATTERMOST_TEAM: get('MATTERMOST_TEAM'),
- };
-
- const missing = Object.entries(required)
- .filter(([, value]) => value === undefined)
- .map(([key]) => key);
- if (missing.length > 0) throw new MissingConfigError(missing);
-
- return {
- switchApiUrl: stripTrailingSlash(required.SWITCH_API_URL as string),
- gatewayAdminEmail: required.SWITCH_GATEWAY_ADMIN_EMAIL as string,
- gatewayAdminPassword: required.SWITCH_GATEWAY_ADMIN_PASSWORD as string,
- agentRegistrationToken: required.SWITCH_AGENT_REGISTRATION_TOKEN as string,
- mattermostUrl: stripTrailingSlash(required.MATTERMOST_URL as string),
- mattermostToken: required.MATTERMOST_TOKEN as string,
- mattermostTeam: required.MATTERMOST_TEAM as string,
- agentRepoDir: get('SWITCH_E2E_AGENT_DIR') ?? null,
- manifestPath: get('SWITCH_E2E_MANIFEST') ?? null,
- keepArtifacts: (get('SWITCH_E2E_KEEP') ?? '') === '1',
- agentType: agentTypeFrom(get('SWITCH_E2E_AGENT_TYPE')),
- questionMode: get('SWITCH_E2E_QUESTION_MODE') === 'prose' ? 'prose' : 'numbered',
- };
-}
-
-const AGENT_TYPES: readonly HarnessAgentType[] = [
- 'opencode',
- 'claude-code',
- 'codex',
- 'antigravity',
- 'cursor',
-];
-
-function agentTypeFrom(raw: string | undefined): HarnessAgentType {
- if (raw === undefined) return 'opencode';
- const found = AGENT_TYPES.find((type) => type === raw);
- if (!found) {
- throw new Error(
- `SWITCH_E2E_AGENT_TYPE must be one of ${AGENT_TYPES.join(', ')} — got '${raw}'.`
- );
- }
- return found;
-}
-
-export function stripTrailingSlash(url: string): string {
- return url.endsWith('/') ? url.slice(0, -1) : url;
-}
-
-/** Lowercase, filesystem- and Mattermost-username-safe run id. */
-export function shortId(): string {
- return Math.random().toString(36).slice(2, 8);
-}
-
-/**
- * A message that names the running config without disclosing any of it. Used in
- * skip reasons and the result table, which are printed to a terminal and may be
- * pasted into a ticket.
- */
-export function describeEnv(env: HarnessEnv): string {
- return `switch=${env.switchApiUrl} mattermost=${env.mattermostUrl} team=${env.mattermostTeam}`;
-}
diff --git a/console/tooling-e2e/src/fast/agent-messaging.ts b/console/tooling-e2e/src/fast/agent-messaging.ts
new file mode 100644
index 000000000..ccd2203b2
--- /dev/null
+++ b/console/tooling-e2e/src/fast/agent-messaging.ts
@@ -0,0 +1,103 @@
+import { expect } from 'vitest';
+import { eventText, sleep } from '../clients/switch-client.ts';
+import { WRITABLE, type FastScenario } from './scenario.ts';
+
+/**
+ * Agent-to-agent messaging in a room with no bridge: the core of the bus with
+ * nothing in front of it. Two throwaway agents, one internal-only room, the
+ * harness acting as both.
+ */
+
+const NOTIFICATION_DEADLINE_MS = 30_000;
+
+/**
+ * An addressed message reaches its recipient's notification stream, carrying
+ * the room, the sender, and the addressed flag.
+ */
+export const addressedDelivery: FastScenario = {
+ header: {
+ id: 'addressed-delivery',
+ covers: [
+ 'POST /agents/register-known',
+ 'POST /gateway/rooms',
+ 'POST /agents/{agent_id}/message',
+ 'GET /agents/{agent_id}/notifications',
+ ],
+ needsProvider: false,
+ environments: WRITABLE,
+ owner: 'platform',
+ requires: [],
+ },
+ async run({ fixtures, names, switchClient }) {
+ const sender = await fixtures.agent('snd');
+ const recipient = await fixtures.agent('rcv');
+ const room = await fixtures.internalRoom('room', [sender, recipient]);
+ expect(room.agentIds).toEqual(expect.arrayContaining([sender.id, recipient.id]));
+
+ const marker = names.marker('addr');
+ const eventId = await switchClient.sendMessage(sender, room.id, `@${recipient.name} ${marker}`);
+ expect(eventId).toBeTruthy();
+
+ const { match, seen } = await switchClient.waitForNotification(
+ recipient,
+ (event) => eventText(event).includes(marker),
+ NOTIFICATION_DEADLINE_MS
+ );
+ expect(
+ match,
+ `no notification carried ${marker}; saw ${JSON.stringify(seen).slice(0, 800)}`
+ ).not.toBeNull();
+ expect(match!.type).toBe('message');
+ expect(match!.room_id).toBe(room.id);
+ expect(match!.payload?.addressed).toBe(true);
+ expect(match!.payload?.sender_name).toBe(sender.name);
+ },
+};
+
+/**
+ * Unaddressed messages are history, not notifications: the recipient's room
+ * history holds every one of them, in the order they were sent, and its
+ * notification stream holds none.
+ *
+ * Ordering is asserted because the bus promises it — `seq` is assigned under a
+ * per-room lock — and a regression there is invisible to every test that sends
+ * one message.
+ */
+export const unaddressedHistory: FastScenario = {
+ header: {
+ id: 'unaddressed-history',
+ covers: [
+ 'POST /agents/{agent_id}/message',
+ 'GET /agents/{agent_id}/rooms/{room_id}/history',
+ 'GET /agents/{agent_id}/notifications',
+ ],
+ needsProvider: false,
+ environments: WRITABLE,
+ owner: 'platform',
+ requires: [],
+ },
+ async run({ fixtures, names, switchClient }) {
+ const sender = await fixtures.agent('snd');
+ const reader = await fixtures.agent('rdr');
+ const room = await fixtures.internalRoom('room', [sender, reader]);
+
+ const marker = names.marker('chat');
+ const sent = Array.from({ length: 5 }, (_, index) => `${marker} #${index + 1}`);
+ for (const body of sent) await switchClient.sendMessage(sender, room.id, body);
+
+ let bodies: string[] = [];
+ const until = Date.now() + NOTIFICATION_DEADLINE_MS;
+ while (Date.now() < until) {
+ const history = await switchClient.roomHistory(reader, room.id, 50);
+ bodies = history.map((event) => event.body).filter((body) => body.includes(marker));
+ if (bodies.length >= sent.length) break;
+ await sleep(1_000);
+ }
+ expect(bodies).toEqual(sent);
+
+ const notified = (await switchClient.pollNotifications(reader, 3)).filter((event) =>
+ eventText(event).includes(marker)
+ );
+ expect(notified, 'an unaddressed message was delivered as a notification').toEqual([]);
+ },
+};
diff --git a/console/tooling-e2e/src/fast/fast.e2e.test.ts b/console/tooling-e2e/src/fast/fast.e2e.test.ts
new file mode 100644
index 000000000..1440f8827
--- /dev/null
+++ b/console/tooling-e2e/src/fast/fast.e2e.test.ts
@@ -0,0 +1,56 @@
+import { beforeAll, describe, expect, it } from 'vitest';
+import { describeEnv } from '../core/env.ts';
+import { RunContext } from '../core/run.ts';
+import { admit, assertValidHeaders } from '../core/scenario.ts';
+import { resolveLane, type LaneResolution } from '../core/suite.ts';
+import { Fixtures } from '../fixtures/fixtures.ts';
+import { FAST_SCENARIOS } from './index.ts';
+
+/**
+ * The fast lane: provider-free scenarios against a Switch server, one fresh
+ * set of agents and rooms per scenario. Runs when `SWITCH_E2E_LANE=fast`.
+ *
+ * A scenario that cannot run here is reported as **skipped**, with the reason,
+ * never as passed — and under `SWITCH_E2E_STRICT=1` a missing prerequisite is a
+ * failure instead.
+ */
+
+assertValidHeaders(FAST_SCENARIOS.map((scenario) => scenario.header));
+
+const run = new RunContext();
+let lane: LaneResolution;
+
+beforeAll(async () => {
+ lane = await resolveLane('fast');
+ if (lane.target) console.log(`fast lane run ${run.runId}: ${describeEnv(lane.target.env)}`);
+}, 60_000);
+
+describe('fast lane', () => {
+ for (const scenario of FAST_SCENARIOS) {
+ it(scenario.header.id, async (context) => {
+ if (!lane.target) return context.skip(lane.skip);
+ const { env, switchClient, mattermost } = lane.target;
+
+ const admission = admit(scenario.header, env);
+ if (!admission.run) {
+ if (admission.failure) {
+ throw new Error(
+ `${scenario.header.id} cannot run (SWITCH_E2E_STRICT=1): ${admission.reason}`
+ );
+ }
+ return context.skip(admission.reason);
+ }
+
+ const names = run.scope();
+ const fixtures = new Fixtures(env, names, switchClient, mattermost);
+ const log = (message: string): void => console.log(`[${scenario.header.id}] ${message}`);
+ try {
+ await scenario.run({ env, names, switchClient, mattermost, fixtures, log });
+ } finally {
+ const leftovers = await fixtures.teardown();
+ for (const line of leftovers) log(`teardown: ${line}`);
+ if (!env.keepArtifacts) expect(leftovers, 'teardown left artefacts behind').toEqual([]);
+ }
+ });
+ }
+});
diff --git a/console/tooling-e2e/src/fast/index.ts b/console/tooling-e2e/src/fast/index.ts
new file mode 100644
index 000000000..c7b7347de
--- /dev/null
+++ b/console/tooling-e2e/src/fast/index.ts
@@ -0,0 +1,12 @@
+import { addressedDelivery, unaddressedHistory } from './agent-messaging.ts';
+import { mattermostRoundtrip } from './mattermost-roundtrip.ts';
+import { operationsFrontDoor } from './operations.ts';
+import type { FastScenario } from './scenario.ts';
+
+/** Every fast-lane scenario. A scenario not listed here does not run. */
+export const FAST_SCENARIOS: readonly FastScenario[] = [
+ addressedDelivery,
+ unaddressedHistory,
+ operationsFrontDoor,
+ mattermostRoundtrip,
+];
diff --git a/console/tooling-e2e/src/fast/mattermost-roundtrip.ts b/console/tooling-e2e/src/fast/mattermost-roundtrip.ts
new file mode 100644
index 000000000..4c09d31b3
--- /dev/null
+++ b/console/tooling-e2e/src/fast/mattermost-roundtrip.ts
@@ -0,0 +1,77 @@
+import { expect } from 'vitest';
+import { eventText } from '../clients/switch-client.ts';
+import { WRITABLE, type FastScenario } from './scenario.ts';
+
+/**
+ * The bridge round trip with **no Switch Console and no model in the loop**:
+ * the harness plays both the human and the agent.
+ *
+ * 1. create a room on the Mattermost bridge and confirm Switch provisions its
+ * channel with the agent in it;
+ * 2. post `@agent …` as a human and confirm the addressed message reaches the
+ * agent-facing notification stream;
+ * 3. `POST /agents/{id}/message` as the agent and confirm it lands back in the
+ * Mattermost channel as the agent's bot.
+ *
+ * If this passes and a provider-lane scenario fails, the fault is in the agent
+ * session, not in Switch, the bridge, or the harness.
+ */
+export const mattermostRoundtrip: FastScenario = {
+ header: {
+ id: 'mattermost-roundtrip',
+ covers: [
+ 'POST /gateway/rooms',
+ 'GET /agents/{agent_id}/notifications',
+ 'POST /agents/{agent_id}/message',
+ ],
+ needsProvider: false,
+ environments: WRITABLE,
+ owner: 'platform',
+ requires: ['mattermost'],
+ },
+ async run({ fixtures, names, switchClient, mattermost, log }) {
+ const agent = await fixtures.agent('loop');
+ const bridged = await fixtures.mattermostRoom('room', [agent]);
+ const bot = bridged.bots.get(agent.id)!;
+ expect(bridged.room.externalChannelId).toBe(bridged.channel.id);
+ expect(bridged.room.agentIds).toContain(agent.id);
+
+ const marker = names.marker('loop');
+ const sinceMs = Date.now();
+ const posted = await mattermost!.post({
+ channelId: bridged.channel.id,
+ message: `@${agent.name} hello ${marker}`,
+ });
+ log(`posted ${posted.id} as ${bridged.human.username}`);
+
+ // ── inbound: room -> Switch -> agent ────────────────────────────────────
+ const inbound = await switchClient.waitForNotification(
+ agent,
+ (event) => eventText(event).includes(marker),
+ 120_000
+ );
+ expect(
+ inbound.match,
+ `the addressed message never reached the agent; saw ${JSON.stringify(inbound.seen).slice(0, 800)}`
+ ).not.toBeNull();
+ expect(inbound.match!.room_id).toBe(bridged.room.id);
+ expect(inbound.match!.payload?.addressed).toBe(true);
+
+ // ── outbound: agent -> Switch -> room ───────────────────────────────────
+ const eventId = await switchClient.sendMessage(agent, bridged.room.id, `ack ${marker}`);
+ expect(eventId).toBeTruthy();
+
+ const outbound = await mattermost!.waitForPost({
+ channelId: bridged.channel.id,
+ fromUserId: bot.id,
+ predicate: (post) => post.message.includes(`ack ${marker}`),
+ sinceMs,
+ deadlineMs: 120_000,
+ });
+ expect(
+ outbound.match,
+ `the agent's reply never appeared as bot ${bot.username}; channel said: ` +
+ outbound.transcript.map((post) => `${post.user_id}: ${post.message}`).join(' | ')
+ ).not.toBeNull();
+ },
+};
diff --git a/console/tooling-e2e/src/fast/operations.ts b/console/tooling-e2e/src/fast/operations.ts
new file mode 100644
index 000000000..5a63dc18f
--- /dev/null
+++ b/console/tooling-e2e/src/fast/operations.ts
@@ -0,0 +1,48 @@
+import { expect } from 'vitest';
+import { WRITABLE, type FastScenario } from './scenario.ts';
+
+/**
+ * The agent-operations front door over HTTP: the same registry the MCP server
+ * serves, called the way a session host's tools are.
+ */
+export const operationsFrontDoor: FastScenario = {
+ header: {
+ id: 'operations-front-door',
+ covers: [
+ 'GET /agents/{agent_id}/ops',
+ 'POST /agents/{agent_id}/ops/{operation}',
+ 'list_rooms',
+ 'read_context',
+ ],
+ needsProvider: false,
+ environments: WRITABLE,
+ owner: 'platform',
+ requires: [],
+ },
+ async run({ fixtures, names, switchClient }) {
+ const agent = await fixtures.agent('ops');
+ const peer = await fixtures.agent('peer');
+ const room = await fixtures.internalRoom('room', [agent, peer]);
+
+ const operations = await switchClient.listOperations(agent);
+ expect(operations).toEqual(
+ expect.arrayContaining(['list_rooms', 'read_context', 'post_message'])
+ );
+
+ const rooms = await switchClient.callOperation<{ room_id: string; name: string }[]>(
+ agent,
+ 'list_rooms',
+ {}
+ );
+ expect(rooms).toContainEqual(expect.objectContaining({ room_id: room.id, name: room.name }));
+
+ const marker = names.marker('ops');
+ await switchClient.sendMessage(peer, room.id, marker);
+ const context = await switchClient.callOperation<{
+ threads: { root: { body: string; sender_name: string } }[];
+ }>(agent, 'read_context', { room_id: room.id, limit: 20 });
+ expect(context.threads.map((thread) => thread.root)).toContainEqual(
+ expect.objectContaining({ body: marker, sender_name: peer.name })
+ );
+ },
+};
diff --git a/console/tooling-e2e/src/fast/scenario.ts b/console/tooling-e2e/src/fast/scenario.ts
new file mode 100644
index 000000000..0df658c1e
--- /dev/null
+++ b/console/tooling-e2e/src/fast/scenario.ts
@@ -0,0 +1,32 @@
+import type { MattermostClient } from '../clients/mattermost-client.ts';
+import type { SwitchClient } from '../clients/switch-client.ts';
+import type { HarnessEnv } from '../core/env.ts';
+import type { NameScope } from '../core/run.ts';
+import type { ScenarioHeader } from '../core/scenario.ts';
+import type { Fixtures } from '../fixtures/fixtures.ts';
+
+/**
+ * A fast-lane scenario: plain code, run against a Switch server over HTTP with
+ * the harness playing every participant. No provider, no Console, no model.
+ *
+ * `run` asserts with `expect` and throws on failure. Everything it creates goes
+ * through `fixtures`, which tears it down afterwards whether or not it passed.
+ */
+export interface FastScenario {
+ header: ScenarioHeader;
+ run(context: FastContext): Promise<void>;
+}
+
+export interface FastContext {
+ env: HarnessEnv;
+ /** Names and markers for this scenario, unique within the run. */
+ names: NameScope;
+ switchClient: SwitchClient;
+ /** Present exactly when the header requires `mattermost`. */
+ mattermost: MattermostClient | null;
+ fixtures: Fixtures;
+ log: (message: string) => void;
+}
+
+/** Where a fast scenario that writes may run: anywhere but prod. */
+export const WRITABLE: ScenarioHeader['environments'] = ['ephemeral', 'local', 'dev', 'staging'];
diff --git a/console/tooling-e2e/src/fixtures/fixtures.ts b/console/tooling-e2e/src/fixtures/fixtures.ts
new file mode 100644
index 000000000..51da21e7b
--- /dev/null
+++ b/console/tooling-e2e/src/fixtures/fixtures.ts
@@ -0,0 +1,237 @@
+import type {
+ MattermostChannel,
+ MattermostClient,
+ MattermostUser,
+} from '../clients/mattermost-client.ts';
+import {
+ sleep,
+ type RegisteredAgent,
+ type RoomDetail,
+ type SwitchClient,
+} from '../clients/switch-client.ts';
+import type { HarnessAgentType, HarnessEnv } from '../core/env.ts';
+import type { NameScope } from '../core/run.ts';
+
+/**
+ * What a scenario stands up, and the teardown that removes it.
+ *
+ * Every artefact is named through the run context and registered for teardown
+ * the moment it exists, so a scenario that fails halfway still cleans up what it
+ * made. Teardown runs newest first — rooms before the agents in them — and
+ * reports every failure rather than stopping at the first, because one leaked
+ * room should not hide another.
+ */
+export class Fixtures {
+ private readonly cleanups: { label: string; run: () => Promise<void> }[] = [];
+
+ constructor(
+ readonly env: HarnessEnv,
+ readonly names: NameScope,
+ readonly switchClient: SwitchClient,
+ readonly mattermost: MattermostClient | null
+ ) {}
+
+ /**
+ * Register an agent named for `role` in this scope.
+ *
+ * `autoSession` is off unless asked for: with it on, Switch expects a
+ * connector to spawn a session when the agent is addressed, and an operator's
+ * Console watching the same server could fight the harness for the room.
+ */
+ async agent(
+ role: string,
+ options: { autoSession?: boolean; agentType?: HarnessAgentType; repoDir?: string } = {}
+ ): Promise<RegisteredAgent> {
+ const name = this.names.name(role);
+ const agent = await this.switchClient.registerKnownAgent({
+ agentType: options.agentType ?? 'opencode',
+ name,
+ description: `Switch end-to-end harness agent (run ${this.names.runId}). Safe to delete.`,
+ options: {
+ auto_session: options.autoSession ?? false,
+ ...(options.repoDir ? { repo_dir: options.repoDir } : {}),
+ },
+ });
+ this.defer(`agent ${name} (${agent.id})`, () => this.switchClient.deleteAgent(agent));
+ await this.switchClient.openAddressing(agent.id);
+ return agent;
+ }
+
+ /** A room with no platform channel: agents only, no bridge involved. */
+ async internalRoom(role: string, agents: RegisteredAgent[]): Promise<RoomDetail> {
+ const name = this.names.name(role);
+ const room = await this.switchClient.createRoom({
+ name,
+ description: `Switch end-to-end harness (run ${this.names.runId}). Safe to delete.`,
+ bridgeId: null,
+ agentNames: agents.map((agent) => agent.name),
+ userNames: [],
+ });
+ this.defer(`room ${name} (${room.id})`, () => this.switchClient.deleteRoom(room.id));
+ if (room.externalChannelId !== null) {
+ throw new Error(`Internal-only room ${room.id} came back with a platform channel.`);
+ }
+ return room;
+ }
+
+ /**
+ * A room on the Mattermost bridge, with the human the harness posts as in its
+ * channel, and the bridge proven live in both directions' first hop.
+ *
+ * Created from the gateway so Switch provisions the channel and adds the bots
+ * itself. The other route — make a channel, add a bot — creates no room when
+ * the bot is the first in a new channel, because nobody's websocket witnesses
+ * the join.
+ */
+ async mattermostRoom(role: string, agents: RegisteredAgent[]): Promise<BridgedRoom> {
+ const mattermost = this.requireMattermost();
+ const human = await mattermost.me();
+ const bridgeId = await this.switchClient.defaultBridgeId('mattermost');
+ // Resolved even though room creation does not need it: it fails fast and
+ // clearly when MATTERMOST_TEAM names a team that does not exist.
+ await mattermost.findTeam(this.mattermostTeam());
+
+ // Registration mints each agent's bot; wait for them so the room's members
+ // and any later `@name` mention have accounts to resolve to.
+ const bots = new Map<string, MattermostUser>();
+ for (const agent of agents) {
+ bots.set(agent.id, await mattermost.waitForBotUser(agent.name, 60_000));
+ }
+
+ const name = this.names.name(role);
+ const room = await this.switchClient.createRoom({
+ name,
+ description: `Switch end-to-end harness (run ${this.names.runId}). Safe to delete.`,
+ bridgeId,
+ agentNames: agents.map((agent) => agent.name),
+ userNames: [human.username],
+ });
+ this.defer(`room ${name} (${room.id})`, () => this.switchClient.deleteRoom(room.id));
+ if (!room.externalChannelId) {
+ throw new Error(
+ `Switch room ${room.id} was created without a Mattermost channel — ` +
+ `the collaboration bridge did not provision one.`
+ );
+ }
+ const channelId = room.externalChannelId;
+ // Deleting the room usually takes the channel with it, so archiving is
+ // normally a no-op; it is the safety net when it does not.
+ this.defer(`channel ${channelId}`, async () => {
+ await mattermost.archiveChannel(channelId).catch(() => undefined);
+ });
+
+ const channel = await mattermost.getChannel(channelId);
+ const bridged: BridgedRoom = { room, channel, human, bots };
+ const witness = agents[0];
+ if (!witness) throw new Error('A bridged room needs at least one agent.');
+ await waitForBridgeReady({
+ switchClient: this.switchClient,
+ mattermost,
+ agent: witness,
+ room,
+ channel,
+ runId: this.names.runId,
+ });
+ return bridged;
+ }
+
+ /** Register a cleanup step for something created outside these helpers. */
+ defer(label: string, run: () => Promise<void>): void {
+ this.cleanups.push({ label, run });
+ }
+
+ /**
+ * Undo everything, newest first. Never throws: a teardown failure must not
+ * mask the scenario's result. Returns what could not be removed, so the
+ * caller can print it and an operator can clean up by hand.
+ */
+ async teardown(): Promise<string[]> {
+ if (this.env.keepArtifacts) {
+ const kept = this.cleanups.map((cleanup) => cleanup.label);
+ this.cleanups.length = 0;
+ return kept.map((label) => `kept (SWITCH_E2E_KEEP=1): ${label}`);
+ }
+ const failures: string[] = [];
+ while (this.cleanups.length > 0) {
+ const cleanup = this.cleanups.pop()!;
+ try {
+ await cleanup.run();
+ } catch (error) {
+ failures.push(
+ `could not remove ${cleanup.label}: ${error instanceof Error ? error.message : String(error)}`
+ );
+ }
+ }
+ return failures;
+ }
+
+ private requireMattermost(): MattermostClient {
+ if (!this.mattermost) {
+ throw new Error(
+ 'This scenario needs Mattermost but none is configured — declare ' +
+ "`requires: ['mattermost']` in its header so it is skipped or failed up front."
+ );
+ }
+ return this.mattermost;
+ }
+
+ private mattermostTeam(): string {
+ if (!this.env.mattermost) throw new Error('No Mattermost configured.');
+ return this.env.mattermost.team;
+ }
+}
+
+export interface BridgedRoom {
+ room: RoomDetail;
+ channel: MattermostChannel;
+ /** The human the harness posts as. */
+ human: MattermostUser;
+ /** Each agent's Mattermost bot, by agent id. A bot's username is the agent's name. */
+ bots: Map<string, MattermostUser>;
+}
+
+/**
+ * Block until a plain channel message actually reaches Switch.
+ *
+ * This is not belt-and-braces. The Mattermost adapter delivers inbound posts off
+ * the *agent bot's own websocket*, and that socket is opened on a background
+ * thread after the bot account is created — the server log shows
+ * `Websocket authentification OK` arriving after the room already exists. A
+ * message posted in that window is silently dropped, which shows up later as a
+ * scenario that times out for no visible reason.
+ *
+ * The probe is deliberately **unaddressed**: it proves Mattermost → Switch
+ * delivery without putting an `@agent` mention on the wire, so it cannot spawn a
+ * console session or consume a turn. It is read back through the agent's room
+ * history, which carries ordinary chatter, rather than the notification stream,
+ * which by design carries only addressed messages.
+ */
+export async function waitForBridgeReady(params: {
+ switchClient: SwitchClient;
+ mattermost: MattermostClient;
+ agent: RegisteredAgent;
+ room: RoomDetail;
+ channel: MattermostChannel;
+ runId: string;
+ deadlineMs?: number;
+}): Promise<void> {
+ const deadlineMs = params.deadlineMs ?? 120_000;
+ const probe = `switch-e2e-bridge-probe ${params.runId}`;
+ const until = Date.now() + deadlineMs;
+
+ while (Date.now() < until) {
+ // Re-post rather than only re-read: if the socket was down when the last
+ // one went out, that message is gone for good and no amount of polling
+ // will produce it.
+ await params.mattermost.post({ channelId: params.channel.id, message: probe });
+ const history = await params.switchClient.roomHistory(params.agent, params.room.id, 50);
+ if (history.some((event) => event.body.includes(probe))) return;
+ await sleep(5_000);
+ }
+
+ throw new Error(
+ `Mattermost -> Switch delivery never came up for room ${params.room.id} ` +
+ `(channel ${params.channel.id}) within ${deadlineMs}ms: a plain channel message ` +
+ `never reached the agent's room history. The agent bot's websocket is probably not connected.`
+ );
+}
diff --git a/console/tooling-e2e/src/loopback.integration.test.ts b/console/tooling-e2e/src/loopback.integration.test.ts
deleted file mode 100644
index 7228fdbc7..000000000
--- a/console/tooling-e2e/src/loopback.integration.test.ts
+++ /dev/null
@@ -1,102 +0,0 @@
-import { afterAll, beforeAll, describe, expect, it } from 'vitest';
-import { describeEnv } from './env.ts';
-import { resolveSkip, setupHarness, teardownHarness, type Harness } from './harness.ts';
-import { eventText } from './switch-client.ts';
-
-/**
- * The harness's own plumbing, proved against a live stack with **no Switch
- * Console and no model in the loop**: the harness plays both parts.
- *
- * 1. register an agent, create a channel, add the agent's bot to it, and confirm
- * Switch provisions a room for that channel;
- * 2. post `@agent …` as a human and confirm the addressed message reaches the
- * agent-facing notification stream;
- * 3. `POST /agents/{id}/message` as the agent and confirm it lands back in the
- * Mattermost channel as the bot.
- *
- * If this file passes and `run.integration.test.ts` fails, the fault is in the
- * console session, not in Switch, the bridge, or the harness.
- */
-
-let harness: Harness | null = null;
-let skip: string | null = null;
-
-beforeAll(async () => {
- const resolved = await resolveSkip();
- skip = resolved.skip;
- if (skip || !resolved.env) return;
- // auto_session off: nothing is expected to spawn a session — this test *is*
- // the agent, so an auto_session profile would only invite the operator's
- // console to fight it for the room.
- harness = await setupHarness(resolved.env, { autoSession: false, prefix: 'e2e-loop' });
- console.log(
- `loopback: ${describeEnv(resolved.env)} agent=${harness.agent.name} (${harness.agent.id}) ` +
- `channel=${harness.channel.name} (${harness.channel.id}) room=${harness.room.id}`
- );
-}, 300_000);
-
-afterAll(async () => {
- await teardownHarness(harness);
-});
-
-describe('Switch <-> Mattermost loopback', () => {
- it('provisions a Switch room when the agent bot joins a Mattermost channel', () => {
- if (skip) return void console.log(`SKIPPED: ${skip}`);
- expect(harness).not.toBeNull();
- expect(harness!.room.externalChannelId).toBe(harness!.channel.id);
- expect(harness!.room.agentIds).toContain(harness!.agent.id);
- });
-
- it('delivers an addressed channel message to the agent, and the agent reply back to the channel', async () => {
- if (skip) return void console.log(`SKIPPED: ${skip}`);
- const h = harness!;
- const marker = `SWITCH_E2E_${h.runId.toUpperCase()}`;
-
- const sinceMs = Date.now();
- const posted = await h.mattermost.post({
- channelId: h.channel.id,
- message: `@${h.agent.name} hello ${marker}`,
- });
- console.log(`loopback: posted ${posted.id} as ${h.human.username}`);
-
- // ── inbound: room -> Switch -> agent ────────────────────────────────────
- const inbound = await h.switch.waitForNotification(
- h.agent,
- (event) => eventText(event).includes(marker),
- 120_000
- );
- if (!inbound.match) {
- throw new Error(
- `Addressed message never reached the agent's notification stream. Saw: ${JSON.stringify(
- inbound.seen
- ).slice(0, 1000)}`
- );
- }
- expect(inbound.match.room_id).toBe(h.room.id);
- console.log(
- `loopback: agent received event type=${inbound.match.type} room=${inbound.match.room_id}`
- );
-
- // ── outbound: agent -> Switch -> room ───────────────────────────────────
- const reply = `ack ${marker}`;
- const eventId = await h.switch.sendMessage(h.agent, h.room.id, reply);
- expect(eventId).toBeTruthy();
-
- const outbound = await h.mattermost.waitForPost({
- channelId: h.channel.id,
- fromUserId: h.bot.id,
- predicate: (post) => post.message.includes(marker),
- sinceMs,
- deadlineMs: 120_000,
- });
- if (!outbound.match) {
- throw new Error(
- `Agent reply never appeared in Mattermost as bot ${h.bot.username}. Transcript: ` +
- outbound.transcript.map((p) => `${p.user_id}: ${p.message}`).join(' | ')
- );
- }
- console.log(
- `loopback: bot post ${outbound.match.id} (matrix event ${eventId}) — round trip closed`
- );
- }, 400_000);
-});
diff --git a/console/tooling-e2e/src/console-credentials.ts b/console/tooling-e2e/src/provider/console-credentials.ts
similarity index 98%
rename from console/tooling-e2e/src/console-credentials.ts
rename to console/tooling-e2e/src/provider/console-credentials.ts
index b4ba8b353..b9f2173de 100644
--- a/console/tooling-e2e/src/console-credentials.ts
+++ b/console/tooling-e2e/src/provider/console-credentials.ts
@@ -1,6 +1,6 @@
import { promises as fs } from 'node:fs';
import path from 'node:path';
-import type { RegisteredAgent } from './switch-client.ts';
+import type { RegisteredAgent } from '../clients/switch-client.ts';
/**
* Write the harness's throwaway agent into a working directory the way Switch
diff --git a/console/tooling-e2e/src/harness.ts b/console/tooling-e2e/src/provider/harness.ts
similarity index 55%
rename from console/tooling-e2e/src/harness.ts
rename to console/tooling-e2e/src/provider/harness.ts
index 25919ab71..17269c4bf 100644
--- a/console/tooling-e2e/src/harness.ts
+++ b/console/tooling-e2e/src/provider/harness.ts
@@ -1,18 +1,25 @@
import { promises as fs } from 'node:fs';
+import type {
+ MattermostChannel,
+ MattermostClient,
+ MattermostUser,
+} from '../clients/mattermost-client.ts';
+import {
+ sleep,
+ type RegisteredAgent,
+ type RoomDetail,
+ type SwitchClient,
+} from '../clients/switch-client.ts';
+import type { HarnessAgentType, HarnessEnv } from '../core/env.ts';
+import { RunContext } from '../core/run.ts';
+import type { LaneTarget } from '../core/suite.ts';
+import { waitForBridgeReady } from '../fixtures/fixtures.ts';
import {
removeAgentCredentials,
removeClaudeAgentDefinition,
writeAgentCredentials,
writeClaudeAgentDefinition,
} from './console-credentials.ts';
-import { loadEnv, shortId, type HarnessAgentType, type HarnessEnv } from './env.ts';
-import {
- isValidMattermostUsername,
- MattermostClient,
- type MattermostChannel,
- type MattermostUser,
-} from './mattermost-client.ts';
-import { sleep, SwitchClient, type RegisteredAgent, type RoomDetail } from './switch-client.ts';
/**
* Everything a scenario needs: the two clients, the throwaway agent, the
@@ -35,142 +42,75 @@ export interface Harness {
credentialsPath?: string;
}
-export interface SetupOptions {
- /**
- * `false` registers the agent with `auto_session` off, so Switch will not
- * expect a connector to spawn a session — used by the loopback check, which
- * plays the agent itself.
- */
- autoSession?: boolean;
- /** Prefix for the agent name and channel name. Must stay short (see below). */
- prefix?: string;
-}
-
-/** The name prefix per agent type, kept inside Mattermost's 22-character cap. */
-const PREFIX_BY_AGENT_TYPE: Record<HarnessAgentType, string> = {
- opencode: 'e2e-opencode',
- antigravity: 'e2e-antigravity',
- cursor: 'e2e-cursor',
- 'claude-code': 'e2e-claude',
- codex: 'e2e-codex',
-};
-
-/**
- * Why the name is built this way: the agent name becomes a **Mattermost bot
- * username** verbatim, and Mattermost caps usernames at 22 characters and allows
- * only `[a-z0-9._-]`. `e2e-opencode-` (13) + a 6-character run id fits with room
- * to spare; anything longer silently fails bot creation on the bridge side and
- * the room is then never provisioned.
- */
-export function agentNameFor(runId: string, prefix = PREFIX_BY_AGENT_TYPE.opencode): string {
- const name = `${prefix}-${runId}`;
- if (!isValidMattermostUsername(name)) {
- throw new Error(
- `Derived agent name '${name}' is not a valid Mattermost bot username ` +
- `(lowercase, 3-22 chars, [a-z0-9._-]). Shorten the prefix.`
- );
- }
- return name;
-}
-
/**
- * Reasons the suite skips rather than fails. A skip is only ever "this machine
- * isn't set up", never "the thing under test is broken".
+ * The agent-name role per agent type. The full name is
+ * `e2e-<runId><scope>-<role>` and becomes a Mattermost bot username, which caps
+ * at 22 characters — so a role can be at most ten.
*/
-export type SkipReason = string;
-
-export async function resolveSkip(): Promise<{ skip: SkipReason | null; env: HarnessEnv | null }> {
- if (process.env.SWITCH_E2E !== '1') {
- return { skip: 'SWITCH_E2E=1 is not set (this suite drives a real Switch server)', env: null };
- }
-
- let env: HarnessEnv;
- try {
- env = loadEnv();
- } catch (error) {
- return { skip: error instanceof Error ? error.message : String(error), env: null };
- }
-
- const client = new SwitchClient({
- apiUrl: env.switchApiUrl,
- registrationToken: env.agentRegistrationToken,
- gatewayAdminEmail: env.gatewayAdminEmail,
- gatewayAdminPassword: env.gatewayAdminPassword,
- });
- try {
- await client.health();
- } catch (error) {
- return {
- skip: `Switch server at ${env.switchApiUrl} is unreachable: ${
- error instanceof Error ? error.message : String(error)
- }`,
- env: null,
- };
- }
-
- const mattermost = new MattermostClient({ url: env.mattermostUrl, token: env.mattermostToken });
- try {
- await mattermost.me();
- } catch (error) {
- return {
- skip: `Mattermost at ${env.mattermostUrl} is unreachable or MATTERMOST_TOKEN is invalid: ${
- error instanceof Error ? error.message : String(error)
- }`,
- env: null,
- };
- }
-
- return { skip: null, env };
-}
+const ROLE_BY_AGENT_TYPE: Record<HarnessAgentType, string> = {
+ opencode: 'opencode',
+ antigravity: 'antigrav',
+ cursor: 'cursor',
+ 'claude-code': 'claude',
+ codex: 'codex',
+};
/**
- * Register a throwaway agent, create a channel, invite the agent's bot into it,
- * and wait for Switch to provision the room.
+ * Register a throwaway agent, create a room on the Mattermost bridge for it,
+ * and wait for the bridge to carry a message.
*
* Order matters and is not interchangeable: the bot account only exists once the
* agent is registered (registration is what calls `create_agent_identity` on
- * every collaboration bridge), and the Switch room only exists once that bot is
- * a channel member.
+ * every collaboration bridge), and the room is created with that bot in it.
+ *
+ * Unlike the fast lane's fixtures this setup outlives a single test: the
+ * provider scenarios share one agent and one channel, and the agent may have
+ * been registered by an earlier process (see {@link HarnessManifest}).
*/
-export async function setupHarness(env: HarnessEnv, options: SetupOptions = {}): Promise<Harness> {
- const reused = await readManifest(env.manifestPath);
- if (reused) return reuseHarness(env, reused);
-
- const runId = shortId();
- const agentName = agentNameFor(runId, options.prefix ?? PREFIX_BY_AGENT_TYPE[env.agentType]);
+export async function setupHarness(target: LaneTarget): Promise<Harness> {
+ const { env, switchClient } = target;
+ const mattermost = target.mattermost;
+ if (!mattermost || !env.mattermost) {
+ throw new Error(
+ 'The provider lane needs Mattermost (MATTERMOST_URL, MATTERMOST_TOKEN, MATTERMOST_TEAM).'
+ );
+ }
+ const reused = await readManifest(env.provider.manifestPath);
+ if (reused) return reuseHarness(target, mattermost, reused);
- const switchClient = new SwitchClient({
- apiUrl: env.switchApiUrl,
- registrationToken: env.agentRegistrationToken,
- gatewayAdminEmail: env.gatewayAdminEmail,
- gatewayAdminPassword: env.gatewayAdminPassword,
- });
- const mattermost = new MattermostClient({ url: env.mattermostUrl, token: env.mattermostToken });
+ const run = new RunContext();
+ const runId = run.runId;
+ const names = run.scope();
+ const agentName = names.name(ROLE_BY_AGENT_TYPE[env.provider.agentType]);
const human = await mattermost.me();
const bridgeId = await switchClient.defaultBridgeId('mattermost');
// Resolved even though the room-create path does not need it: it fails fast
// and clearly when MATTERMOST_TEAM names a team that does not exist.
- await mattermost.findTeam(env.mattermostTeam);
+ await mattermost.findTeam(env.mattermost.team);
// `ClaudeCodeOptions` and `OpencodeOptions` agree on the two keys used here
// (`auto_session`, `repo_dir`), so the payload is the same for both types.
const agent = await switchClient.registerKnownAgent({
- agentType: env.agentType,
+ agentType: env.provider.agentType,
name: agentName,
description: `Switch Console end-to-end harness agent (run ${runId}). Safe to delete.`,
options: {
- auto_session: options.autoSession ?? true,
- ...(env.agentRepoDir ? { repo_dir: env.agentRepoDir } : {}),
+ auto_session: true,
+ ...(env.provider.agentRepoDir ? { repo_dir: env.provider.agentRepoDir } : {}),
},
});
+ // Owner-only by default, and the human the harness posts as is an unclaimed
+ // Mattermost identity, so its messages would be refused rather than delivered.
+ await switchClient.openAddressing(agent.id);
+
// Registration is what mints the bot; wait for it so a later `@name` mention
// has something to resolve to.
const bot = await mattermost.waitForBotUser(agent.name, 60_000);
const room = await switchClient.createRoom({
- name: `e2e-${runId}`,
+ name: names.name('room'),
description: `Switch Console end-to-end harness (run ${runId}). Safe to delete.`,
bridgeId,
agentNames: [agent.name],
@@ -200,17 +140,17 @@ export async function setupHarness(env: HarnessEnv, options: SetupOptions = {}):
// With SWITCH_E2E_AGENT_DIR set, provision the agent's identity into that
// working directory so a session started there IS this agent. Without it the
// operator has to add the agent in Switch Console by hand before the run.
- if (env.agentRepoDir) {
+ if (env.provider.agentRepoDir) {
harness.credentialsPath = await writeAgentCredentials({
- workingDir: env.agentRepoDir,
+ workingDir: env.provider.agentRepoDir,
agent,
- apiEndpoint: env.switchApiUrl,
+ apiEndpoint: env.switch.apiUrl,
});
// A Claude Code session runs *as* a named definition, so the definition has
// to be on disk before the console starts one — see the writer's docblock.
- if (env.agentType === 'claude-code') {
+ if (env.provider.agentType === 'claude-code') {
await writeClaudeAgentDefinition({
- workingDir: env.agentRepoDir,
+ workingDir: env.provider.agentRepoDir,
agentName: agent.name,
description: `Switch Console end-to-end harness agent (run ${runId}).`,
// "Do exactly what you are asked" is load-bearing, and so is the second
@@ -230,8 +170,8 @@ export async function setupHarness(env: HarnessEnv, options: SetupOptions = {}):
}
}
- await waitForBridgeReady(harness);
- await writeManifest(env.manifestPath, harness);
+ await waitForBridgeReady({ switchClient, mattermost, agent, room, channel, runId });
+ await writeManifest(env.provider.manifestPath, harness);
return harness;
}
@@ -277,15 +217,12 @@ async function readManifest(path: string | null): Promise<HarnessManifest | null
* bridge-readiness probe still runs: this process has proved nothing about the
* bot's websocket, and reusing an agent is not evidence its socket is up now.
*/
-async function reuseHarness(env: HarnessEnv, manifest: HarnessManifest): Promise<Harness> {
- const switchClient = new SwitchClient({
- apiUrl: env.switchApiUrl,
- registrationToken: env.agentRegistrationToken,
- gatewayAdminEmail: env.gatewayAdminEmail,
- gatewayAdminPassword: env.gatewayAdminPassword,
- });
- const mattermost = new MattermostClient({ url: env.mattermostUrl, token: env.mattermostToken });
-
+async function reuseHarness(
+ target: LaneTarget,
+ mattermost: MattermostClient,
+ manifest: HarnessManifest
+): Promise<Harness> {
+ const { env, switchClient } = target;
const harness: Harness = {
env,
switch: switchClient,
@@ -298,53 +235,17 @@ async function reuseHarness(env: HarnessEnv, manifest: HarnessManifest): Promise
runId: manifest.runId,
};
- await waitForBridgeReady(harness);
+ await waitForBridgeReady({
+ switchClient,
+ mattermost,
+ agent: harness.agent,
+ room: harness.room,
+ channel: harness.channel,
+ runId: harness.runId,
+ });
return harness;
}
-/**
- * Block until a plain channel message actually reaches Switch.
- *
- * This is not belt-and-braces. The Mattermost adapter delivers inbound posts off
- * the *agent bot's own websocket*, and that socket is opened on a background
- * thread after the bot account is created — the server log shows
- * `Websocket authentification OK` arriving after the room already exists. A
- * message posted in that window is silently dropped, which shows up later as a
- * scenario that times out for no visible reason.
- *
- * The probe is deliberately **unaddressed**: it proves Mattermost → Switch
- * delivery without putting an `@agent` mention on the wire, so it cannot spawn a
- * console session or consume a turn. It is read back through the agent's room
- * history, which carries ordinary chatter, rather than the notification stream,
- * which by design carries only addressed messages.
- */
-export async function waitForBridgeReady(harness: Harness, deadlineMs = 120_000): Promise<void> {
- const probe = `switch-e2e-bridge-probe ${harness.runId}`;
- const until = Date.now() + deadlineMs;
- let posted = false;
-
- while (Date.now() < until) {
- if (!posted) {
- await harness.mattermost.post({ channelId: harness.channel.id, message: probe });
- posted = true;
- }
- const history = await harness.switch.roomHistory(harness.agent, harness.room.id, 50);
- if (history.some((event) => event.body.includes(probe))) return;
-
- // Re-post rather than only re-read: if the socket was down when the first
- // one went out, that message is gone for good and no amount of polling
- // will produce it.
- await sleep(5_000);
- posted = false;
- }
-
- throw new Error(
- `Mattermost -> Switch delivery never came up for room ${harness.room.id} ` +
- `(channel ${harness.channel.id}) within ${deadlineMs}ms: a plain channel message ` +
- `never reached the agent's room history. The agent bot's websocket is probably not connected.`
- );
-}
-
/** Statuses that mean a session is attending — or is about to be spawned for — the room. */
const READY_STATUSES = new Set(['live', 'dormant']);
@@ -392,17 +293,17 @@ export async function teardownHarness(
// Before anything is deleted: a manifest that outlives its agent sends the
// next run at a room that is gone.
- if (harness.env.manifestPath) {
- await fs.rm(harness.env.manifestPath, { force: true }).catch(() => undefined);
+ if (harness.env.provider.manifestPath) {
+ await fs.rm(harness.env.provider.manifestPath, { force: true }).catch(() => undefined);
}
- if (harness.env.agentRepoDir) {
+ if (harness.env.provider.agentRepoDir) {
await removeAgentCredentials({
- workingDir: harness.env.agentRepoDir,
+ workingDir: harness.env.provider.agentRepoDir,
agentName: harness.agent.name,
}).catch(() => undefined);
await removeClaudeAgentDefinition({
- workingDir: harness.env.agentRepoDir,
+ workingDir: harness.env.provider.agentRepoDir,
agentName: harness.agent.name,
}).catch(() => undefined);
}
diff --git a/console/tooling-e2e/src/provider/provider.e2e.test.ts b/console/tooling-e2e/src/provider/provider.e2e.test.ts
new file mode 100644
index 000000000..76f93e93c
--- /dev/null
+++ b/console/tooling-e2e/src/provider/provider.e2e.test.ts
@@ -0,0 +1,86 @@
+import { afterAll, beforeAll, describe, expect, it } from 'vitest';
+import { describeEnv } from '../core/env.ts';
+import { admit, assertValidHeaders } from '../core/scenario.ts';
+import { resolveLane, type LaneResolution } from '../core/suite.ts';
+import { setupHarness, teardownHarness, waitForSession, type Harness } from './harness.ts';
+import { formatResultsTable, SCENARIOS, type ScenarioResult } from './scenarios.ts';
+
+/**
+ * The provider lane. One throwaway agent, one Mattermost channel, the scenarios
+ * run in order against them, then teardown and a summary table. Runs when
+ * `SWITCH_E2E_LANE=provider`.
+ *
+ * A failure only means something when a real agent session is attending the
+ * room: a Switch Console session, or Console's auto-session watcher, for the
+ * agent this setup registers — see README.md, "What must be running".
+ */
+
+assertValidHeaders(SCENARIOS.map((scenario) => scenario.header));
+
+let lane: LaneResolution;
+let harness: Harness | null = null;
+const results: ScenarioResult[] = [];
+
+beforeAll(async () => {
+ lane = await resolveLane('provider');
+ if (!lane.target) return;
+ // Nothing to set up when no scenario can run here — say, a target that
+ // lacks the prerequisites every scenario shares.
+ if (!SCENARIOS.some((scenario) => admit(scenario.header, lane.target!.env).run)) return;
+
+ harness = await setupHarness(lane.target);
+ console.log(
+ [
+ '',
+ `Provider lane run ${harness.runId}`,
+ ` ${describeEnv(lane.target.env)}`,
+ ` agent ${harness.agent.name} (${harness.agent.id})`,
+ ` bot @${harness.bot.username} (${harness.bot.id})`,
+ ` channel ${harness.channel.name} (${harness.channel.id})`,
+ ` room ${harness.room.name} (${harness.room.id})`,
+ '',
+ ` A Switch Console session must now be running for '${harness.agent.name}'.`,
+ '',
+ ].join('\n')
+ );
+
+ const status = await waitForSession(
+ harness,
+ Number(process.env.SWITCH_E2E_SESSION_TIMEOUT_MS ?? 120_000)
+ );
+ console.log(` Switch reports the agent as '${status}' in the room — running scenarios.\n`);
+}, 600_000);
+
+afterAll(async () => {
+ if (results.length > 0) console.log(formatResultsTable(results));
+ await teardownHarness(harness);
+});
+
+describe('provider lane', () => {
+ for (const scenario of SCENARIOS) {
+ it(scenario.header.id, async (context) => {
+ if (!lane.target) return context.skip(lane.skip);
+ const admission = admit(scenario.header, lane.target.env);
+ if (!admission.run) {
+ if (admission.failure) {
+ throw new Error(
+ `${scenario.header.id} cannot run (SWITCH_E2E_STRICT=1): ${admission.reason}`
+ );
+ }
+ return context.skip(admission.reason);
+ }
+ const h = harness!;
+ const result = await scenario.run(h);
+ results.push(result);
+ if (!result.ok) {
+ console.log(
+ `${result.name} transcript:\n` +
+ result.transcript
+ .map((post) => ` [${post.user_id === h.bot.id ? 'bot ' : 'user'}] ${post.message}`)
+ .join('\n')
+ );
+ }
+ expect(result.ok, result.details).toBe(true);
+ });
+ }
+});
diff --git a/console/tooling-e2e/src/scenarios.test.ts b/console/tooling-e2e/src/provider/scenarios.test.ts
similarity index 100%
rename from console/tooling-e2e/src/scenarios.test.ts
rename to console/tooling-e2e/src/provider/scenarios.test.ts
diff --git a/console/tooling-e2e/src/scenarios.ts b/console/tooling-e2e/src/provider/scenarios.ts
similarity index 91%
rename from console/tooling-e2e/src/scenarios.ts
rename to console/tooling-e2e/src/provider/scenarios.ts
index 1a54be982..eeb6b4d5f 100644
--- a/console/tooling-e2e/src/scenarios.ts
+++ b/console/tooling-e2e/src/provider/scenarios.ts
@@ -1,7 +1,8 @@
import { readFile } from 'node:fs/promises';
import path from 'node:path';
+import { sleep, type MattermostPost } from '../clients/mattermost-client.ts';
+import type { ScenarioHeader } from '../core/scenario.ts';
import type { Harness } from './harness.ts';
-import { sleep, type MattermostPost } from './mattermost-client.ts';
/**
* The scenarios. Each one drives the full path
@@ -218,7 +219,7 @@ const CHOSEN = 'green';
*/
export const question: Scenario = (harness) =>
scenario('question', async (record) => {
- const prose = harness.env.questionMode === 'prose';
+ const prose = harness.env.provider.questionMode === 'prose';
const since = await ask(
harness,
prose
@@ -322,7 +323,7 @@ async function markerContents(file: string): Promise<string | null> {
/** A real local command must start and then be cancelled before its final write. */
export const interrupt: Scenario = (harness) =>
scenario('interrupt', async (record) => {
- const dir = harness.env.agentRepoDir;
+ const dir = harness.env.provider.agentRepoDir;
if (!dir)
throw new Error('The interrupt scenario requires SWITCH_E2E_AGENT_DIR on this machine.');
const stem = `interrupt-${harness.runId}-${Date.now()}`;
@@ -365,7 +366,47 @@ export const interrupt: Scenario = (harness) =>
return 'command start confirmed; completion marker absent after 95s';
});
-export const SCENARIOS: Scenario[] = [greet, question, approval, interrupt];
+/**
+ * A provider-lane scenario: its header, and the function that drives it. All of
+ * them share one agent and one channel, set up once per run.
+ */
+export interface ProviderScenario {
+ header: ScenarioHeader;
+ run: Scenario;
+}
+
+/** Writes to a room, so never prod. */
+const PROVIDER_ENVIRONMENTS: ScenarioHeader['environments'] = [
+ 'ephemeral',
+ 'local',
+ 'dev',
+ 'staging',
+];
+
+function providerHeader(
+ id: string,
+ covers: string[],
+ requires: ScenarioHeader['requires']
+): ScenarioHeader {
+ return {
+ id,
+ covers,
+ needsProvider: true,
+ environments: PROVIDER_ENVIRONMENTS,
+ owner: 'console',
+ requires: ['mattermost', ...requires],
+ };
+}
+
+export const SCENARIOS: readonly ProviderScenario[] = [
+ { header: providerHeader('greet', ['connect_to_room', 'post_message'], []), run: greet },
+ { header: providerHeader('question', ['connect_to_room', 'post_message'], []), run: question },
+ { header: providerHeader('approval', ['connect_to_room', 'post_message'], []), run: approval },
+ {
+ header: providerHeader('interrupt', ['connect_to_room', 'post_message'], ['agent-dir']),
+ run: interrupt,
+ },
+];
/** Fixed-width summary of a run, printed at the end of the suite. */
export function formatResultsTable(results: ScenarioResult[]): string {
diff --git a/console/tooling-e2e/src/seed.ts b/console/tooling-e2e/src/provider/seed.ts
similarity index 76%
rename from console/tooling-e2e/src/seed.ts
rename to console/tooling-e2e/src/provider/seed.ts
index 53421f464..8977efe59 100644
--- a/console/tooling-e2e/src/seed.ts
+++ b/console/tooling-e2e/src/provider/seed.ts
@@ -9,10 +9,10 @@
* has to be up before the first scenario posts. Splitting setup out gives the
* three steps somewhere to happen in:
*
- * SWITCH_E2E=1 SWITCH_E2E_MANIFEST=… SWITCH_E2E_AGENT_DIR=… \
- * node --experimental-strip-types src/seed.ts
+ * SWITCH_E2E_LANE=provider SWITCH_E2E_MANIFEST=… SWITCH_E2E_AGENT_DIR=… \
+ * node --experimental-strip-types src/provider/seed.ts
* # seed the console's database from the manifest, start the console
- * SWITCH_E2E=1 SWITCH_E2E_MANIFEST=… … vitest run src/run.integration.test.ts
+ * SWITCH_E2E_LANE=provider SWITCH_E2E_MANIFEST=… … vitest run src/provider
*
* The scenario run finds the manifest, reuses what is in it rather than
* registering a second agent, and deletes it during teardown.
@@ -26,23 +26,25 @@
* `provider_id` `'opencode'` / `'claude'` and a `provider_config` of
* `{"version":"2","providerId":"<that>","values":{},"runtime":"provider"}`.
*/
-import { loadEnv } from './env.ts';
+import { resolveLane } from '../core/suite.ts';
import { setupHarness } from './harness.ts';
export async function seed(log: (message: string) => void = console.log): Promise<void> {
- const env = loadEnv();
- if (!env.manifestPath) {
+ const lane = await resolveLane('provider');
+ if (!lane.target) throw new Error(`Cannot seed: ${lane.skip}`);
+ const env = lane.target.env;
+ if (!env.provider.manifestPath) {
throw new Error('SWITCH_E2E_MANIFEST must name the file to record the agent and room in.');
}
- const harness = await setupHarness(env);
+ const harness = await setupHarness(lane.target);
log(
[
- `type ${env.agentType} (question mode: ${env.questionMode})`,
+ `type ${env.provider.agentType} (question mode: ${env.provider.questionMode})`,
`agent ${harness.agent.name} (${harness.agent.id})`,
`bot @${harness.bot.username} (${harness.bot.id})`,
`channel ${harness.channel.name} (${harness.channel.id})`,
`room ${harness.room.name} (${harness.room.id})`,
- `manifest ${env.manifestPath}`,
+ `manifest ${env.provider.manifestPath}`,
].join('\n')
);
}
diff --git a/console/tooling-e2e/src/run.integration.test.ts b/console/tooling-e2e/src/run.integration.test.ts
deleted file mode 100644
index 730d7ce19..000000000
--- a/console/tooling-e2e/src/run.integration.test.ts
+++ /dev/null
@@ -1,84 +0,0 @@
-import { afterAll, beforeAll, describe, expect, it } from 'vitest';
-import { describeEnv } from './env.ts';
-import {
- resolveSkip,
- setupHarness,
- teardownHarness,
- waitForSession,
- type Harness,
-} from './harness.ts';
-import { formatResultsTable, SCENARIOS, type ScenarioResult } from './scenarios.ts';
-
-/**
- * The suite. One throwaway agent, one Mattermost channel, four scenarios run in
- * order against them, then teardown and a summary table.
- *
- * It **skips** — never fails — when the machine is not set up for it:
- * `SWITCH_E2E=1` unset, configuration missing, or Switch/Mattermost unreachable.
- * The reason is printed, so a skip is never mysterious.
- *
- * It **fails** when the stack is there and the agent does not behave. For a
- * failure to mean anything, a Switch Console session must be running for the
- * registered agent — see README.md, "What must be running".
- */
-
-let harness: Harness | null = null;
-let skip: string | null = null;
-const results: ScenarioResult[] = [];
-
-beforeAll(async () => {
- const resolved = await resolveSkip();
- skip = resolved.skip;
- if (skip || !resolved.env) return;
-
- harness = await setupHarness(resolved.env);
- console.log(
- [
- '',
- `Switch Console end-to-end run ${harness.runId}`,
- ` ${describeEnv(resolved.env)}`,
- ` agent ${harness.agent.name} (${harness.agent.id})`,
- ` bot @${harness.bot.username} (${harness.bot.id})`,
- ` channel ${harness.channel.name} (${harness.channel.id})`,
- ` room ${harness.room.name} (${harness.room.id})`,
- '',
- ` A Switch Console session must now be running for '${harness.agent.name}'.`,
- '',
- ].join('\n')
- );
-
- const status = await waitForSession(
- harness,
- Number(process.env.SWITCH_E2E_SESSION_TIMEOUT_MS ?? 120_000)
- );
- console.log(` Switch reports the agent as '${status}' in the room — running scenarios.\n`);
-}, 600_000);
-
-afterAll(async () => {
- if (results.length > 0) console.log(formatResultsTable(results));
- await teardownHarness(harness);
-});
-
-describe('Switch Console provider-backed session, end to end', () => {
- for (const runScenario of SCENARIOS) {
- // The scenario functions are named after the behaviour they exercise, so the
- // test name comes from the function rather than being restated here.
- it(runScenario.name, async () => {
- if (skip) {
- console.log(`SKIPPED (${runScenario.name}): ${skip}`);
- return;
- }
- const result = await runScenario(harness!);
- results.push(result);
- if (!result.ok) {
- console.log(
- `${result.name} transcript:\n` +
- result.transcript
- .map((post) => ` [${post.user_id === harness!.bot.id ? 'bot ' : 'user'}] ${post.message}`)
- .join('\n')
- );
- }
- expect(result.ok, result.details).toBe(true);
- });
- }
-});
diff --git a/console/tooling-e2e/vitest.config.ts b/console/tooling-e2e/vitest.config.ts
index 16c4d985e..98fb90585 100644
--- a/console/tooling-e2e/vitest.config.ts
+++ b/console/tooling-e2e/vitest.config.ts
@@ -3,11 +3,12 @@ import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
include: ['src/**/*.test.ts'],
- // Real network round-trips through Switch, Matrix and Mattermost; a whole
+ // Real network round-trips through Switch and Mattermost; a whole
// scenario suite can idle for minutes waiting on a model turn.
testTimeout: 15 * 60_000,
hookTimeout: 5 * 60_000,
- // Scenarios share one channel and one agent, so they must not interleave.
+ // Provider scenarios share one channel and one agent, and a fast-lane run
+ // shares one run id, so nothing may interleave.
fileParallelism: false,
pool: 'forks',
reporters: ['verbose'],
diff --git a/justfile b/justfile
index f053838f2..6fcbd3ccf 100644
--- a/justfile
+++ b/justfile
@@ -171,6 +171,14 @@ test *args:
e2e-follow-ups *args:
core/tests/e2e/follow_the_conversation/run.sh {{ args }}
+# ── End-to-end harness, fast lane, on a throwaway stack ────────────────────────
+# Fresh Postgres and Mattermost, switch-core from this checkout, then the
+# provider-free scenarios in console/tooling-e2e, then teardown. Needs Docker
+# and your local stack stopped. `--keep` leaves it running; arguments after
+# `--` go to vitest. Output: build/e2e/ephemeral.
+e2e-fast *args:
+ console/tooling-e2e/scripts/ephemeral-stack.sh {{ args }}
+
# ── Run integration tests (real Postgres via testcontainers) ───────────────────
# DOCKER_HOST is auto-resolved from the active docker context in conftest, so this
# works under Docker Desktop / OrbStack / colima without extra setup.