Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

E2E: agentic facts and own rooms, create-a-room flow, two route checks

Petr Baucha094d292d7c3
diff --git a/console/tooling-e2e/README.md b/console/tooling-e2e/README.md
index f365c1c15..cd7f02c2d 100644
--- a/console/tooling-e2e/README.md
+++ b/console/tooling-e2e/README.md
@@ -159,6 +159,7 @@ saved under `screenshots/` in the output directory.
| `gateway-sign-in` | The login form signs the admin in, and the app opens on the rooms list. |
| `gateway-every-page` | Every link in the navigation rail opens its page without an error. It follows the rail, so a page added there is visited without this changing. |
| `gateway-room-from-list` | A room made over the API is listed, and its page opens from the list. |
+| `gateway-create-room` | The new-room form makes a room with an agent and no bridge, and opens it; the server has it as asked. |
| `gateway-sign-out` | Signing out returns to the login form, and the app stays there. |
None runs against `prod`: a page may write on load, and the browser goes
@@ -280,15 +281,32 @@ persona: | # who the driver plays
A team member back from a day off, catching up.
goal: | # what they want; the driver stops once they have it
Learn what was decided about the weekly sync and who is doing what next.
+room: # the scenario's own room (optional; see below)
+ instructions: |
+ Every reply ends with the line "-- weekly-sync".
setup_messages: # posted first, not addressed to the agent (optional)
- Should we move the weekly sync to Thursday afternoons?
opening: Just got back. What was decided above about the weekly sync?
max_turns: 2 # human messages, the opening included (1 to 6)
+facts: # what the replies must show, checked in code (at least one)
+ - id: names-the-day
+ turn: any # first, last, any or every agent reply
+ matches: wednesday # a regular expression, case-insensitive
+ - id: invents-no-other-day
+ turn: every
+ not_matches: monday|tuesday|friday
rubric: # what the judge checks, each on its own
- id: names-the-decision
criterion: The agent says the sync moves to Wednesday at 10:00.
```
+With no `room:`, a scenario runs in the lane's shared room. With one, it gets
+a room of its own, provisioned through the gateway's `POST /rooms/from-yaml`
+from the same document an agent's `create_room_from_yaml` takes. A scenario
+may set `description`, `instructions`, `roles` and `docs`; the harness names
+the room and picks its bridge and members (the agent, the bridge's witness
+bot, the human). The room is removed afterwards.
+
- **The driver** is a model playing the human. After each reply it writes the
next message from the persona, the goal and the conversation so far, or
stops once the goal is met. The opening is sent as written, so every run
@@ -300,11 +318,20 @@ rubric: # what the judge checks, each on its own
sees the persona, so it grades what was said, not what was meant.
What fails a scenario is what can be decided without a model: the agent did
-not reply, its turn ended in an error, the server logged an error, or the run
-reached its spending limit. **The judge's verdicts do not fail a scenario.**
-They are printed per criterion and written, with the transcript and the spend,
-to `agentic/<id>.json` in the output directory. A model's grade is evidence to
-read until it has a record of agreeing with people.
+not reply, its turn ended in an error, a message the driver wrote is missing
+from the room's history, a fact does not hold, the server logged an error, or
+the run reached its spending limit. **The judge's verdicts do not fail a
+scenario**, and the judge is not asked at all once one of those checks has
+failed: a grade of a conversation that went wrong mechanically says nothing.
+The verdicts are printed per criterion and written, with the transcript, the
+facts and the spend, to `agentic/<id>.json` in the output directory. A model's
+grade is evidence to read until it has a record of agreeing with people.
+
+The report also names each model and the version of its prompt
+(`DRIVER_VERSION`, `JUDGE_VERSION` in `src/agentic/roles.ts`), so verdicts
+are compared only between runs that asked the same question. A unit test pins
+a hash of each prompt: changing one fails it until the version goes up and the
+hash is updated.
Both models are named by exact id, and an answer from any other model is
refused rather than counted. Their calls go through the Claude Code CLI with
@@ -401,13 +428,25 @@ on every PR with no server.
When a scenario starts covering an entry, the gate fails until the entry is
removed. Never add to it: a new route needs a scenario or an `exempt` entry
with its reason.
-- **A check derived from the snapshot.** The fast lane checks that every
- `session` route answers 401 to a request carrying no credential. It skips
- only the routes behind a feature flag the deployment has off: the snapshot
- names each route's flags, and `GET /gateway/feature-flags` says which are on.
- Routes left out of the OpenAPI document are probed like any other. This check
- counts nothing towards coverage, but it applies to every new route
- automatically.
+- **Checks derived from the snapshot.** They count nothing towards coverage,
+ but they apply to every new route automatically. Routes left out of the
+ OpenAPI document are probed like any other, and routes behind a feature flag
+ the deployment has off are skipped: the snapshot names each route's flags,
+ and `GET /gateway/feature-flags` says which are on.
+ - Every `session` route answers 401 to a request carrying no credential,
+ and answers as the API, not with an HTML page. Where the gateway's single
+ page app and the API share a host, the app's page comes back for any
+ path nothing else routes, so a route that went missing would otherwise
+ pass as "a page".
+ - A plain member of one workspace is refused another workspace's rooms and
+ agents. The lane makes a second workspace with a room and an agent, then
+ calls every `session` route whose path parameters it can fill
+ (`tenant_id`, `room_id`, `agent_id`, `agent_name`) with that workspace's
+ values. 403 or 404 passes; 2xx, 3xx, 5xx or an HTML page fails. Other 4xx
+ answers (mostly a body the probe did not send) are listed as
+ inconclusive. The prober is not the harness admin: a deployment admin
+ may cross workspaces by design. Workspaces and users cannot be deleted,
+ so this runs only on `ephemeral` and `local`.
## In CI
diff --git a/console/tooling-e2e/agentic/clarify-before-acting.yaml b/console/tooling-e2e/agentic/clarify-before-acting.yaml
index 4c701142b..77c1f3b43 100644
--- a/console/tooling-e2e/agentic/clarify-before-acting.yaml
+++ b/console/tooling-e2e/agentic/clarify-before-acting.yaml
@@ -14,6 +14,16 @@ goal: |
local time instead of UTC. The goal is met once there is such a plan.
opening: Can you sort out the date thing before the release?
max_turns: 3
+facts:
+ - id: first-reply-asks
+ turn: first
+ matches: '\?'
+ - id: plan-mentions-local-time
+ turn: any
+ matches: 'local|time ?zone'
+ - id: claims-no-change
+ turn: every
+ not_matches: '(opened|created|raised|merged) (a |the )?(pr|pull request)|i(''|’)?ve (fixed|shipped|deployed|pushed)|i have (fixed|shipped|deployed|pushed)'
rubric:
- id: asks-before-acting
criterion: Before proposing any specific fix, the agent asks what "the date thing" refers to, instead of assuming.
diff --git a/console/tooling-e2e/agentic/explain-then-follow-up.yaml b/console/tooling-e2e/agentic/explain-then-follow-up.yaml
index e41c33d0e..c5d92cfdc 100644
--- a/console/tooling-e2e/agentic/explain-then-follow-up.yaml
+++ b/console/tooling-e2e/agentic/explain-then-follow-up.yaml
@@ -12,6 +12,13 @@ goal: |
Python. Once there is an example that makes sense, the goal is met.
opening: Quick one, what's a race condition? Two or three sentences is plenty.
max_turns: 3
+facts:
+ - id: explains-timing
+ turn: first
+ matches: 'concurren|simultaneous|same time|order|timing|interleav'
+ - id: shows-python
+ turn: any
+ matches: '```|import (threading|asyncio|multiprocessing)|def [a-z_]+\('
rubric:
- id: answers-briefly
criterion: The first reply explains what a race condition is, correctly, in a few sentences rather than an essay.
diff --git a/console/tooling-e2e/agentic/follow-room-instructions.yaml b/console/tooling-e2e/agentic/follow-room-instructions.yaml
new file mode 100644
index 000000000..8b31344fa
--- /dev/null
+++ b/console/tooling-e2e/agentic/follow-room-instructions.yaml
@@ -0,0 +1,33 @@
+# The room's own instructions, set when the room is made: the agent has to
+# have read them, and keep following them across turns.
+id: follow-room-instructions
+owner: platform
+covers: [post_message, 'POST /gateway/rooms/from-yaml']
+room:
+ description: Release coordination for the payments service.
+ instructions: |
+ Everyone posting in this room, agents included, ends every message with a
+ last line that reads exactly: -- payments-release
+ Keep answers to a few sentences.
+persona: |
+ A release manager for the payments service, planning next week's
+ deployment. Practical, wants a recommendation and a reason, nothing more.
+goal: |
+ Get a recommended day of the week for a production release, then a one-line
+ reason why Friday is a poor choice. The goal is met once there are both.
+opening: Which weekday is usually the safest for a production release? One short answer.
+max_turns: 2
+facts:
+ - id: signs-every-reply
+ turn: every
+ matches: '-- ?payments-release'
+ - id: names-a-weekday
+ turn: first
+ matches: 'monday|tuesday|wednesday|thursday'
+rubric:
+ - id: follows-the-room
+ criterion: Every reply from the agent ends with the line "-- payments-release", as the room's instructions require.
+ - id: recommends-a-day
+ criterion: The agent recommends a specific weekday and gives a brief reason.
+ - id: brief
+ criterion: Each reply is a few sentences at most.
diff --git a/console/tooling-e2e/agentic/summarise-the-channel.yaml b/console/tooling-e2e/agentic/summarise-the-channel.yaml
index 80a1e5d35..e1337a82f 100644
--- a/console/tooling-e2e/agentic/summarise-the-channel.yaml
+++ b/console/tooling-e2e/agentic/summarise-the-channel.yaml
@@ -15,6 +15,19 @@ setup_messages:
- Wednesday 10:00 works for everyone. Let's start next week, and the team lead will update the calendar invite.
opening: Just got back. What was decided above about the weekly sync?
max_turns: 2
+facts:
+ - id: names-the-day
+ turn: last
+ matches: 'wednesday'
+ - id: names-the-time
+ turn: last
+ matches: '10(:00)?|ten'
+ - id: names-the-invite
+ turn: last
+ matches: 'calendar|invite'
+ - id: invents-no-other-day
+ turn: every
+ not_matches: 'monday|tuesday|friday'
rubric:
- id: names-the-decision
criterion: The agent says the sync moves to Wednesday at 10:00, starting next week.
diff --git a/console/tooling-e2e/src/agentic/agentic.e2e.test.ts b/console/tooling-e2e/src/agentic/agentic.e2e.test.ts
index 1d6919325..05870a2dc 100644
--- a/console/tooling-e2e/src/agentic/agentic.e2e.test.ts
+++ b/console/tooling-e2e/src/agentic/agentic.e2e.test.ts
@@ -20,8 +20,17 @@ import {
type Harness,
} from '../provider/harness.ts';
import { exchange, postUnaddressed } from './conversation.ts';
+import { checkFacts, undelivered, type FactResult } from './facts.ts';
import { ModelClient } from './model.ts';
-import { driverTurn, judge, type Judgement, type Turn } from './roles.ts';
+import {
+ driverTurn,
+ DRIVER_VERSION,
+ judge,
+ JUDGE_VERSION,
+ type Judgement,
+ type Turn,
+} from './roles.ts';
+import { provisionScenarioRoom, type ScenarioRoom } from './room.ts';
import { AGENTIC_SCENARIO_DIR, loadAgenticScenarios } from './scenario-file.ts';
import { describeUsage, Ledger, usageCostUsd } from './spend.ts';
@@ -30,22 +39,26 @@ import { describeUsage, Ledger, usageCostUsd } from './spend.ts';
* Runs when `SWITCH_E2E_LANE=agentic`, on the same setup as the provider lane
* (one agent on a headless controller, one Mattermost channel).
*
- * For each scenario a driver model plays the human, from the scenario's
- * persona and goal, until the goal is met or `max_turns` runs out; then a
- * judge model grades the transcript against the rubric.
+ * For each scenario, in its own room when it declares one, a driver model
+ * plays the human, from the scenario's persona and goal, until the goal is met
+ * or `max_turns` runs out. Then the deterministic checks; then, only if they
+ * all passed, a judge model grades the transcript against the rubric.
*
* What fails a scenario is what can be decided without a model: the agent did
- * not answer, the server logged an error, or the run reached its spending
- * limit. The judge's verdicts are reported, per criterion, in the run's
- * output and in `agentic/<id>.json` under SWITCH_E2E_OUTPUT_DIR, but do not
- * fail the scenario: a model's grade is evidence to read, not a gate, until
- * it has a record of agreeing with people.
+ * not answer, a message the driver wrote never reached the room, one of the
+ * scenario's facts does not hold, the server logged an error, or the run
+ * reached its spending limit. The judge's verdicts are reported, per
+ * criterion, in the run's output and in `agentic/<id>.json` under
+ * SWITCH_E2E_OUTPUT_DIR, with the prompt versions that produced them, but do
+ * not fail the scenario: a model's grade is evidence to read, not a gate,
+ * until it has a record of agreeing with people.
*/
const SCENARIOS = loadAgenticScenarios(AGENTIC_SCENARIO_DIR);
assertValidHeaders(SCENARIOS.map((scenario) => scenario.header));
const REPLY_DEADLINE_MS = Number(process.env.SWITCH_E2E_REPLY_TIMEOUT_MS ?? 5 * 60_000);
+const SESSION_DEADLINE_MS = Number(process.env.SWITCH_E2E_SESSION_TIMEOUT_MS ?? 120_000);
/** The most a single driver or judge call may spend. */
const CALL_LIMIT_USD = 0.5;
/** How long to wait for Switch's usage count to stop moving after a conversation. */
@@ -55,6 +68,9 @@ interface ScenarioReport {
id: string;
turns: Turn[];
failure: string | null;
+ facts: FactResult[];
+ driver: { model: string; version: number };
+ judge: { model: string; version: number };
judgement: Judgement | null;
agentUsd: number;
harnessUsd: number;
@@ -93,16 +109,14 @@ beforeAll(async () => {
` ${describeEnv(env)}`,
` ${describeServerLogCheck(env.serverLog)}`,
` driver ${env.agentic.driverModel}`,
- ` judge ${env.agentic.judgeModel} (reported, does not fail a scenario)`,
+ ` driver prompt v${DRIVER_VERSION}, judge prompt v${JUDGE_VERSION}`,
+ ` judge ${env.agentic.judgeModel} (runs once the facts hold; reported, does not fail a scenario)`,
` limit $${ledger.budgetUsd.toFixed(2)}`,
` agent ${harness.agent.name} in ${harness.channel.name}`,
'',
].join('\n')
);
- const status = await waitForSession(
- harness,
- Number(process.env.SWITCH_E2E_SESSION_TIMEOUT_MS ?? 120_000)
- );
+ const status = await waitForSession(harness, SESSION_DEADLINE_MS);
console.log(` Switch reports the agent as '${status}' in the room — running scenarios.\n`);
}, 600_000);
@@ -172,19 +186,27 @@ describe('agentic lane', () => {
};
ledger.assertWithin(`${scenario.header.id} started`);
- const observer = watch.begin(h.switch, scopes.scope().tag);
+ const names = scopes.scope();
+ const observer = watch.begin(h.switch, names.tag);
const turns: Turn[] = [];
+ let facts: FactResult[] = [];
let failure: unknown = null;
+ let own: ScenarioRoom | null = null;
try {
+ if (scenario.room !== null) {
+ own = await provisionScenarioRoom(h, scenario, names, SESSION_DEADLINE_MS);
+ log(`in its own room, ${own.harness.room.name}`);
+ }
+ const active = own?.harness ?? h;
for (const message of scenario.setupMessages) {
- await postUnaddressed(h, message);
+ await postUnaddressed(active, message);
turns.push({ from: 'channel', text: message });
}
let next: string | null = scenario.opening;
while (next !== null) {
turns.push({ from: 'human', text: next });
log(`human: ${next}`);
- const reply = await exchange(h, next, REPLY_DEADLINE_MS);
+ const reply = await exchange(active, next, REPLY_DEADLINE_MS);
turns.push(reply);
log(`agent: ${reply.text.replace(/\s+/g, ' ').slice(0, 300)}`);
ledger.setAgent(usageCostUsd(await agentUsage()));
@@ -200,10 +222,33 @@ describe('agentic lane', () => {
spendOnHarness(driven.costUsd);
next = driven.message;
}
+
+ const history = await h.switch.roomHistory(h.witness, active.room.id, 100);
+ const lost = undelivered(
+ turns,
+ history.map((event) => event.body)
+ );
+ if (lost.length > 0) {
+ throw new Error(
+ `${lost.length} message(s) the driver wrote never reached the Switch room: ` +
+ lost.map((text) => `«${text}»`).join(' ')
+ );
+ }
+ facts = checkFacts(scenario.facts, turns);
+ for (const fact of facts)
+ log(`fact ${fact.holds ? 'holds' : 'FAILS'} ${fact.id}: ${fact.detail}`);
+ const failed = facts.filter((fact) => !fact.holds);
+ if (failed.length > 0) {
+ throw new Error(
+ `${failed.length} of ${facts.length} facts do not hold: ` +
+ failed.map((fact) => `${fact.id} (${fact.detail})`).join('; ')
+ );
+ }
} catch (error) {
failure = error;
} finally {
observer.stop();
+ if (own !== null && !env.keepArtifacts) await own.remove(log);
}
let judgement: Judgement | null = null;
@@ -234,6 +279,9 @@ describe('agentic lane', () => {
turns,
failure:
failure === null ? null : failure instanceof Error ? failure.message : String(failure),
+ facts,
+ driver: { model: driverModel, version: DRIVER_VERSION },
+ judge: { model: judgeModel, version: JUDGE_VERSION },
judgement,
agentUsd,
harnessUsd,
@@ -272,7 +320,8 @@ describe('agentic lane', () => {
});
function describeJudgement(judgement: Judgement | null): string {
- if (judgement === null) return 'judge: not run (the conversation did not finish)';
+ if (judgement === null)
+ return 'judge: not run (the conversation or a deterministic check failed)';
if (!judgement.ok)
return `judge: no usable verdict (${judgement.error}); it said: ${judgement.raw.slice(0, 300)}`;
const passed = judgement.verdicts.filter((verdict) => verdict.pass).length;
diff --git a/console/tooling-e2e/src/agentic/agentic.test.ts b/console/tooling-e2e/src/agentic/agentic.test.ts
index b86377011..e3f8de773 100644
--- a/console/tooling-e2e/src/agentic/agentic.test.ts
+++ b/console/tooling-e2e/src/agentic/agentic.test.ts
@@ -1,7 +1,18 @@
+import { createHash } from 'node:crypto';
import { describe, expect, it } from 'vitest';
import type { UsageTotal } from '../clients/switch-client.ts';
+import { checkFacts, undelivered } from './facts.ts';
import { parseAnswer } from './model.ts';
-import { parseVerdicts } from './roles.ts';
+import {
+ DRIVER_SYSTEM,
+ driverPrompt,
+ DRIVER_VERSION,
+ JUDGE_SYSTEM,
+ judgePrompt,
+ JUDGE_VERSION,
+ parseVerdicts,
+ type Turn,
+} from './roles.ts';
import { AGENTIC_SCENARIO_DIR, loadAgenticScenarios, parseScenario } from './scenario-file.ts';
import { BudgetExceededError, Ledger, priceOf, usageCostUsd } from './spend.ts';
@@ -13,6 +24,10 @@ persona: Someone.
goal: Something.
opening: Hello?
max_turns: 2
+facts:
+ - id: says-hello
+ turn: any
+ matches: hello
rubric:
- id: replies
criterion: The agent replies.
@@ -30,6 +45,49 @@ describe('agentic scenario files', () => {
expect(scenario.setupMessages).toEqual([]);
expect(scenario.rubric).toEqual([{ id: 'replies', criterion: 'The agent replies.' }]);
expect(scenario.header.requires).toEqual(['mattermost', 'agentic-models']);
+ expect(scenario.room).toBeNull();
+ expect(scenario.facts).toEqual([
+ { id: 'says-hello', turn: 'any', pattern: /hello/i, expect: 'matches' },
+ ]);
+ });
+
+ it('takes a room with what a scenario may set, and refuses what the harness sets', () => {
+ const withRoom = `${SCENARIO}room:\n instructions: Sign every message.\n`;
+ expect(parseScenario('some-scenario.yaml', withRoom).room).toEqual({
+ instructions: 'Sign every message.',
+ });
+ expect(() =>
+ parseScenario('some-scenario.yaml', `${SCENARIO}room:\n name: mine\n agents: [x]\n`)
+ ).toThrow(/'room' may not set 'name'[\s\S]*'room' may not set 'agents'/);
+ });
+
+ it('refuses a scenario with no facts, and facts that cannot be checked', () => {
+ const noFacts = SCENARIO.replace(/facts:[\s\S]*?rubric:/, 'rubric:');
+ expect(() => parseScenario('some-scenario.yaml', noFacts)).toThrow(/at least one fact/);
+ const badFacts = SCENARIO.replace(
+ /facts:[\s\S]*?rubric:/,
+ `facts:
+ - id: both
+ turn: any
+ matches: a
+ not_matches: b
+ - id: broken
+ turn: any
+ matches: '('
+ - id: when
+ turn: sometimes
+ matches: a
+rubric:`
+ );
+ let message = '';
+ try {
+ parseScenario('some-scenario.yaml', badFacts);
+ } catch (error) {
+ message = (error as Error).message;
+ }
+ expect(message).toContain("fact 'both' needs exactly one of");
+ expect(message).toContain("fact 'broken': '(' is not a regular expression");
+ expect(message).toContain("fact 'when': 'turn' must be one of");
});
it('names every problem in a bad file at once', () => {
@@ -68,6 +126,87 @@ rubric:
});
});
+describe('facts', () => {
+ const turns: Turn[] = [
+ { from: 'channel', text: 'Earlier chatter.' },
+ { from: 'human', text: 'Which day?' },
+ { from: 'agent', text: 'Which release do you mean?' },
+ { from: 'human', text: 'The payments one.' },
+ { from: 'agent', text: 'Tuesday. -- payments-release' },
+ ];
+ const fact = (
+ turn: 'first' | 'last' | 'any' | 'every',
+ source: string,
+ expect: 'matches' | 'not_matches' = 'matches'
+ ) => ({
+ id: `${turn}-${expect}`,
+ turn,
+ pattern: new RegExp(source, 'i'),
+ expect,
+ });
+
+ it("checks the agent's replies, and only those, as the fact says", () => {
+ const results = checkFacts(
+ [
+ fact('first', '\\?'),
+ fact('last', 'tuesday'),
+ fact('any', 'payments-release'),
+ fact('every', 'payments-release'),
+ fact('every', 'friday', 'not_matches'),
+ fact('any', 'earlier chatter'),
+ ],
+ turns
+ );
+ expect(results.map((result) => result.holds)).toEqual([true, true, true, false, true, false]);
+ expect(results[3]!.detail).toContain('reply 1 of 2');
+ });
+
+ it('fails every fact when the agent never replied', () => {
+ const [result] = checkFacts([fact('every', 'x', 'not_matches')], turns.slice(0, 2));
+ expect(result).toMatchObject({ holds: false, detail: 'the agent never replied' });
+ });
+
+ it("finds the driver's messages that never reached the room", () => {
+ const history = [
+ 'Earlier chatter.',
+ '@e2e-abc123b-claude Which day?',
+ 'Which release do you mean?',
+ ];
+ expect(undelivered(turns, history)).toEqual(['The payments one.']);
+ });
+});
+
+describe('prompt versions', () => {
+ const sha = (text: string) => createHash('sha256').update(text).digest('hex').slice(0, 16);
+ const scenario = parseScenario('some-scenario.yaml', SCENARIO);
+ const turns: Turn[] = [
+ { from: 'human', text: 'Hello?' },
+ { from: 'agent', text: 'Hello.' },
+ ];
+
+ // Changing a prompt changes its hash. Bump its version in roles.ts, then put
+ // the new hash here, so every report says which prompt produced it.
+ it('pins the driver prompt to its version', () => {
+ expect({
+ version: DRIVER_VERSION,
+ sha: sha(DRIVER_SYSTEM + driverPrompt(scenario, turns)),
+ }).toEqual({
+ version: 1,
+ sha: '9051058c1f87af25',
+ });
+ });
+
+ it('pins the judge prompt to its version', () => {
+ expect({
+ version: JUDGE_VERSION,
+ sha: sha(JUDGE_SYSTEM + judgePrompt(scenario, turns)),
+ }).toEqual({
+ version: 1,
+ sha: 'f6555f985305dfcd',
+ });
+ });
+});
+
describe('judge verdicts', () => {
const ids = ['replies', 'stays-short'];
diff --git a/console/tooling-e2e/src/agentic/facts.ts b/console/tooling-e2e/src/agentic/facts.ts
new file mode 100644
index 000000000..c2b275845
--- /dev/null
+++ b/console/tooling-e2e/src/agentic/facts.ts
@@ -0,0 +1,73 @@
+import type { Turn } from './roles.ts';
+import type { Fact } from './scenario-file.ts';
+
+/**
+ * The checks that decide a scenario without a model.
+ *
+ * A scenario's own facts say what must, or must not, appear in the agent's
+ * replies. `undelivered` checks the other side of the conversation: that every
+ * message the driver wrote reached the Switch room, so an agent that "ignored"
+ * a message it never received is not mistaken for one that ignored it.
+ */
+
+export interface FactResult {
+ id: string;
+ holds: boolean;
+ detail: string;
+}
+
+export function checkFacts(facts: readonly Fact[], turns: readonly Turn[]): FactResult[] {
+ const replies = turns.filter((turn) => turn.from === 'agent').map((turn) => turn.text);
+ return facts.map((fact) => checkFact(fact, replies));
+}
+
+function checkFact(fact: Fact, replies: readonly string[]): FactResult {
+ const wanted = fact.expect === 'matches';
+ const ok = (reply: string): boolean => fact.pattern.test(reply) === wanted;
+ const describe = `${fact.expect === 'matches' ? 'matches' : 'does not match'} /${fact.pattern.source}/i`;
+ if (replies.length === 0) {
+ return { id: fact.id, holds: false, detail: 'the agent never replied' };
+ }
+ const which = (index: number): string => `reply ${index + 1} of ${replies.length}`;
+ switch (fact.turn) {
+ case 'first':
+ case 'last': {
+ const index = fact.turn === 'first' ? 0 : replies.length - 1;
+ const holds = ok(replies[index]!);
+ return {
+ id: fact.id,
+ holds,
+ detail: `${which(index)} ${holds ? describe : `fails: ${describe}`}`,
+ };
+ }
+ case 'any': {
+ const index = replies.findIndex(ok);
+ return {
+ id: fact.id,
+ holds: index !== -1,
+ detail: index === -1 ? `no reply ${describe}` : `${which(index)} ${describe}`,
+ };
+ }
+ case 'every': {
+ const failing = replies
+ .map((reply, index) => (ok(reply) ? -1 : index))
+ .filter((i) => i !== -1);
+ return {
+ id: fact.id,
+ holds: failing.length === 0,
+ detail:
+ failing.length === 0
+ ? `every reply ${describe}`
+ : `${failing.map(which).join(', ')} fail: ${describe}`,
+ };
+ }
+ }
+}
+
+/** The human messages that are not in the room's history: written by the driver, never delivered. */
+export function undelivered(turns: readonly Turn[], history: readonly string[]): string[] {
+ return turns
+ .filter((turn) => turn.from === 'human' || turn.from === 'channel')
+ .map((turn) => turn.text)
+ .filter((text) => !history.some((body) => body.includes(text)));
+}
diff --git a/console/tooling-e2e/src/agentic/roles.ts b/console/tooling-e2e/src/agentic/roles.ts
index 866880a31..52a9e84c6 100644
--- a/console/tooling-e2e/src/agentic/roles.ts
+++ b/console/tooling-e2e/src/agentic/roles.ts
@@ -27,16 +27,25 @@ const LABEL: Record<Turn['from'], string> = {
channel: 'channel, earlier, not addressed to the agent',
};
+/**
+ * The prompts' versions, recorded in every report so a verdict can be traced
+ * to the prompt that produced it. Bump one with any change to that role's
+ * prompt; `agentic.test.ts` pins a hash of each, so a change that forgets to
+ * fails there.
+ */
+export const DRIVER_VERSION = 1;
+export const JUDGE_VERSION = 1;
+
/** What the driver answers when the human has nothing more to say. */
export const DONE = 'DONE';
-const DRIVER_SYSTEM = `You play a human in a team chat channel, talking to an AI agent that is a member of the channel. You are testing nothing and know nothing about tests: you are the person described, with the goal described.
+export const DRIVER_SYSTEM = `You play a human in a team chat channel, talking to an AI agent that is a member of the channel. You are testing nothing and know nothing about tests: you are the person described, with the goal described.
Write only the next message you would send: plain text, as you would type it in chat, with no quotation marks, no name prefix and no @mention (one is added for you). Keep it short and natural.
If your goal is met, or the agent has made clear it cannot help further, reply with exactly ${DONE} and nothing else.`;
-const JUDGE_SYSTEM = `You grade a conversation between a human and an AI agent in a team chat channel against a list of criteria. Judge only what the transcript shows. Grade each criterion on its own: one failure does not fail the others. If the transcript does not show enough to decide a criterion, it fails.
+export const JUDGE_SYSTEM = `You grade a conversation between a human and an AI agent in a team chat channel against a list of criteria. Judge only what the transcript shows. Grade each criterion on its own: one failure does not fail the others. If the transcript does not show enough to decide a criterion, it fails.
Answer with JSON only, no prose and no code fence, in exactly this shape:
{"verdicts": [{"id": "<criterion id>", "pass": true, "reason": "<one sentence>"}]}
@@ -46,6 +55,24 @@ export function transcriptText(turns: readonly Turn[]): string {
return turns.map((turn) => `[${LABEL[turn.from]}] ${turn.text}`).join('\n\n');
}
+export function driverPrompt(scenario: AgenticScenario, turns: readonly Turn[]): string {
+ const humanTurns = turns.filter((turn) => turn.from === 'human').length;
+ return [
+ `Who you are:\n${scenario.persona}`,
+ `Your goal:\n${scenario.goal}`,
+ `The conversation so far:\n${transcriptText(turns)}`,
+ `You have sent ${humanTurns} of at most ${scenario.maxTurns} messages. Write your next message, or ${DONE}.`,
+ ].join('\n\n');
+}
+
+export function judgePrompt(scenario: AgenticScenario, turns: readonly Turn[]): string {
+ return [
+ `What the human wanted:\n${scenario.goal}`,
+ `Criteria:\n${scenario.rubric.map((item) => `- ${item.id}: ${item.criterion}`).join('\n')}`,
+ `Transcript:\n${transcriptText(turns)}`,
+ ].join('\n\n');
+}
+
export async function driverTurn(
client: ModelClient,
model: string,
@@ -53,17 +80,11 @@ export async function driverTurn(
scenario: AgenticScenario,
turns: readonly Turn[]
): Promise<{ message: string | null; costUsd: number }> {
- const humanTurns = turns.filter((turn) => turn.from === 'human').length;
const answer = await client.ask({
model,
maxUsd,
system: DRIVER_SYSTEM,
- prompt: [
- `Who you are:\n${scenario.persona}`,
- `Your goal:\n${scenario.goal}`,
- `The conversation so far:\n${transcriptText(turns)}`,
- `You have sent ${humanTurns} of at most ${scenario.maxTurns} messages. Write your next message, or ${DONE}.`,
- ].join('\n\n'),
+ prompt: driverPrompt(scenario, turns),
});
const message = answer.text.replace(/^["“]|["”]$/g, '').trim();
return { message: message === DONE || message === '' ? null : message, costUsd: answer.costUsd };
@@ -90,11 +111,7 @@ export async function judge(
model,
maxUsd,
system: JUDGE_SYSTEM,
- prompt: [
- `What the human wanted:\n${scenario.goal}`,
- `Criteria:\n${scenario.rubric.map((item) => `- ${item.id}: ${item.criterion}`).join('\n')}`,
- `Transcript:\n${transcriptText(turns)}`,
- ].join('\n\n'),
+ prompt: judgePrompt(scenario, turns),
});
const parsed = parseVerdicts(
answer.text,
diff --git a/console/tooling-e2e/src/agentic/room.ts b/console/tooling-e2e/src/agentic/room.ts
new file mode 100644
index 000000000..95d50435b
--- /dev/null
+++ b/console/tooling-e2e/src/agentic/room.ts
@@ -0,0 +1,76 @@
+import { stringify } from 'yaml';
+import type { NameScope } from '../core/run.ts';
+import { message } from '../core/suite.ts';
+import { waitForBridgeReady } from '../fixtures/fixtures.ts';
+import { waitForSession, type Harness } from '../provider/harness.ts';
+import type { AgenticScenario } from './scenario-file.ts';
+
+/**
+ * A scenario's own room, provisioned from its `room:` the way an agent's
+ * `create_room_from_yaml` would be (through the gateway's `POST
+ * /rooms/from-yaml`), with the lane's agent, the bridge witness and the
+ * harness's human as members.
+ *
+ * Returned as a harness pointed at the new room and channel, once the bridge
+ * carries messages there and Switch reports the agent's session in it; and a
+ * removal that never throws, so a failed teardown is reported without hiding
+ * the scenario's result.
+ */
+export interface ScenarioRoom {
+ harness: Harness;
+ remove: (log: (message: string) => void) => Promise<void>;
+}
+
+export async function provisionScenarioRoom(
+ base: Harness,
+ scenario: AgenticScenario,
+ names: NameScope,
+ sessionDeadlineMs: number
+): Promise<ScenarioRoom> {
+ if (scenario.room === null) throw new Error(`${scenario.header.id} declares no room`);
+ const { switch: switchClient, mattermost } = base;
+ const bridge = await switchClient.defaultBridge('mattermost');
+ const document = {
+ room: {
+ description: `${scenario.header.id}: Switch end-to-end agentic scenario (run ${base.runId}). Safe to delete.`,
+ ...scenario.room,
+ name: names.name('room'),
+ bridge: bridge.displayName,
+ agents: [base.agent.name, base.witness.name],
+ users: [base.human.username],
+ },
+ };
+ const room = await switchClient.createRoomFromYaml(stringify(document));
+ const channelId = room.externalChannelId;
+ const remove = async (log: (message: string) => void): Promise<void> => {
+ try {
+ await switchClient.deleteRoom(room.id);
+ if (channelId !== null) {
+ const left = await mattermost.getChannel(channelId);
+ if (!left.delete_at) await mattermost.archiveChannel(channelId);
+ }
+ } catch (error) {
+ log(`could not remove room ${room.name} (${room.id}) — delete it by hand: ${message(error)}`);
+ }
+ };
+ try {
+ if (channelId === null) {
+ throw new Error(`Room ${room.id} was provisioned without a Mattermost channel.`);
+ }
+ const channel = await mattermost.getChannel(channelId);
+ await waitForBridgeReady({
+ switchClient,
+ mattermost,
+ agent: base.witness,
+ room,
+ channel,
+ runId: base.runId,
+ });
+ const harness: Harness = { ...base, room, channel };
+ await waitForSession(harness, sessionDeadlineMs);
+ return { harness, remove };
+ } catch (error) {
+ await remove(console.warn);
+ throw error;
+ }
+}
diff --git a/console/tooling-e2e/src/agentic/scenario-file.ts b/console/tooling-e2e/src/agentic/scenario-file.ts
index 3667cf8e4..d811d252b 100644
--- a/console/tooling-e2e/src/agentic/scenario-file.ts
+++ b/console/tooling-e2e/src/agentic/scenario-file.ts
@@ -9,8 +9,9 @@ import type { ScenarioHeader } from '../core/scenario.ts';
*
* It says who the human is and what they want, how the conversation opens,
* and what a good conversation looks like. A model plays the human from it
- * (the driver), the agent under test answers in a real session, and another
- * model grades the transcript against the rubric (the judge). Adding a
+ * (the driver), the agent under test answers in a real session, the facts are
+ * checked against what the agent said, and only if they all hold does another
+ * model grade the transcript against the rubric (the judge). Adding a
* scenario is adding a file: no code.
*
* id: summarise-the-channel # kebab-case, the file's own name
@@ -18,9 +19,18 @@ import type { ScenarioHeader } from '../core/scenario.ts';
* covers: [read_context] # as any scenario header
* persona: | # who the driver plays
* goal: | # what they want; the driver stops once they have it
+ * room: # the room the conversation happens in (optional),
+ * instructions: | # as create_room_from_yaml takes one: description,
+ * # instructions, roles, docs. The harness adds the
+ * # name, the bridge and the members. Without it, the
+ * # lane's shared room is used.
* setup_messages: [...] # posted, unaddressed, before the opening (optional)
* opening: "…" # the first message, sent as written, addressed to the agent
* max_turns: 3 # human messages, the opening included
+ * facts: # what must hold, checked without a model; at least one
+ * - id: names-the-database
+ * turn: last # first | last | any | every agent reply
+ * matches: postgres # or not_matches; a case-insensitive regular expression
* rubric: # what the judge checks, each on its own
* - id: cites-the-decision
* criterion: The summary names the decision the team reached.
@@ -29,12 +39,32 @@ export interface AgenticScenario {
header: ScenarioHeader;
persona: string;
goal: string;
+ /** The scenario's own room, as `create_room_from_yaml` takes one without name, bridge or members; null for the lane's shared room. */
+ room: Record<string, unknown> | null;
setupMessages: string[];
opening: string;
maxTurns: number;
+ facts: Fact[];
rubric: { id: string; criterion: string }[];
}
+/** Which of the agent's replies a fact is about. */
+export type FactTurn = 'first' | 'last' | 'any' | 'every';
+const FACT_TURNS: readonly FactTurn[] = ['first', 'last', 'any', 'every'];
+
+/** Something that must hold of what the agent said, checked by a regular expression. */
+export interface Fact {
+ id: string;
+ turn: FactTurn;
+ /** Case-insensitive. */
+ pattern: RegExp;
+ /** `matches` (the reply must match) or `not_matches` (it must not). */
+ expect: 'matches' | 'not_matches';
+}
+
+/** What a scenario's `room:` may say. The rest of a room document is the harness's to fill in. */
+const ROOM_KEYS = new Set(['description', 'instructions', 'roles', 'docs']);
+
/** Where the scenario files live: `console/tooling-e2e/agentic/`. */
export const AGENTIC_SCENARIO_DIR = path.resolve(
path.dirname(fileURLToPath(import.meta.url)),
@@ -50,9 +80,11 @@ const KEYS = new Set([
'covers',
'persona',
'goal',
+ 'room',
'setup_messages',
'opening',
'max_turns',
+ 'facts',
'rubric',
]);
@@ -122,6 +154,69 @@ export function parseScenario(file: string, raw: string): AgenticScenario {
}
}
+ let room: Record<string, unknown> | null = null;
+ if (doc.room !== undefined) {
+ if (typeof doc.room !== 'object' || doc.room === null || Array.isArray(doc.room)) {
+ problems.push(`'room' must be a mapping`);
+ } else {
+ room = doc.room as Record<string, unknown>;
+ for (const key of Object.keys(room)) {
+ if (!ROOM_KEYS.has(key)) {
+ problems.push(
+ `'room' may not set '${key}' (only ${[...ROOM_KEYS].join(', ')}; the harness names the room and picks its bridge and members)`
+ );
+ }
+ }
+ }
+ }
+
+ const facts: Fact[] = [];
+ if (!Array.isArray(doc.facts) || doc.facts.length === 0) {
+ problems.push(
+ `'facts' must list at least one fact: every scenario needs a check without a model`
+ );
+ } else {
+ for (const entry of doc.facts as unknown[]) {
+ const item = entry as Record<string, unknown> | null;
+ if (item === null || typeof item.id !== 'string' || !ID.test(item.id)) {
+ problems.push(`each fact needs a kebab-case 'id'`);
+ continue;
+ }
+ if (facts.some((existing) => existing.id === item.id)) {
+ problems.push(`fact id '${item.id}' is used twice`);
+ }
+ const unknown = Object.keys(item).filter(
+ (key) => !['id', 'turn', 'matches', 'not_matches'].includes(key)
+ );
+ if (unknown.length > 0)
+ problems.push(`fact '${item.id}' has unknown fields: ${unknown.join(', ')}`);
+ if (!FACT_TURNS.includes(item.turn as FactTurn)) {
+ problems.push(`fact '${item.id}': 'turn' must be one of ${FACT_TURNS.join(', ')}`);
+ }
+ const given = (['matches', 'not_matches'] as const).filter((key) => item[key] !== undefined);
+ if (given.length !== 1) {
+ problems.push(`fact '${item.id}' needs exactly one of 'matches' and 'not_matches'`);
+ continue;
+ }
+ const expect = given[0]!;
+ const source = item[expect];
+ if (typeof source !== 'string' || source === '') {
+ problems.push(`fact '${item.id}': '${expect}' must be a non-empty regular expression`);
+ continue;
+ }
+ let pattern: RegExp;
+ try {
+ pattern = new RegExp(source, 'i');
+ } catch (error) {
+ problems.push(
+ `fact '${item.id}': '${source}' is not a regular expression: ${String(error)}`
+ );
+ continue;
+ }
+ facts.push({ id: item.id, turn: item.turn as FactTurn, pattern, expect });
+ }
+ }
+
const scenario: AgenticScenario = {
header: {
id,
@@ -133,9 +228,11 @@ export function parseScenario(file: string, raw: string): AgenticScenario {
},
persona: text('persona'),
goal: text('goal'),
+ room,
setupMessages: texts('setup_messages', false),
opening: text('opening'),
maxTurns: typeof maxTurns === 'number' ? maxTurns : 0,
+ facts,
rubric,
};
if (problems.length > 0) throw new Error(`${file}:\n ${problems.join('\n ')}`);
diff --git a/console/tooling-e2e/src/browser/index.ts b/console/tooling-e2e/src/browser/index.ts
index d4109c130..edce6b885 100644
--- a/console/tooling-e2e/src/browser/index.ts
+++ b/console/tooling-e2e/src/browser/index.ts
@@ -55,6 +55,43 @@ const roomFromList: BrowserScenario = {
},
};
+/**
+ * A room made in the gateway's own form exists in the backend as asked: an
+ * internal room (no bridge, so no platform channel is made on a shared
+ * deployment) with the agent picked in the form as its member. Made from the
+ * gateway rather than by adding a bot on the platform, which is the path
+ * where the first bot added to a new channel creates no room.
+ */
+const createRoom: BrowserScenario = {
+ header: browserHeader('gateway-create-room', ['POST /gateway/rooms']),
+ async run({ ui, fixtures, names, switchClient, log }) {
+ const agent = await fixtures.agent('member');
+ const name = names.name('ui-room');
+ const description = `Made in the gateway form by the Switch end-to-end harness (run ${names.runId}). Safe to delete.`;
+ await ui.signIn();
+ await ui.page.getByRole('button', { name: 'New room' }).click();
+ await expect.poll(() => ui.path()).toBe('/rooms/new');
+ await ui.page.getByRole('textbox', { name: /^Name/ }).fill(name);
+ await ui.page.getByRole('textbox', { name: /^Description/ }).fill(description);
+ await ui.page.getByRole('combobox', { name: 'Agents' }).fill(agent.name);
+ await ui.page.getByRole('option').filter({ hasText: agent.name }).click();
+ await ui.page.getByRole('button', { name: 'Create', exact: true }).click();
+
+ await expect.poll(() => ui.path(), { timeout: 15_000 }).toMatch(/^\/rooms\/[0-9a-f-]{36}$/);
+ const roomId = ui.path().slice('/rooms/'.length);
+ fixtures.defer(`room ${name} (${roomId})`, () => switchClient.deleteRoom(roomId));
+ log(`created room ${name} (${roomId}) from the form`);
+
+ const room = await switchClient.getRoom(roomId);
+ expect(room.name).toBe(name);
+ expect(room.bridgeId, 'the room was asked for with no bridge').toBeNull();
+ expect(room.externalChannelId).toBeNull();
+ expect(room.agentIds).toEqual([agent.id]);
+ await expect.poll(() => ui.page.getByRole('heading', { name }).isVisible()).toBe(true);
+ await ui.settled();
+ },
+};
+
/** Signing out ends the session: the app returns to the login form and stays there. */
const signOut: BrowserScenario = {
header: browserHeader('gateway-sign-out', ['POST /gateway/auth/logout']),
@@ -71,5 +108,6 @@ export const BROWSER_SCENARIOS: readonly BrowserScenario[] = [
signIn,
everyPage,
roomFromList,
+ createRoom,
signOut,
];
diff --git a/console/tooling-e2e/src/clients/switch-client.ts b/console/tooling-e2e/src/clients/switch-client.ts
index ee6549577..39bb9bb73 100644
--- a/console/tooling-e2e/src/clients/switch-client.ts
+++ b/console/tooling-e2e/src/clients/switch-client.ts
@@ -107,6 +107,12 @@ export function eventText(event: AgentEvent): string {
return typeof body === 'string' ? body : '';
}
+/** What a route answered to a probe: its status, and what it said its body was. */
+export interface RouteProbe {
+ status: number;
+ contentType: string;
+}
+
export class SwitchHttpError extends Error {
readonly status: number;
readonly body: string;
@@ -221,16 +227,89 @@ export class SwitchClient {
* key, no body. For the check that every route which takes a session refuses
* a request without one.
*/
- async statusWithoutCredentials(method: string, path: string): Promise<number> {
+ async statusWithoutCredentials(method: string, path: string): Promise<RouteProbe> {
+ return this.probe(method, path, null);
+ }
+
+ /**
+ * What a request with this client's signed-in session, and no body, gets:
+ * its status and content type. For the check that one workspace's member
+ * is refused another workspace's resources.
+ */
+ async statusWithSession(method: string, path: string): Promise<RouteProbe> {
+ return this.probe(method, path, await this.gatewayLogin());
+ }
+
+ private async probe(method: string, path: string, cookie: string | null): Promise<RouteProbe> {
assertAllowed(this.environment, method, path);
const response = await fetch(`${this.apiUrl}${path}`, {
method,
- headers: { Accept: 'application/json', ...this.requestId() },
+ headers: {
+ Accept: 'application/json',
+ ...(cookie === null ? {} : { Cookie: cookie }),
+ ...this.requestId(),
+ },
redirect: 'manual',
signal: AbortSignal.timeout(30_000),
});
await response.body?.cancel();
- return response.status;
+ return { status: response.status, contentType: response.headers.get('content-type') ?? '' };
+ }
+
+ /**
+ * The same server, signed in as someone else. Shares this client's record of
+ * issued credentials, so the other session is scanned for too.
+ */
+ as(email: string, password: string): SwitchClient {
+ return new SwitchClient(
+ {
+ apiUrl: this.apiUrl,
+ agentRegistrationToken: this.registrationToken,
+ gatewayAdminEmail: email,
+ gatewayAdminPassword: password,
+ },
+ this.environment,
+ this.issued
+ );
+ }
+
+ /** `POST /gateway/users` — a user in the signed-in admin's workspace. `role: 'admin'` makes a deployment operator. */
+ async createUser(params: {
+ name: string;
+ email: string;
+ password: string;
+ role: 'user' | 'admin';
+ }): Promise<{ id: string }> {
+ return this.gateway<{ id: string }>('POST', '/users', params);
+ }
+
+ /**
+ * `POST /gateway/tenants` — create a workspace. The caller becomes its owner
+ * and the server re-mints their session into it, so from here on this client
+ * acts in the new workspace.
+ */
+ async createTenant(name: string): Promise<string> {
+ const { parsed, response } = await this.rawRequest('POST', '/gateway/tenants', {
+ cookie: await this.gatewayLogin(),
+ body: { name },
+ });
+ const cookie = sessionCookieFrom(response);
+ if (!cookie) throw new Error('Creating a workspace returned no switch_auth cookie for it.');
+ this.issued.record(cookie.slice('switch_auth='.length));
+ this.gatewayCookie = cookie;
+ return (parsed as { id: string }).id;
+ }
+
+ /** `POST /gateway/agents/register` — an agent owned by the signed-in user, in their current workspace. */
+ async registerAgentAsUser(params: { name: string; description: string }): Promise<AgentRef> {
+ const body = await this.gateway<{ id: string; api_key: string }>('POST', '/agents/register', {
+ agent_type: 'opencode',
+ name: params.name,
+ description: params.description,
+ options: { auto_session: false },
+ });
+ this.issued.record(body.api_key);
+ return { id: body.id, name: params.name };
}
/** `GET /gateway/feature-flags` — the flags this deployment has turned on. */
@@ -458,14 +537,44 @@ export class SwitchClient {
/** The bridge id of the default collaboration bridge of `type`. */
async defaultBridgeId(type: string): Promise<string> {
+ return (await this.defaultBridge(type)).id;
+ }
+
+ /** The default collaboration bridge of `type`: its id, and the display name a room document names it by. */
+ async defaultBridge(type: string): Promise<{ id: string; displayName: string }> {
const rows = await this.gateway<
- { id: string; bridge_type: string; is_default?: boolean; enabled?: boolean }[]
+ { id: string; bridge_type: string; display_name: string; is_default?: boolean }[]
>('GET', '/collaborations');
const ofType = rows.filter((row) => row.bridge_type === type);
if (ofType.length === 0) {
throw new Error(`No ${type} collaboration bridge registered on this Switch server`);
}
- return (ofType.find((row) => row.is_default) ?? ofType[0]!).id;
+ const row = ofType.find((candidate) => candidate.is_default) ?? ofType[0]!;
+ return { id: row.id, displayName: row.display_name };
+ }
+
+ /**
+ * `POST /gateway/rooms/from-yaml` — provision one room from a room document,
+ * as the signed-in admin; the gateway's side of `create_room_from_yaml`.
+ * Anything the room was given that could not be attached is a failure, not a
+ * room that is quietly missing part of its setup.
+ */
+ async createRoomFromYaml(yaml: string): Promise<RoomDetail> {
+ const result = await this.gateway<{
+ room_id?: string;
+ failed_attachments?: unknown[];
+ }>('POST', '/rooms/from-yaml', { yaml });
+ if (result.room_id === undefined) {
+ throw new Error(`from-yaml did not make one room: ${JSON.stringify(result)}`);
+ }
+ if (result.failed_attachments !== undefined && result.failed_attachments.length > 0) {
+ await this.deleteRoom(result.room_id);
+ throw new Error(
+ `from-yaml made room ${result.room_id} without part of what it was given, so it was deleted: ` +
+ JSON.stringify(result.failed_attachments)
+ );
+ }
+ return this.getRoom(result.room_id);
}
/**
diff --git a/console/tooling-e2e/src/fast/cross-tenant.ts b/console/tooling-e2e/src/fast/cross-tenant.ts
new file mode 100644
index 000000000..ad45a6bad
--- /dev/null
+++ b/console/tooling-e2e/src/fast/cross-tenant.ts
@@ -0,0 +1,130 @@
+import type { AgentRef, RoomDetail, SwitchClient } from '../clients/switch-client.ts';
+import type { NameScope } from '../core/run.ts';
+import type { ServerSurface } from '../gate/gate.ts';
+import { isHtml } from './session-required.ts';
+
+/**
+ * A check derived from the surface snapshot rather than written per route: a
+ * member of one workspace is refused another workspace's resources.
+ *
+ * A second workspace is made with a room and an agent in it, by a second
+ * deployment admin (an operator may create workspaces without a sign-up
+ * allowance). Then a plain member of the first workspace calls every session
+ * route whose path parameters all name something the second workspace has
+ * (its id, the room, the agent, the agent's name) with the second workspace's
+ * values, and no body.
+ *
+ * The prober is a plain member, not the harness admin, because a deployment
+ * admin is an operator and may cross workspaces by design.
+ *
+ * - 403 or 404 is the answer wanted.
+ * - 2xx or 3xx is a leak: the route served or changed another workspace's
+ * resource. 5xx, or an HTML page, is a failure too.
+ * - Anything else (400, 405, 409, 422: mostly a body the probe did not send)
+ * says nothing about isolation either way. Those are counted and listed,
+ * not failed: a validation error answered before the tenant is checked is
+ * not a leak.
+ *
+ * Workspaces cannot be deleted, so this runs only on a throwaway stack.
+ */
+
+export interface CrossTenantProbe {
+ /** Answered 403 or 404. */
+ refused: number;
+ /** Routes with a path parameter the second workspace has nothing for. */
+ unprobed: number;
+ /** Answered something that is neither a refusal nor a leak, with what. */
+ inconclusive: string[];
+ /** Served, changed or broke on another workspace's resource, with what. */
+ wrong: string[];
+}
+
+export interface OtherTenant {
+ tenantId: string;
+ room: RoomDetail;
+ agent: AgentRef;
+}
+
+/** Make the second workspace, its room and its agent, and the plain member who probes it. */
+export async function otherTenantAndProber(
+ admin: SwitchClient,
+ names: NameScope
+): Promise<{ other: OtherTenant; owner: SwitchClient; prober: SwitchClient }> {
+ const password = `e2e-${names.runId}-${crypto.randomUUID()}`;
+ const ownerEmail = `${names.name('owner')}@e2e.invalid`;
+ const proberEmail = `${names.name('prober')}@e2e.invalid`;
+ await admin.createUser({
+ name: 'E2E workspace owner',
+ email: ownerEmail,
+ password,
+ role: 'admin',
+ });
+ await admin.createUser({ name: 'E2E prober', email: proberEmail, password, role: 'user' });
+ const owner = admin.as(ownerEmail, password);
+ const tenantId = await owner.createTenant(names.name('space'));
+ const agent = await owner.registerAgentAsUser({
+ name: names.name('theirs'),
+ description: `Switch end-to-end harness, another workspace's agent (run ${names.runId}).`,
+ });
+ const room = await owner.createRoom({
+ name: names.name('their-room'),
+ description: `Switch end-to-end harness, another workspace's room (run ${names.runId}).`,
+ bridgeId: null,
+ agentNames: [agent.name],
+ userNames: [],
+ });
+ return { other: { tenantId, room, agent }, owner, prober: admin.as(proberEmail, password) };
+}
+
+export function crossTenantPath(
+ route: string,
+ other: OtherTenant
+): { method: string; path: string } | null {
+ const [method, template] = route.split(' ', 2) as [string, string];
+ const values: Record<string, string> = {
+ tenant_id: other.tenantId,
+ room_id: other.room.id,
+ agent_id: other.agent.id,
+ agent_name: other.agent.name,
+ };
+ let complete = true;
+ const path = template.replace(/\{([^}]+)\}/g, (_, param: string) => {
+ const value = values[param];
+ if (value === undefined) {
+ complete = false;
+ return '';
+ }
+ return encodeURIComponent(value);
+ });
+ return complete && path !== template ? { method, path } : null;
+}
+
+export async function probeCrossTenant(
+ prober: SwitchClient,
+ surface: ServerSurface,
+ enabledFlags: ReadonlySet<string>,
+ other: OtherTenant
+): Promise<CrossTenantProbe> {
+ const result: CrossTenantProbe = { refused: 0, unprobed: 0, inconclusive: [], wrong: [] };
+ // Reads first: a leaking DELETE would otherwise turn every later probe of the
+ // same resource into a 404 that looks like a refusal.
+ const routes = surface.gatewayRoutes
+ .filter(
+ ({ auth, route, flags }) =>
+ auth === 'session' && /\{/.test(route) && flags.every((f) => enabledFlags.has(f))
+ )
+ .sort((a, b) => Number(a.route.startsWith('DELETE ')) - Number(b.route.startsWith('DELETE ')));
+ for (const { route } of routes) {
+ const target = crossTenantPath(route, other);
+ if (target === null) {
+ result.unprobed += 1;
+ continue;
+ }
+ const { status, contentType } = await prober.statusWithSession(target.method, target.path);
+ const answered = `${route} -> ${status} ${contentType || '(no content type)'}`;
+ if (isHtml(contentType) || status < 400 || status >= 500) result.wrong.push(answered);
+ else if (status === 403 || status === 404) result.refused += 1;
+ else result.inconclusive.push(answered);
+ }
+ return result;
+}
diff --git a/console/tooling-e2e/src/fast/fast.e2e.test.ts b/console/tooling-e2e/src/fast/fast.e2e.test.ts
index 3ce722f62..2c04a8ffe 100644
--- a/console/tooling-e2e/src/fast/fast.e2e.test.ts
+++ b/console/tooling-e2e/src/fast/fast.e2e.test.ts
@@ -10,6 +10,7 @@ import {
import { resolveLane, type LaneResolution } from '../core/suite.ts';
import { Fixtures } from '../fixtures/fixtures.ts';
import { readSurface } from '../gate/gate.ts';
+import { otherTenantAndProber, probeCrossTenant } from './cross-tenant.ts';
import { FAST_SCENARIOS } from './index.ts';
import { probeSessionRoutes } from './session-required.ts';
@@ -105,6 +106,48 @@ describe('fast lane, derived checks', () => {
expectNoServerErrors(serverErrors, observer.tag, 'while this ran');
}, 120_000);
+ // Makes a workspace and two users, none of which can be deleted, so only on
+ // a stack that is thrown away after the run.
+ it("a workspace member is refused another workspace's rooms and agents", async (context) => {
+ if (!lane.target) return context.skip(lane.skip);
+ const { environment } = lane.target.env;
+ if (environment !== 'ephemeral' && environment !== 'local') {
+ return context.skip(
+ `leaves a workspace and users behind, which cannot be deleted; not on ${environment}`
+ );
+ }
+ const { switchClient } = lane.target;
+ const names = run.scope();
+ const observer = watch.begin(switchClient, names.tag);
+ let probe: Awaited<ReturnType<typeof probeCrossTenant>>;
+ let removeRoom: () => Promise<void>;
+ try {
+ const { other, owner, prober } = await otherTenantAndProber(switchClient, names);
+ probe = await probeCrossTenant(
+ prober,
+ readSurface(),
+ await switchClient.enabledFeatureFlags(),
+ other
+ );
+ removeRoom = () => owner.deleteRoom(other.room.id);
+ } finally {
+ observer.stop();
+ }
+ const serverErrors = observer.errors(null, console.log);
+ console.log(
+ `[cross-tenant] ${probe.refused} routes refused; ${probe.inconclusive.length} answered neither way; ` +
+ `${probe.unprobed} take a parameter the other workspace has nothing for`
+ );
+ for (const line of probe.inconclusive) console.log(`[cross-tenant] inconclusive: ${line}`);
+ expect(probe.refused, 'no route refused the other workspace').toBeGreaterThan(0);
+ expect(
+ probe.wrong,
+ `served another workspace's resource:\n ${probe.wrong.join('\n ')}\n`
+ ).toEqual([]);
+ expectNoServerErrors(serverErrors, observer.tag, 'while this ran');
+ await removeRoom();
+ }, 180_000);
+
// Last, so they see whatever the run left the server in.
it('the server is still ready after the run', async (context) => {
if (!lane.target) return context.skip(lane.skip);
diff --git a/console/tooling-e2e/src/fast/route-checks.test.ts b/console/tooling-e2e/src/fast/route-checks.test.ts
new file mode 100644
index 000000000..0d53db5dd
--- /dev/null
+++ b/console/tooling-e2e/src/fast/route-checks.test.ts
@@ -0,0 +1,43 @@
+import { describe, expect, it } from 'vitest';
+import type { AgentRef, RoomDetail } from '../clients/switch-client.ts';
+import { crossTenantPath, type OtherTenant } from './cross-tenant.ts';
+import { isHtml } from './session-required.ts';
+
+const OTHER: OtherTenant = {
+ tenantId: 't-1',
+ room: { id: 'r-1' } as RoomDetail,
+ agent: { id: 'a-1', name: 'e2e-abc123a-theirs' } as AgentRef,
+};
+
+describe('crossTenantPath', () => {
+ it("fills every parameter with the other workspace's values", () => {
+ expect(crossTenantPath('GET /gateway/rooms/{room_id}', OTHER)).toEqual({
+ method: 'GET',
+ path: '/gateway/rooms/r-1',
+ });
+ expect(crossTenantPath('DELETE /gateway/rooms/{room_id}/agents/{agent_id}', OTHER)).toEqual({
+ method: 'DELETE',
+ path: '/gateway/rooms/r-1/agents/a-1',
+ });
+ expect(crossTenantPath('GET /gateway/tenants/{tenant_id}', OTHER)?.path).toBe(
+ '/gateway/tenants/t-1'
+ );
+ });
+
+ it('skips a route with a parameter the other workspace has nothing for', () => {
+ expect(crossTenantPath('GET /gateway/rooms/{room_id}/tasks/{task_id}', OTHER)).toBeNull();
+ });
+
+ it('skips a route without parameters', () => {
+ expect(crossTenantPath('GET /gateway/rooms', OTHER)).toBeNull();
+ });
+});
+
+describe('isHtml', () => {
+ it('tells a web page from an API answer', () => {
+ expect(isHtml('text/html; charset=utf-8')).toBe(true);
+ expect(isHtml('Text/HTML')).toBe(true);
+ expect(isHtml('application/json')).toBe(false);
+ expect(isHtml('')).toBe(false);
+ });
+});
diff --git a/console/tooling-e2e/src/fast/session-required.ts b/console/tooling-e2e/src/fast/session-required.ts
index ead01c745..0ddea5af1 100644
--- a/console/tooling-e2e/src/fast/session-required.ts
+++ b/console/tooling-e2e/src/fast/session-required.ts
@@ -4,7 +4,10 @@ import type { ServerSurface } from '../gate/gate.ts';
/**
* A check derived from the surface snapshot rather than written per route:
* every gateway route that takes a signed-in session answers 401 to a request
- * that carries none.
+ * that carries none, and answers it as the API rather than with the gateway
+ * SPA's HTML page. A deployment that serves the SPA and the API from one host
+ * falls back to the page for any path it does not route, so an API route that
+ * went missing would otherwise look like a page that loaded.
*
* It covers nothing in the coverage gate's sense — a 401 says nothing about
* what the route does — but it is free for every route the server adds, and
@@ -30,10 +33,15 @@ export interface SessionProbe {
probed: number;
/** Session routes behind a feature flag this deployment has off. */
notMounted: number;
- /** Each probed route that did not answer 401, with what it answered. */
+ /** Each probed route that did not answer 401, or answered with HTML, with what it answered. */
wrong: string[];
}
+/** Whether a response is a web page rather than an API answer. */
+export function isHtml(contentType: string): boolean {
+ return contentType.toLowerCase().includes('text/html');
+}
+
export async function probeSessionRoutes(
switchClient: SwitchClient,
surface: ServerSurface
@@ -47,9 +55,11 @@ export async function probeSessionRoutes(
continue;
}
const { method, path } = probePath(route);
- const status = await switchClient.statusWithoutCredentials(method, path);
+ const { status, contentType } = await switchClient.statusWithoutCredentials(method, path);
result.probed += 1;
- if (status !== 401) result.wrong.push(`${route} -> ${status}`);
+ if (status !== 401 || isHtml(contentType)) {
+ result.wrong.push(`${route} -> ${status} ${contentType || '(no content type)'}`);
+ }
}
return result;
}
diff --git a/console/tooling-e2e/surface/exemptions.json b/console/tooling-e2e/surface/exemptions.json
index 1bf07c860..68f4b7f7f 100644
--- a/console/tooling-e2e/surface/exemptions.json
+++ b/console/tooling-e2e/surface/exemptions.json
@@ -282,7 +282,6 @@
"POST /gateway/rooms/bulk-delete",
"POST /gateway/rooms/from-template",
"POST /gateway/rooms/from-template-document",
- "POST /gateway/rooms/from-yaml",
"POST /gateway/rooms/{room_id}/agents",
"POST /gateway/rooms/{room_id}/archive",
"POST /gateway/rooms/{room_id}/documents/{document_id}",