Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Add an explore lane: Bombadil drives the gateway UI at random against properties

Petr Bauch5f28e9894fe8
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
index 7e64267eb..6e74772b2 100644
--- a/.github/workflows/e2e.yml
+++ b/.github/workflows/e2e.yml
@@ -11,6 +11,8 @@ name: E2E
# model plays the human, another judges the result
# browser the gateway UI in headless Chromium, built from this
# checkout
+# explore the same gateway UI, driven at random by Bombadil
+# against the properties in its specification
#
# Runs against the deployed environments are not here: they live with the
# deployment configuration, which checks out this harness at the commit the
@@ -50,7 +52,7 @@ on:
inputs:
lane:
type: choice
- options: [fast, provider, browser, agentic]
+ options: [fast, provider, browser, agentic, explore]
default: fast
agent_type:
description: Provider for the provider lane
@@ -85,7 +87,7 @@ jobs:
env:
AGENT_TYPE: ${{ inputs.agent_type }}
run: |
- case "$LANE" in fast|provider|browser|agentic) ;; *) echo "::error::Unknown lane '$LANE'."; exit 1 ;; esac
+ case "$LANE" in fast|provider|browser|agentic|explore) ;; *) echo "::error::Unknown lane '$LANE'."; exit 1 ;; esac
case "$LANE" in provider|agentic) REAL_SESSION=1 ;; *) REAL_SESSION=0 ;; esac
if [ "$REAL_SESSION" = 1 ] && [ "$AGENT_TYPE" != claude-code ]; then
echo "::error::CI has credentials for claude-code only, not '$AGENT_TYPE'."; exit 1
@@ -114,7 +116,7 @@ jobs:
run: pnpm install --frozen-lockfile
- name: Install Chromium
- if: ${{ inputs.lane == 'browser' }}
+ if: ${{ inputs.lane == 'browser' || inputs.lane == 'explore' }}
working-directory: console/tooling-e2e
run: pnpm exec playwright install --with-deps chromium
@@ -146,6 +148,8 @@ jobs:
SWITCH_E2E_DRIVER_MODEL: claude-sonnet-5
SWITCH_E2E_JUDGE_MODEL: claude-sonnet-5
SWITCH_E2E_AGENTIC_BUDGET_USD: '2'
+ # How long the explore lane explores.
+ SWITCH_E2E_EXPLORE_TIME_LIMIT: 10m
run: console/tooling-e2e/scripts/ephemeral-stack.sh
- name: Summarise the run
diff --git a/console/pnpm-lock.yaml b/console/pnpm-lock.yaml
index af0383e18..79d52f725 100644
--- a/console/pnpm-lock.yaml
+++ b/console/pnpm-lock.yaml
@@ -482,6 +482,9 @@ importers:
tooling-e2e:
devDependencies:
+ '@antithesishq/bombadil':
+ specifier: 0.7.10
+ version: 0.7.10
playwright:
specifier: 1.60.0
version: 1.60.0
@@ -561,6 +564,10 @@ packages:
zod:
optional: true
+ '@antithesishq/[email protected]':
+ resolution: {integrity: sha512-71aLKQBB3GNDBQZXU6MCKbt99dl9I+/pjg0EfGD9eCKXDIqpKPXD2rFH6yCJude37RGtnKq6hZFfZhuTp+Tykg==}
+ hasBin: true
+
'@asamuzakjp/[email protected]':
resolution: {integrity: sha512-KVw6qIiCTUQhByfTd78h2yD1/00waTmm9uy/R7Ck/ctUyAPj+AEDLkQIdJW0T8+qGgj3j5bpNKK7Q3G+LedJWg==}
engines: {node: ^20.19.0 || ^22.12.0 || >=24.0.0}
@@ -5864,6 +5871,8 @@ snapshots:
optionalDependencies:
zod: 4.4.3
+ '@antithesishq/[email protected]': {}
+
'@asamuzakjp/[email protected]':
dependencies:
'@asamuzakjp/generational-cache': 1.0.1
diff --git a/console/tooling-e2e/README.md b/console/tooling-e2e/README.md
index 7759fc615..db4d3619c 100644
--- a/console/tooling-e2e/README.md
+++ b/console/tooling-e2e/README.md
@@ -2,7 +2,7 @@
End-to-end tests for Switch: **real messages through a real Switch server**, and
through a real Mattermost bridge where a scenario crosses one. One package,
-four lanes:
+five lanes:
- **The fast lane** needs no provider, no Switch Console and no LLM key. The
harness plays every participant over plain HTTP: it registers throwaway
@@ -20,6 +20,10 @@ four lanes:
conversations come from scenario files: a model plays the human, and another
model grades the result against a rubric. Adding a scenario is adding a
file. See "The agentic lane" below.
+- **The explore lane** drives the same gateway UI as the browser lane, but at
+ random for a fixed time, with [Bombadil](https://github.com/antithesishq/bombadil).
+ Instead of following a script it checks properties every page must keep, in
+ every state it reaches. See "The explore lane" below.
Nothing here talks to Switch Console directly. If a scenario passes, an operator
doing the same thing by hand would have seen the same result.
@@ -83,9 +87,10 @@ Typecheck: `pnpm typecheck`.
| `src/fast/` | Fast-lane scenarios, one file per area, listed in `index.ts` |
| `src/provider/` | The provider lane: Console credential files, the shared agent and channel, the scenarios, `seed.ts` |
| `src/browser/` | The browser lane: the gateway UI driver (`gateway-ui.ts`) and the scenarios, listed in `index.ts` |
+| `src/explore/` | The explore lane: the Bombadil specification (`spec/gateway.ts`), the trace reader, the property list |
| `src/agentic/` | The agentic lane: scenario files' format, the driver and judge, model calls, spend |
| `agentic/` | The agentic scenarios, one YAML file each |
-| `scripts/ephemeral-stack.sh` | The throwaway-stack runner behind `just e2e-fast`, `just e2e-provider` and `just e2e-browser` |
+| `scripts/ephemeral-stack.sh` | The throwaway-stack runner behind the `just e2e-*` recipes |
| `scripts/scan-artifacts.sh` | Fails if a run's output holds one of the secrets the run used |
| `src/ci-summary.ts` | The CI run summary, read from the JUnit report |
| `src/gate/` | The coverage gate |
@@ -372,6 +377,59 @@ agent's session is nearly nine tenths of it ($0.045 to $0.21 a scenario,
most where it calls tools); the driver and judge together are about $0.01 a
scenario.
+## The explore lane
+
+`just e2e-explore` sets up what `just e2e-browser` does, then hands the gateway
+UI to [Bombadil](https://github.com/antithesishq/bombadil) for
+`SWITCH_E2E_EXPLORE_TIME_LIMIT` (`5m` by default, `10m` in CI). Bombadil
+clicks, types, scrolls and navigates at random, and after every action checks
+the properties that `src/explore/spec/gateway.ts` exports:
+
+| Property | Holds when |
+| --- | --- |
+| `noUncaughtExceptions`, `noUnhandledPromiseRejections` | no page throws (Bombadil's defaults) |
+| `noConsoleErrors` | nothing calls `console.error` (Bombadil's default) |
+| `noErrorBanner` | no page opens with an MUI error alert (one after a click is the gateway refusing random input) |
+| `noServerErrors` | the server answers no `/gateway` request with a 5xx |
+| `signsIn` | the explorer gets past the login page within 30 s |
+| `staysSignedIn` | once signed in, it is never sent back to the login page |
+
+These are the browser lane's rules for a page that renders but is broken,
+checked everywhere the explorer reaches rather than on a scripted path. A 4xx
+is not a failure here, because random input is supposed to be refused. As in
+the browser lane, the server log must stay free of errors for the whole run.
+
+The explorer deletes, archives and renames whatever it finds, so the lane
+refuses to run on anything but an ephemeral stack. It cannot sign out or
+change the password, since that would end the session it is testing.
+Everything else is fair game, including deleting things behind a confirmation
+dialog. The lane seeds an agent and two rooms first, so the pages have
+something on them.
+
+The explorer signs in from inside the page, using a custom action with the
+admin's credentials. Those credentials are written next to a copy of the spec
+in a private temporary directory. A session cookie is never passed on
+Bombadil's command line, because Bombadil prints its arguments back, and
+nothing an action types is a secret, because actions are written to the trace.
+
+**Reading a failure.** The JUnit report has one case for the exploration, one
+per property, and one for the server log. A failed property case says when and
+on which page the property first failed, what the condition saw, and which
+screenshot shows it. Bombadil's own log, `trace.jsonl` and screenshots are
+written to `build/e2e/ephemeral/explore/` (`explore/` in the CI artifact). To
+step through the run state by state:
+
+```bash
+pnpm --dir console/tooling-e2e exec bombadil browser inspect build/e2e/ephemeral/explore
+```
+
+**Adding a property.** Export it from `spec/gateway.ts` and add its name to
+`src/explore/properties.ts`; a unit test keeps the two lists the same. The spec
+runs inside Bombadil, not Node, so it may import only Bombadil's modules and
+JSON. It has its own `tsconfig.json`, and `pnpm typecheck` checks it. The
+[Bombadil manual](https://antithesishq.github.io/bombadil/) covers
+extractors, the temporal operators and action generators.
+
## Server health and errors
A scenario can pass every assertion while the server logs an error and carries
@@ -478,7 +536,7 @@ hand.
| After a merge to main (`e2e-scheduled.yml`) | provider | ephemeral | no |
| Nightly (`e2e-scheduled.yml`) | provider | ephemeral | no |
| A release tag, candidates included (`e2e-scheduled.yml`) | fast; provider; agentic | ephemeral | no |
-| By hand (`e2e.yml`, dispatch) | any | ephemeral | no |
+| By hand (`e2e.yml`, dispatch) | any, including browser and explore | ephemeral | no |
- **Secrets.** An ephemeral fast run generates every secret it uses on the
runner, so it works the same on a fork PR. Every other run reads its secrets
diff --git a/console/tooling-e2e/package.json b/console/tooling-e2e/package.json
index bca6e1544..561602646 100644
--- a/console/tooling-e2e/package.json
+++ b/console/tooling-e2e/package.json
@@ -8,9 +8,10 @@
"test": "vitest run",
"test:e2e": "vitest run",
"cleanup": "node --experimental-strip-types src/cleanup.ts",
- "typecheck": "tsc --noEmit"
+ "typecheck": "tsc --noEmit && tsc --noEmit -p src/explore/spec"
},
"devDependencies": {
+ "@antithesishq/bombadil": "0.7.10",
"playwright": "1.60.0",
"typescript": "^5.9.2",
"vitest": "^5.0.1",
diff --git a/console/tooling-e2e/scripts/ephemeral-stack.sh b/console/tooling-e2e/scripts/ephemeral-stack.sh
index dd9425b4e..d1df6ff37 100755
--- a/console/tooling-e2e/scripts/ephemeral-stack.sh
+++ b/console/tooling-e2e/scripts/ephemeral-stack.sh
@@ -7,6 +7,7 @@
# just e2e-provider the provider lane, on a headless agent controller
# just e2e-browser the browser lane: the gateway UI in headless Chromium
# just e2e-agentic the agentic lane: scenario files, a driver and a judge model
+# just e2e-explore the explore lane: Bombadil drives the gateway UI at random
# just e2e-fast --keep leave the stack running afterwards
# just e2e-fast -- <vitest args> pass arguments through to vitest
#
@@ -20,9 +21,10 @@
# own .env is never read or written, so this cannot rotate a developer's live
# secrets or pick up a stale value from them.
#
-# The browser lane also builds the gateway UI from gateway/ and serves it with
-# `vite preview` on :4173, whose proxy sends its /gateway requests to :8000 as
-# nginx does in a deployment, and installs Playwright's Chromium.
+# The browser and explore lanes also build the gateway UI from gateway/ and
+# serve it with `vite preview` on :4173, whose proxy sends its /gateway requests
+# to :8000 as nginx does in a deployment, and install Playwright's Chromium.
+# SWITCH_E2E_EXPLORE_TIME_LIMIT sets how long the explore lane explores (5m).
#
# Ports 4173, 5432, 8000, 8065 and 8081 must be free: stop a local stack first.
# Logs and the JUnit report go to build/e2e/ephemeral/ (or $E2E_OUT); the env
@@ -236,7 +238,7 @@ if [ "$NEEDS_CONTROLLER" = 1 ]; then
fi
BROWSER_ENV=()
-if [ "$LANE" = browser ]; then
+if [ "$LANE" = browser ] || [ "$LANE" = explore ]; then
echo "Building the gateway UI…"
(cd "$REPO/gateway" && npm ci --no-audit --no-fund && npm run build) >"$OUT/gateway-build.log" 2>&1 \
|| { echo "gateway UI build failed; see $OUT/gateway-build.log" >&2; exit 1; }
@@ -283,5 +285,6 @@ if [ "$status" != 0 ]; then
echo "The $LANE lane failed. switch-core log: $OUT/switch-core.log" >&2
[ "$NEEDS_CONTROLLER" = 1 ] && echo "Agent controller log: $OUT/controller.log" >&2
[ "$LANE" = browser ] && echo "Screenshots of failed pages: $OUT/screenshots" >&2
+ [ "$LANE" = explore ] && echo "The explorer's trace: pnpm --dir $HARNESS exec bombadil browser inspect $OUT/explore" >&2
fi
exit "$status"
diff --git a/console/tooling-e2e/src/core/env.ts b/console/tooling-e2e/src/core/env.ts
index 158e99390..23dfac5b4 100644
--- a/console/tooling-e2e/src/core/env.ts
+++ b/console/tooling-e2e/src/core/env.ts
@@ -20,12 +20,14 @@ import { fileURLToPath } from 'node:url';
* Chromium against the same server: a smoke test of what an operator sees,
* which no HTTP-level scenario can see break. `agentic` runs the same real
* session as `provider`, but a model plays the human from a scenario file and
- * another model judges the conversation.
+ * another model judges the conversation. `explore` drives the same gateway UI
+ * as `browser`, but at random for a fixed time (Bombadil), checking properties
+ * every page must keep rather than following a script.
*/
-export type Lane = 'fast' | 'provider' | 'browser' | 'agentic';
+export type Lane = 'fast' | 'provider' | 'browser' | 'agentic' | 'explore';
/** Every lane: the one list anything that takes a lane by name checks against. */
-export const LANES: readonly Lane[] = ['fast', 'provider', 'browser', 'agentic'];
+export const LANES: readonly Lane[] = ['fast', 'provider', 'browser', 'agentic', 'explore'];
export function isLane(raw: string): raw is Lane {
return (LANES as readonly string[]).includes(raw);
@@ -132,11 +134,19 @@ export interface HarnessEnv {
serverLog: string | null;
/**
* The gateway UI, served against `switch.apiUrl` (its `/gateway` requests
- * reach that server). Only the browser lane reads it.
+ * reach that server). Only the browser and explore lanes read it.
*/
gatewayUiUrl: string | null;
- /** Where the browser lane saves a screenshot of a page that failed, if anywhere. */
+ /**
+ * Where the browser lane saves a screenshot of a page that failed, and the
+ * explore lane its trace, if anywhere.
+ */
outputDir: string | null;
+ /**
+ * How long the explore lane explores, as Bombadil's `--time-limit` takes it
+ * (`90s`, `5m`, `1h`). `SWITCH_E2E_EXPLORE_TIME_LIMIT`, default `5m`.
+ */
+ exploreTimeLimit: string;
switch: SwitchEnv;
/** `null` when no Mattermost is configured; scenarios that need one then skip or fail. */
mattermost: MattermostEnv | null;
@@ -236,6 +246,33 @@ export function strictFrom(
return reader(source, file)('SWITCH_E2E_STRICT') === '1';
}
+/**
+ * The explore lane's time limit. Read on its own as well as by `loadEnv`,
+ * because the lane sizes its case's timeout from it before the rest of the
+ * configuration is loaded.
+ */
+export function exploreTimeLimitFrom(
+ source: Source = process.env,
+ file: Record<string, string> = readDotEnvFile()
+): string {
+ const raw = reader(source, file)('SWITCH_E2E_EXPLORE_TIME_LIMIT') ?? '5m';
+ durationMs(raw);
+ return raw;
+}
+
+const DURATION_UNITS_MS = { s: 1_000, m: 60_000, h: 3_600_000, d: 86_400_000 } as const;
+
+/** A duration as Bombadil's `--time-limit` takes it (`90s`, `5m`, `1h`, `1d`), in milliseconds. */
+export function durationMs(raw: string): number {
+ const match = /^([1-9][0-9]*)([smhd])$/.exec(raw);
+ if (match === null) {
+ throw new Error(
+ `SWITCH_E2E_EXPLORE_TIME_LIMIT must be a whole number with a unit, s, m, h or d (e.g. 5m) — got '${raw}'.`
+ );
+ }
+ return Number(match[1]) * DURATION_UNITS_MS[match[2] as keyof typeof DURATION_UNITS_MS];
+}
+
/**
* Resolve the harness configuration from `process.env` layered over `.env`.
* Throws {@link MissingConfigError} naming every absent variable at once rather
@@ -288,6 +325,7 @@ export function loadEnv(
serverLog: get('SWITCH_E2E_SERVER_LOG') ?? null,
gatewayUiUrl: optionalUrl(get('SWITCH_GATEWAY_UI_URL')),
outputDir: get('SWITCH_E2E_OUTPUT_DIR') ?? null,
+ exploreTimeLimit: exploreTimeLimitFrom(source, file),
switch: {
apiUrl: stripTrailingSlash(required.SWITCH_API_URL as string),
gatewayAdminEmail: required.SWITCH_GATEWAY_ADMIN_EMAIL as string,
@@ -383,6 +421,6 @@ export function describeEnv(env: HarnessEnv): string {
const mattermost = env.mattermost
? `mattermost=${env.mattermost.url} team=${env.mattermost.team}`
: 'mattermost=none';
- const ui = env.lane === 'browser' ? ` gateway-ui=${env.gatewayUiUrl ?? 'none'}` : '';
+ const ui = env.lane === 'browser' || env.lane === 'explore' ? ` gateway-ui=${env.gatewayUiUrl ?? 'none'}` : '';
return `lane=${env.lane} environment=${env.environment} switch=${env.switch.apiUrl} ${mattermost}${ui}`;
}
diff --git a/console/tooling-e2e/src/explore/explore.e2e.test.ts b/console/tooling-e2e/src/explore/explore.e2e.test.ts
new file mode 100644
index 000000000..947860aa5
--- /dev/null
+++ b/console/tooling-e2e/src/explore/explore.e2e.test.ts
@@ -0,0 +1,267 @@
+import { spawn } from 'node:child_process';
+import {
+ copyFileSync,
+ createWriteStream,
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ rmSync,
+ writeFileSync,
+} from 'node:fs';
+import { tmpdir } from 'node:os';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { chromium } from 'playwright';
+import { afterAll, beforeAll, describe, expect, it } from 'vitest';
+import { SwitchHttpError, type SwitchClient } from '../clients/switch-client.ts';
+import { describeEnv, durationMs, exploreTimeLimitFrom, type HarnessEnv } from '../core/env.ts';
+import { RunContext, type NameScope } from '../core/run.ts';
+import {
+ describeServerLogCheck,
+ expectNoServerErrors,
+ ServerLogWatch,
+ type ScenarioObserver,
+ type ServerLogError,
+} from '../core/server-log.ts';
+import { resolveLane, type LaneResolution } from '../core/suite.ts';
+import { Fixtures } from '../fixtures/fixtures.ts';
+import { PROPERTIES } from './properties.ts';
+import { readTrace, type TraceSummary } from './trace.ts';
+
+/**
+ * The explore lane: Bombadil drives the gateway UI at random for
+ * `SWITCH_E2E_EXPLORE_TIME_LIMIT`, checking the properties in `spec/gateway.ts`
+ * in every state it reaches. Runs when `SWITCH_E2E_LANE=explore`, with
+ * `SWITCH_GATEWAY_UI_URL` naming where the UI is served.
+ *
+ * The explorer deletes, archives and renames whatever it finds, so the lane
+ * runs only on an ephemeral stack. It starts on the login page and signs in
+ * from there (the spec's `signIn`), so no session cookie is ever passed on
+ * Bombadil's command line, which it prints back. The admin's credentials go
+ * beside a copy of the spec in a private temporary directory, never into the
+ * output.
+ *
+ * One case explores; then one case per property reports what the trace says
+ * about it, with the first violating state's page and screenshot; then the
+ * server log is checked for the whole exploration, as the browser lane checks
+ * it per scenario. Bombadil's own log and trace go to
+ * `$SWITCH_E2E_OUTPUT_DIR/explore`, readable with `bombadil browser inspect`.
+ */
+
+const SPEC_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), 'spec');
+
+/** Bombadil's exit codes (`bombadil browser test`). */
+const EXIT_CLEAN = 0;
+const EXIT_STALLED = 1;
+const EXIT_VIOLATIONS = 2;
+
+/** Over the time limit, what the explorer may take to start Chromium and write its trace. */
+const SLACK_MS = 3 * 60_000;
+
+const run = new RunContext();
+const timeLimitMs = durationMs(exploreTimeLimitFrom());
+let lane: LaneResolution;
+let watch: ServerLogWatch;
+let fixtures: Fixtures | null = null;
+let names: NameScope;
+let observer: ScenarioObserver | null = null;
+let outputDir: string;
+let trace: TraceSummary | null = null;
+let serverErrors: ServerLogError[] | null = null;
+
+beforeAll(async () => {
+ lane = await resolveLane('explore');
+ if (!lane.target) return;
+ const { env, switchClient, mattermost } = lane.target;
+ if (env.environment !== 'ephemeral') {
+ // The explorer deletes what it finds; on any other stack that is someone's data.
+ const reason = `the explore lane runs only on an ephemeral stack, not '${env.environment}'`;
+ if (env.strict) throw new Error(`The explore lane cannot run (SWITCH_E2E_STRICT=1): ${reason}`);
+ lane = { target: null, skip: reason };
+ return;
+ }
+ if (env.gatewayUiUrl === null) {
+ throw new Error('The explore lane needs SWITCH_GATEWAY_UI_URL: where the gateway UI is served.');
+ }
+ console.log(`explore lane run ${run.runId}: ${describeEnv(env)}`);
+ console.log(` ${describeServerLogCheck(env.serverLog)}`);
+ watch = new ServerLogWatch(env.serverLog);
+ outputDir =
+ env.outputDir === null
+ ? mkdtempSync(path.join(tmpdir(), 'switch-explore-'))
+ : path.join(env.outputDir, 'explore');
+ names = run.scope();
+ fixtures = new Fixtures(env, names, switchClient, mattermost);
+ await seed(fixtures, switchClient, names);
+}, 120_000);
+
+afterAll(async () => {
+ observer?.stop();
+ if (fixtures === null) return;
+ const leftovers = await fixtures.teardown();
+ for (const left of leftovers) console.log(`[explore] teardown: ${left}`);
+});
+
+describe('explore lane', () => {
+ it(`explores the gateway UI`, { timeout: timeLimitMs + SLACK_MS }, async (context) => {
+ if (!lane.target) return context.skip(lane.skip);
+ const { env, switchClient } = lane.target;
+ observer = watch.begin(switchClient, names.tag, 'explore');
+ const exit = await explore(env, names.tag);
+ serverErrors = observer.errors(null, (message) => console.log(`[explore] ${message}`));
+ observer.stop();
+ observer = null;
+
+ const traceFile = path.join(outputDir, 'trace.jsonl');
+ if (!existsSync(traceFile)) {
+ throw new Error(`Bombadil exited ${exit} and wrote no trace; see ${logFile()}.`);
+ }
+ trace = readTrace(traceFile);
+ console.log(
+ `[explore] ${trace.states} states, ${trace.violations.size} properties violated; ` +
+ `inspect with: pnpm exec bombadil browser inspect ${outputDir}`
+ );
+ if (exit === EXIT_STALLED) {
+ throw new Error(`The browser stalled: Bombadil stopped getting new states; see ${logFile()}.`);
+ }
+ if (exit !== EXIT_CLEAN && exit !== EXIT_VIOLATIONS) {
+ throw new Error(`Bombadil exited ${exit}; see ${logFile()}.`);
+ }
+ const unknown = [...trace.violations.keys()].filter(
+ (name) => !(PROPERTIES as readonly string[]).includes(name)
+ );
+ expect(unknown, 'violated properties missing from PROPERTIES').toEqual([]);
+ expect(exit === EXIT_VIOLATIONS, 'Bombadil exit code agrees with the trace').toBe(
+ trace.violations.size > 0
+ );
+ });
+
+ for (const property of PROPERTIES) {
+ it(property, (context) => {
+ if (!lane.target) return context.skip(lane.skip);
+ if (trace === null) throw new Error('The exploration did not finish, so nothing was checked.');
+ const violation = trace.violations.get(property);
+ if (violation === undefined) return;
+ throw new Error(
+ `${property} was violated ${(violation.firstAtMicros / 1e6).toFixed(1)} s in, on ${violation.url}, ` +
+ `and in ${violation.states} of ${trace.states} states: ${violation.detail}\n` +
+ ` screenshot: ${violation.screenshot}`
+ );
+ });
+ }
+
+ it('the server logged no errors while it explored', (context) => {
+ if (!lane.target) return context.skip(lane.skip);
+ if (trace === null) throw new Error('The exploration did not finish, so nothing was checked.');
+ expectNoServerErrors(serverErrors, names.tag, 'while the explorer ran');
+ });
+
+ it('the server logged no errors outside the exploration', async (context) => {
+ if (!lane.target) return context.skip(lane.skip);
+ expectNoServerErrors(await watch.sweep(), null, 'outside the exploration');
+ });
+});
+
+/**
+ * Something on the pages: an agent, in two rooms (an internal room needs one).
+ * The explorer may delete them before teardown does, so one already gone is
+ * not a leftover.
+ */
+async function seed(fixtures: Fixtures, client: SwitchClient, names: NameScope): Promise<void> {
+ const agentName = names.name('agent');
+ await client.registerKnownAgent({
+ agentType: 'opencode',
+ name: agentName,
+ description: `Switch end-to-end harness agent (run ${names.runId}). Safe to delete.`,
+ options: { auto_session: false },
+ });
+ fixtures.defer(`agent ${agentName}`, () => unlessGone(client.deleteAgentByName(agentName)));
+ for (const role of ['room', 'other-room']) {
+ const name = names.name(role);
+ const room = await client.createRoom({
+ name,
+ description: `Switch end-to-end harness (run ${names.runId}). Safe to delete.`,
+ bridgeId: null,
+ agentNames: [agentName],
+ userNames: [],
+ });
+ fixtures.defer(`room ${name} (${room.id})`, () => unlessGone(client.deleteRoom(room.id)));
+ }
+}
+
+function logFile(): string {
+ return path.join(outputDir, 'bombadil.log');
+}
+
+async function unlessGone(removal: Promise<void>): Promise<void> {
+ try {
+ await removal;
+ } catch (error) {
+ if (error instanceof SwitchHttpError && error.status === 404) return;
+ throw error;
+ }
+}
+
+/**
+ * Run Bombadil against the gateway UI, and return its exit code.
+ *
+ * Its output goes to a log file rather than this process's: it prints a line
+ * per action, thousands in a run. Every request the page makes carries the
+ * scope's tag, so an error the server logs serving one is attributed here.
+ */
+async function explore(env: HarnessEnv, tag: string): Promise<number> {
+ rmSync(outputDir, { recursive: true, force: true });
+ mkdirSync(outputDir, { recursive: true });
+ const specDir = mkdtempSync(path.join(tmpdir(), 'switch-explore-spec-'));
+ try {
+ copyFileSync(path.join(SPEC_DIR, 'gateway.ts'), path.join(specDir, 'gateway.ts'));
+ writeFileSync(
+ path.join(specDir, 'credentials.json'),
+ JSON.stringify({
+ email: env.switch.gatewayAdminEmail,
+ password: env.switch.gatewayAdminPassword,
+ }),
+ { mode: 0o600 }
+ );
+ const args = [
+ 'exec',
+ 'bombadil',
+ 'browser',
+ 'test',
+ '--headless',
+ '--no-sandbox',
+ '--time-limit',
+ env.exploreTimeLimit,
+ '--output-path',
+ outputDir,
+ '--width',
+ '1440',
+ '--height',
+ '900',
+ '--header',
+ `X-Request-Id=${tag}-ui`,
+ `${env.gatewayUiUrl}/login`,
+ path.join(specDir, 'gateway.ts'),
+ ];
+ console.log(`[explore] exploring for ${env.exploreTimeLimit}; Bombadil's log: ${logFile()}`);
+ const log = createWriteStream(logFile());
+ const child = spawn('pnpm', args, {
+ // Bombadil finds no browser of its own; Playwright's is installed for this lane.
+ env: { ...process.env, CHROME: chromium.executablePath() },
+ stdio: ['ignore', 'pipe', 'pipe'],
+ });
+ child.stdout.pipe(log);
+ child.stderr.pipe(log);
+ const code = await new Promise<number>((resolve, reject) => {
+ child.on('error', reject);
+ child.on('close', (exitCode, signal) =>
+ exitCode === null ? reject(new Error(`Bombadil was killed by ${signal}.`)) : resolve(exitCode)
+ );
+ });
+ await new Promise((resolve) => log.end(resolve));
+ return code;
+ } finally {
+ rmSync(specDir, { recursive: true, force: true });
+ }
+}
+
diff --git a/console/tooling-e2e/src/explore/explore.test.ts b/console/tooling-e2e/src/explore/explore.test.ts
new file mode 100644
index 000000000..fd77d534e
--- /dev/null
+++ b/console/tooling-e2e/src/explore/explore.test.ts
@@ -0,0 +1,97 @@
+import { readFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { describe, expect, it } from 'vitest';
+import { ACTION_GENERATORS, PROPERTIES } from './properties.ts';
+import { describeViolation, summarizeTrace } from './trace.ts';
+
+const SPEC = path.join(path.dirname(fileURLToPath(import.meta.url)), 'spec', 'gateway.ts');
+
+/** Every name the spec exports: its own `export const`s and what it re-exports. */
+function specExports(source: string): string[] {
+ const own = [...source.matchAll(/^export const (\w+)/gm)].map((match) => match[1]!);
+ const reexported = [...source.matchAll(/^export \{([^}]*)\} from/gm)].flatMap((match) =>
+ match[1]!
+ .split(',')
+ .map((name) => name.trim())
+ .filter((name) => name !== '')
+ );
+ return [...own, ...reexported];
+}
+
+const banner = (condition: string, value: unknown) => ({
+ Always: {
+ violation: {
+ False: { time: 2, condition, snapshots: [{ index: 1, name: 'errorBanners', value, time: 2 }] },
+ },
+ subformula: { Thunk: { function: condition, negated: false } },
+ start: 1,
+ end: null,
+ time: 2,
+ },
+});
+
+const line = (timestamp: number, url: string, violations: unknown[]) =>
+ JSON.stringify({
+ timestamp,
+ action: null,
+ state: { url, screenshot: `/out/screenshots/${timestamp}.jpeg`, resources: {} },
+ snapshots: [],
+ violations,
+ });
+
+describe('explore lane', () => {
+ it("lists exactly the spec's properties", () => {
+ const exported = specExports(readFileSync(SPEC, 'utf8'));
+ expect([...exported].sort()).toEqual([...PROPERTIES, ...ACTION_GENERATORS].sort());
+ });
+
+ it('reports each property by the first state it was violated in', () => {
+ const condition = '() => errorBanners.current.length === 0';
+ const trace = [
+ line(1_000_000, 'http://ui/rooms', []),
+ line(1_250_000, 'http://ui/agents', [{ name: 'noErrorBanner', violation: banner(condition, ['Boom']) }]),
+ line(1_500_000, 'http://ui/rooms', [{ name: 'noErrorBanner', violation: banner(condition, ['Boom']) }]),
+ '',
+ ].join('\n');
+ const summary = summarizeTrace(trace);
+ expect(summary.states).toBe(3);
+ expect([...summary.violations.values()]).toEqual([
+ {
+ name: 'noErrorBanner',
+ states: 2,
+ firstAtMicros: 250_000,
+ url: 'http://ui/agents',
+ screenshot: '/out/screenshots/1250000.jpeg',
+ detail: `${condition}, with errorBanners = ["Boom"]`,
+ },
+ ]);
+ });
+
+ it('says which formula an eventually was still waiting for', () => {
+ const violation = {
+ Eventually: {
+ subformula: { Thunk: { function: '() => !onLoginPage.current', negated: false } },
+ reason: { TimedOut: 30_000_000 },
+ },
+ };
+ expect(describeViolation(violation)).toBe(
+ 'never held within its time limit: () => !onLoginPage.current'
+ );
+ });
+
+ it('describes the consequent of an implication', () => {
+ const violation = {
+ Implies: {
+ left: { Thunk: { function: '() => a', negated: false } },
+ right: { False: { time: 1, condition: '() => b', snapshots: [] } },
+ antecedent_snapshots: [],
+ },
+ };
+ expect(describeViolation(violation)).toBe('() => b');
+ });
+
+ it('refuses a trace line that is not JSON', () => {
+ expect(() => summarizeTrace('{"timestamp":')).toThrow(/trace line 1 is not JSON/);
+ });
+});
diff --git a/console/tooling-e2e/src/explore/properties.ts b/console/tooling-e2e/src/explore/properties.ts
new file mode 100644
index 000000000..f8f4040c4
--- /dev/null
+++ b/console/tooling-e2e/src/explore/properties.ts
@@ -0,0 +1,18 @@
+/**
+ * The properties `spec/gateway.ts` exports, by export name, which is the name
+ * Bombadil reports a violation under. The lane reports one case per property,
+ * and vitest needs the cases before the run that finds the violations, so the
+ * list is written out here; a unit test holds it to the spec's exports.
+ */
+export const PROPERTIES = [
+ 'noConsoleErrors',
+ 'noUncaughtExceptions',
+ 'noUnhandledPromiseRejections',
+ 'signsIn',
+ 'staysSignedIn',
+ 'noErrorBanner',
+ 'noServerErrors',
+] as const;
+
+/** The spec's other exports: its action generators. */
+export const ACTION_GENERATORS = ['signingIn', 'exploring'] as const;
diff --git a/console/tooling-e2e/src/explore/spec/credentials.json b/console/tooling-e2e/src/explore/spec/credentials.json
new file mode 100644
index 000000000..dd78002c7
--- /dev/null
+++ b/console/tooling-e2e/src/explore/spec/credentials.json
@@ -0,0 +1 @@
+{ "email": "", "password": "" }
diff --git a/console/tooling-e2e/src/explore/spec/gateway.ts b/console/tooling-e2e/src/explore/spec/gateway.ts
new file mode 100644
index 000000000..86f3c7df8
--- /dev/null
+++ b/console/tooling-e2e/src/explore/spec/gateway.ts
@@ -0,0 +1,188 @@
+/**
+ * The explore lane's Bombadil specification: what every gateway page must keep
+ * to, and what the explorer may do, while it drives the UI at random.
+ *
+ * This runs inside Bombadil, not Node: it may import only Bombadil's own
+ * modules and JSON. Every property and action generator exported here is used,
+ * so nothing else may be exported.
+ *
+ * The properties port the browser lane's rules for a page that renders but is
+ * broken (`src/browser/gateway-ui.ts`), checked in every state the explorer
+ * reaches instead of on a scripted path. A 4xx is not one of them: random input
+ * is meant to be refused.
+ */
+import { always, eventually, now } from '@antithesishq/bombadil';
+import { isAction } from '@antithesishq/bombadil/actions';
+import {
+ actions,
+ extract,
+ registerCustomAction,
+ weighted,
+ type ActionTemplate,
+} from '@antithesishq/bombadil/browser';
+import {
+ clicks,
+ inputs,
+ navigation,
+ scroll,
+} from '@antithesishq/bombadil/browser/defaults/actions';
+// The admin's credentials, written by the lane beside a copy of this file for
+// the run; the copy in the repository is an empty placeholder.
+import credentials from './credentials.json';
+
+export {
+ noConsoleErrors,
+ noUncaughtExceptions,
+ noUnhandledPromiseRejections,
+} from '@antithesishq/bombadil/browser/defaults/properties';
+
+/** The pages in the navigation rail, and the form for a new room. */
+const ROUTES = [
+ '/rooms',
+ '/rooms/new',
+ '/agents',
+ '/resources',
+ '/collaborations',
+ '/registration-keys',
+ '/workspace',
+ '/usage',
+ '/ecosystem',
+];
+
+/**
+ * What would end the session the explorer is testing. Signing out and back in
+ * proves nothing the browser lane does not, and a changed password locks the
+ * lane out for the rest of the run.
+ */
+const SESSION_ENDING = ['Sign out', 'Change password'];
+
+const path = extract((state) => state.window.location.pathname);
+
+/**
+ * Whether the login form is shown. Not the path: going back to `/login` while
+ * signed in shows that path for a moment, until the app redirects to `/`.
+ */
+const onLoginPage = extract(
+ (state) =>
+ state.document.querySelector('form input[type="password"]') !== null &&
+ state.document.querySelector('nav[aria-label="Main"]') === null
+);
+
+/** Whether the app's navigation rail is shown: only to someone signed in. */
+const signedIn = extract(
+ (state) => state.document.querySelector('nav[aria-label="Main"]') !== null
+);
+
+/**
+ * Whether the last action opened a page: a link, back, forward, reload or a
+ * jump. Read loosely because Bombadil hands the action over in its trace shape
+ * (snake case, empty fields left out), not the one its types declare.
+ */
+const openedPage = extract((state) => {
+ const action = state.lastAction as unknown;
+ if (action === null || typeof action === 'string') return action !== 'Wait';
+ const { Click, Custom } = action as {
+ Click?: { fingerprint: { tag: string; href?: string } };
+ Custom?: { name: string };
+ };
+ if (Click !== undefined) return Click.fingerprint.tag === 'a' && Click.fingerprint.href !== undefined;
+ return Custom !== undefined && (Custom.name === 'goTo' || Custom.name === 'signIn');
+});
+
+const errorBanners = extract((state) =>
+ Array.from(state.document.querySelectorAll('[role="alert"].MuiAlert-colorError'))
+ .filter((element) => element.getClientRects().length > 0)
+ .map((element) => (element.textContent ?? '').trim().slice(0, 200))
+);
+
+/**
+ * Every `/gateway/` response since the page last loaded that the server
+ * failed. Resource timing keeps 250 entries by default, fewer than a long
+ * session on one page makes, so the buffer is raised before it is read.
+ */
+const serverErrors = extract((state) => {
+ const performance = state.window.performance;
+ performance.setResourceTimingBufferSize(10_000);
+ return performance
+ .getEntriesByType('resource')
+ .filter(
+ (entry): entry is PerformanceResourceTiming =>
+ entry instanceof PerformanceResourceTiming &&
+ new URL(entry.name).pathname.startsWith('/gateway/') &&
+ entry.responseStatus >= 500
+ )
+ .map((entry) => `${entry.responseStatus} ${new URL(entry.name).pathname}`);
+});
+
+/** The explorer signs in from the login page it starts on. */
+export const signsIn = eventually(() => signedIn.current).within(30, 'seconds');
+
+/**
+ * Once signed in, the explorer stays signed in. It cannot sign out (see
+ * `SESSION_ENDING`), so the login page again means the session was lost.
+ */
+export const staysSignedIn = always(
+ now(() => !onLoginPage.current).implies(always(() => !onLoginPage.current))
+);
+
+/**
+ * No page opens with an error banner: MUI renders each as an `alert` with error
+ * severity. A banner after a click is how the gateway refuses what random input
+ * asked for (removing a member who owns agents, say), so only a banner that
+ * comes with opening a page counts.
+ */
+export const noErrorBanner = always(
+ now(() => openedPage.current).implies(() => errorBanners.current.length === 0)
+);
+
+/** The server answered no `/gateway` request with a 5xx. */
+export const noServerErrors = always(() => serverErrors.current.length === 0);
+
+/**
+ * Sign in as the gateway admin, from the login page.
+ *
+ * Over `fetch` rather than the form, and with no arguments, because an action's
+ * arguments are written to the trace the lane uploads: typed into the form, the
+ * password would be too.
+ */
+const signIn = registerCustomAction('signIn', async (_document, window) => {
+ if (credentials.email === '' || credentials.password === '') {
+ throw new Error(
+ 'credentials.json is the empty placeholder: run this spec through the explore lane.'
+ );
+ }
+ const response = await window.fetch('/gateway/auth/login', {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json' },
+ body: JSON.stringify(credentials),
+ credentials: 'same-origin',
+ });
+ if (!response.ok) throw new Error(`Signing in answered ${response.status}.`);
+ // Replacing the login page in history, so going back does not restore it.
+ window.location.replace('/rooms');
+});
+
+const goTo = registerCustomAction('goTo', async (_document, window, route: string) => {
+ window.location.assign(route);
+});
+
+export const signingIn = actions(() => [signIn()]).when(() => onLoginPage.current);
+
+function endsSession(action: ActionTemplate): boolean {
+ if (!isAction(action, 'Click')) return false;
+ const { textContent, accessibleName } = action.Click.fingerprint;
+ return SESSION_ENDING.some((label) => textContent === label || accessibleName === label);
+}
+
+/**
+ * Bombadil's default exploration, without what ends the session, plus a jump to
+ * any page in the rail so a dead end (a dialog with nothing to click, a page
+ * scrolled away from its links) does not hold the explorer.
+ */
+export const exploring = weighted([
+ [100, clicks.filter((action) => !endsSession(action))],
+ [100, inputs],
+ [50, scroll],
+ [10, navigation],
+ [5, actions(() => ROUTES.filter((route) => route !== path.current).map((route) => goTo(route)))],
+]).when(() => !onLoginPage.current);
diff --git a/console/tooling-e2e/src/explore/spec/tsconfig.json b/console/tooling-e2e/src/explore/spec/tsconfig.json
new file mode 100644
index 000000000..25fda6aaa
--- /dev/null
+++ b/console/tooling-e2e/src/explore/spec/tsconfig.json
@@ -0,0 +1,16 @@
+{
+ "compilerOptions": {
+ "target": "ES2023",
+ "module": "ESNext",
+ "moduleResolution": "bundler",
+ "lib": ["ES2023", "DOM", "DOM.Iterable"],
+ "types": [],
+ "strict": true,
+ "noUncheckedIndexedAccess": true,
+ "noEmit": true,
+ "skipLibCheck": true,
+ "resolveJsonModule": true,
+ "verbatimModuleSyntax": true
+ },
+ "include": ["*.ts"]
+}
diff --git a/console/tooling-e2e/src/explore/trace.ts b/console/tooling-e2e/src/explore/trace.ts
new file mode 100644
index 000000000..dfd286027
--- /dev/null
+++ b/console/tooling-e2e/src/explore/trace.ts
@@ -0,0 +1,129 @@
+import { readFileSync } from 'node:fs';
+
+/**
+ * What Bombadil's `trace.jsonl` says about the properties: one line per state
+ * it captured, each listing the properties violated in that state. A property
+ * stays violated once it is, so a violation is reported by its first state.
+ */
+
+/** The parts of a trace line the lane reads. */
+interface TraceLine {
+ timestamp: number;
+ state: { url: string; screenshot: string };
+ violations: { name: string; violation: unknown }[];
+}
+
+export interface PropertyViolation {
+ name: string;
+ /** How many states it was reported in. */
+ states: number;
+ /** Microseconds since the first state captured. */
+ firstAtMicros: number;
+ url: string;
+ screenshot: string;
+ /** The condition that failed, and the extracted values it failed on. */
+ detail: string;
+}
+
+export interface TraceSummary {
+ states: number;
+ violations: Map<string, PropertyViolation>;
+}
+
+export function readTrace(file: string): TraceSummary {
+ return summarizeTrace(readFileSync(file, 'utf8'));
+}
+
+export function summarizeTrace(raw: string): TraceSummary {
+ const violations = new Map<string, PropertyViolation>();
+ let states = 0;
+ let start: number | null = null;
+ for (const [index, text] of raw.split('\n').entries()) {
+ if (text.trim() === '') continue;
+ let line: TraceLine;
+ try {
+ line = JSON.parse(text) as TraceLine;
+ } catch (error) {
+ throw new Error(
+ `trace line ${index + 1} is not JSON: ${error instanceof Error ? error.message : String(error)}`
+ );
+ }
+ states += 1;
+ start ??= line.timestamp;
+ for (const { name, violation } of line.violations) {
+ const seen = violations.get(name);
+ if (seen !== undefined) {
+ seen.states += 1;
+ continue;
+ }
+ violations.set(name, {
+ name,
+ states: 1,
+ firstAtMicros: line.timestamp - start,
+ url: line.state.url,
+ screenshot: line.state.screenshot,
+ detail: describeViolation(violation),
+ });
+ }
+ }
+ return { states, violations };
+}
+
+/** A violation as Bombadil serializes it: one variant per formula that can fail. */
+type Violation =
+ | { False: { condition: string; snapshots: { name: string | null; value: unknown }[] } }
+ | { Always: { violation: Violation } }
+ | { Eventually: { subformula: unknown; reason: 'TestEnded' | { TimedOut: number } } }
+ | { Implies: { right: Violation } }
+ | { And: { left: Violation; right: Violation } }
+ | { Or: { left: Violation; right: Violation } };
+
+/**
+ * The condition that was false, with the values it was false on, or the
+ * formula an `eventually` was still waiting for. Bombadil wraps a violation
+ * once per operator around it, and both sides of an `and` or `or`.
+ */
+export function describeViolation(raw: unknown): string {
+ const violation = raw as Violation;
+ if ('False' in violation) {
+ const { condition, snapshots } = violation.False;
+ const values = snapshots
+ .map((snapshot) => `${snapshot.name ?? '?'} = ${JSON.stringify(snapshot.value)}`)
+ .join('; ');
+ return values === '' ? condition : `${condition}, with ${values}`;
+ }
+ if ('Always' in violation) return describeViolation(violation.Always.violation);
+ if ('Implies' in violation) return describeViolation(violation.Implies.right);
+ if ('And' in violation) {
+ return `${describeViolation(violation.And.left)} and ${describeViolation(violation.And.right)}`;
+ }
+ if ('Or' in violation) {
+ return `${describeViolation(violation.Or.left)} and ${describeViolation(violation.Or.right)}`;
+ }
+ if ('Eventually' in violation) {
+ const { subformula, reason } = violation.Eventually;
+ const why = reason === 'TestEnded' ? 'by the end of the run' : 'within its time limit';
+ return `never held ${why}: ${formulaText(subformula)}`;
+ }
+ throw new Error(`Unknown violation in the trace: ${JSON.stringify(raw).slice(0, 300)}`);
+}
+
+/** A formula as Bombadil serializes it, written back the way a specification writes it. */
+function formulaText(raw: unknown): string {
+ const formula = raw as Record<string, unknown>;
+ if ('Thunk' in formula) return (formula.Thunk as { function: string }).function;
+ if ('Pure' in formula) return (formula.Pure as { pretty: string }).pretty;
+ for (const operator of ['And', 'Or', 'Implies'] as const) {
+ if (operator in formula) {
+ const [left, right] = formula[operator] as [unknown, unknown];
+ return `(${formulaText(left)}) ${operator.toLowerCase()} (${formulaText(right)})`;
+ }
+ }
+ for (const operator of ['Next', 'Always', 'Eventually'] as const) {
+ if (operator in formula) {
+ const inner = formula[operator];
+ return `${operator.toLowerCase()}(${formulaText(Array.isArray(inner) ? inner[0] : inner)})`;
+ }
+ }
+ throw new Error(`Unknown formula in the trace: ${JSON.stringify(raw).slice(0, 300)}`);
+}
diff --git a/console/tooling-e2e/tsconfig.json b/console/tooling-e2e/tsconfig.json
index 641ae596e..a1866a376 100644
--- a/console/tooling-e2e/tsconfig.json
+++ b/console/tooling-e2e/tsconfig.json
@@ -22,5 +22,8 @@
"include": [
"src/**/*.ts",
"vitest.config.ts"
+ ],
+ "exclude": [
+ "src/explore/spec"
]
}
diff --git a/justfile b/justfile
index 85963459e..cba829484 100644
--- a/justfile
+++ b/justfile
@@ -190,6 +190,14 @@ e2e-provider *args:
e2e-browser *args:
SWITCH_E2E_LANE=browser console/tooling-e2e/scripts/ephemeral-stack.sh {{ args }}
+# ── End-to-end harness, explore lane, on a throwaway stack ─────────────────────
+# As e2e-browser, but Bombadil drives the gateway UI at random for
+# SWITCH_E2E_EXPLORE_TIME_LIMIT (default 5m), checking the properties in
+# console/tooling-e2e/src/explore/spec/gateway.ts in every state. Its trace:
+# build/e2e/ephemeral/explore, read with `bombadil browser inspect`.
+e2e-explore *args:
+ SWITCH_E2E_LANE=explore console/tooling-e2e/scripts/ephemeral-stack.sh {{ args }}
+
# ── End-to-end harness, agentic lane, on a throwaway stack ─────────────────────
# As e2e-provider, then the scenario files in console/tooling-e2e/agentic: a
# driver model plays the human, a judge model grades the conversation. Needs