Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Add an explore lane: Bombadil drives the gateway UI at random against properties
Petr Bauch5f28e9894fe8
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.ymlindex 7e64267eb..6e74772b2 100644--- a/.github/workflows/e2e.yml+++ b/.github/workflows/e2e.yml@@ -11,6 +11,8 @@ name: E2E# model plays the human, another judges the result# browser the gateway UI in headless Chromium, built from this# checkout+# explore the same gateway UI, driven at random by Bombadil+# against the properties in its specification## Runs against the deployed environments are not here: they live with the# deployment configuration, which checks out this harness at the commit the@@ -50,7 +52,7 @@ on:inputs:lane:type: choice- options: [fast, provider, browser, agentic]+ options: [fast, provider, browser, agentic, explore]default: fastagent_type:description: Provider for the provider lane@@ -85,7 +87,7 @@ jobs:env:AGENT_TYPE: ${{ inputs.agent_type }}run: |- case "$LANE" in fast|provider|browser|agentic) ;; *) echo "::error::Unknown lane '$LANE'."; exit 1 ;; esac+ case "$LANE" in fast|provider|browser|agentic|explore) ;; *) echo "::error::Unknown lane '$LANE'."; exit 1 ;; esaccase "$LANE" in provider|agentic) REAL_SESSION=1 ;; *) REAL_SESSION=0 ;; esacif [ "$REAL_SESSION" = 1 ] && [ "$AGENT_TYPE" != claude-code ]; thenecho "::error::CI has credentials for claude-code only, not '$AGENT_TYPE'."; exit 1@@ -114,7 +116,7 @@ jobs:run: pnpm install --frozen-lockfile- name: Install Chromium- if: ${{ inputs.lane == 'browser' }}+ if: ${{ inputs.lane == 'browser' || inputs.lane == 'explore' }}working-directory: console/tooling-e2erun: pnpm exec playwright install --with-deps chromium@@ -146,6 +148,8 @@ jobs:SWITCH_E2E_DRIVER_MODEL: claude-sonnet-5SWITCH_E2E_JUDGE_MODEL: claude-sonnet-5SWITCH_E2E_AGENTIC_BUDGET_USD: '2'+ # How long the explore lane explores.+ SWITCH_E2E_EXPLORE_TIME_LIMIT: 10mrun: console/tooling-e2e/scripts/ephemeral-stack.sh- name: Summarise the rundiff --git a/console/pnpm-lock.yaml b/console/pnpm-lock.yamlindex af0383e18..79d52f725 100644--- a/console/pnpm-lock.yaml+++ b/console/pnpm-lock.yaml@@ -482,6 +482,9 @@ importers:tooling-e2e:devDependencies:+ '@antithesishq/bombadil':+ specifier: 0.7.10+ version: 0.7.10playwright:specifier: 1.60.0version: 1.60.0@@ -561,6 +564,10 @@ packages:zod:optional: true+ '@antithesishq/[email protected]':+ resolution: {integrity: sha512-71aLKQBB3GNDBQZXU6MCKbt99dl9I+/pjg0EfGD9eCKXDIqpKPXD2rFH6yCJude37RGtnKq6hZFfZhuTp+Tykg==}+ hasBin: true+'@asamuzakjp/[email protected]':resolution: {integrity: sha512-KVw6qIiCTUQhByfTd78h2yD1/00waTmm9uy/R7Ck/ctUyAPj+AEDLkQIdJW0T8+qGgj3j5bpNKK7Q3G+LedJWg==}engines: {node: ^20.19.0 || ^22.12.0 || >=24.0.0}@@ -5864,6 +5871,8 @@ snapshots:optionalDependencies:zod: 4.4.3+ '@antithesishq/[email protected]': {}+'@asamuzakjp/[email protected]':dependencies:'@asamuzakjp/generational-cache': 1.0.1diff --git a/console/tooling-e2e/README.md b/console/tooling-e2e/README.mdindex 7759fc615..db4d3619c 100644--- a/console/tooling-e2e/README.md+++ b/console/tooling-e2e/README.md@@ -2,7 +2,7 @@End-to-end tests for Switch: **real messages through a real Switch server**, andthrough a real Mattermost bridge where a scenario crosses one. One package,-four lanes:+five lanes:- **The fast lane** needs no provider, no Switch Console and no LLM key. Theharness plays every participant over plain HTTP: it registers throwaway@@ -20,6 +20,10 @@ four lanes:conversations come from scenario files: a model plays the human, and anothermodel grades the result against a rubric. Adding a scenario is adding afile. See "The agentic lane" below.+- **The explore lane** drives the same gateway UI as the browser lane, but at+ random for a fixed time, with [Bombadil](https://github.com/antithesishq/bombadil).+ Instead of following a script it checks properties every page must keep, in+ every state it reaches. See "The explore lane" below.Nothing here talks to Switch Console directly. If a scenario passes, an operatordoing the same thing by hand would have seen the same result.@@ -83,9 +87,10 @@ Typecheck: `pnpm typecheck`.| `src/fast/` | Fast-lane scenarios, one file per area, listed in `index.ts` || `src/provider/` | The provider lane: Console credential files, the shared agent and channel, the scenarios, `seed.ts` || `src/browser/` | The browser lane: the gateway UI driver (`gateway-ui.ts`) and the scenarios, listed in `index.ts` |+| `src/explore/` | The explore lane: the Bombadil specification (`spec/gateway.ts`), the trace reader, the property list || `src/agentic/` | The agentic lane: scenario files' format, the driver and judge, model calls, spend || `agentic/` | The agentic scenarios, one YAML file each |-| `scripts/ephemeral-stack.sh` | The throwaway-stack runner behind `just e2e-fast`, `just e2e-provider` and `just e2e-browser` |+| `scripts/ephemeral-stack.sh` | The throwaway-stack runner behind the `just e2e-*` recipes || `scripts/scan-artifacts.sh` | Fails if a run's output holds one of the secrets the run used || `src/ci-summary.ts` | The CI run summary, read from the JUnit report || `src/gate/` | The coverage gate |@@ -372,6 +377,59 @@ agent's session is nearly nine tenths of it ($0.045 to $0.21 a scenario,most where it calls tools); the driver and judge together are about $0.01 ascenario.+## The explore lane++`just e2e-explore` sets up what `just e2e-browser` does, then hands the gateway+UI to [Bombadil](https://github.com/antithesishq/bombadil) for+`SWITCH_E2E_EXPLORE_TIME_LIMIT` (`5m` by default, `10m` in CI). Bombadil+clicks, types, scrolls and navigates at random, and after every action checks+the properties that `src/explore/spec/gateway.ts` exports:++| Property | Holds when |+| --- | --- |+| `noUncaughtExceptions`, `noUnhandledPromiseRejections` | no page throws (Bombadil's defaults) |+| `noConsoleErrors` | nothing calls `console.error` (Bombadil's default) |+| `noErrorBanner` | no page opens with an MUI error alert (one after a click is the gateway refusing random input) |+| `noServerErrors` | the server answers no `/gateway` request with a 5xx |+| `signsIn` | the explorer gets past the login page within 30 s |+| `staysSignedIn` | once signed in, it is never sent back to the login page |++These are the browser lane's rules for a page that renders but is broken,+checked everywhere the explorer reaches rather than on a scripted path. A 4xx+is not a failure here, because random input is supposed to be refused. As in+the browser lane, the server log must stay free of errors for the whole run.++The explorer deletes, archives and renames whatever it finds, so the lane+refuses to run on anything but an ephemeral stack. It cannot sign out or+change the password, since that would end the session it is testing.+Everything else is fair game, including deleting things behind a confirmation+dialog. The lane seeds an agent and two rooms first, so the pages have+something on them.++The explorer signs in from inside the page, using a custom action with the+admin's credentials. Those credentials are written next to a copy of the spec+in a private temporary directory. A session cookie is never passed on+Bombadil's command line, because Bombadil prints its arguments back, and+nothing an action types is a secret, because actions are written to the trace.++**Reading a failure.** The JUnit report has one case for the exploration, one+per property, and one for the server log. A failed property case says when and+on which page the property first failed, what the condition saw, and which+screenshot shows it. Bombadil's own log, `trace.jsonl` and screenshots are+written to `build/e2e/ephemeral/explore/` (`explore/` in the CI artifact). To+step through the run state by state:++```bash+pnpm --dir console/tooling-e2e exec bombadil browser inspect build/e2e/ephemeral/explore+```++**Adding a property.** Export it from `spec/gateway.ts` and add its name to+`src/explore/properties.ts`; a unit test keeps the two lists the same. The spec+runs inside Bombadil, not Node, so it may import only Bombadil's modules and+JSON. It has its own `tsconfig.json`, and `pnpm typecheck` checks it. The+[Bombadil manual](https://antithesishq.github.io/bombadil/) covers+extractors, the temporal operators and action generators.+## Server health and errorsA scenario can pass every assertion while the server logs an error and carries@@ -478,7 +536,7 @@ hand.| After a merge to main (`e2e-scheduled.yml`) | provider | ephemeral | no || Nightly (`e2e-scheduled.yml`) | provider | ephemeral | no || A release tag, candidates included (`e2e-scheduled.yml`) | fast; provider; agentic | ephemeral | no |-| By hand (`e2e.yml`, dispatch) | any | ephemeral | no |+| By hand (`e2e.yml`, dispatch) | any, including browser and explore | ephemeral | no |- **Secrets.** An ephemeral fast run generates every secret it uses on therunner, so it works the same on a fork PR. Every other run reads its secretsdiff --git a/console/tooling-e2e/package.json b/console/tooling-e2e/package.jsonindex bca6e1544..561602646 100644--- a/console/tooling-e2e/package.json+++ b/console/tooling-e2e/package.json@@ -8,9 +8,10 @@"test": "vitest run","test:e2e": "vitest run","cleanup": "node --experimental-strip-types src/cleanup.ts",- "typecheck": "tsc --noEmit"+ "typecheck": "tsc --noEmit && tsc --noEmit -p src/explore/spec"},"devDependencies": {+ "@antithesishq/bombadil": "0.7.10","playwright": "1.60.0","typescript": "^5.9.2","vitest": "^5.0.1",diff --git a/console/tooling-e2e/scripts/ephemeral-stack.sh b/console/tooling-e2e/scripts/ephemeral-stack.shindex dd9425b4e..d1df6ff37 100755--- a/console/tooling-e2e/scripts/ephemeral-stack.sh+++ b/console/tooling-e2e/scripts/ephemeral-stack.sh@@ -7,6 +7,7 @@# just e2e-provider the provider lane, on a headless agent controller# just e2e-browser the browser lane: the gateway UI in headless Chromium# just e2e-agentic the agentic lane: scenario files, a driver and a judge model+# just e2e-explore the explore lane: Bombadil drives the gateway UI at random# just e2e-fast --keep leave the stack running afterwards# just e2e-fast -- <vitest args> pass arguments through to vitest#@@ -20,9 +21,10 @@# own .env is never read or written, so this cannot rotate a developer's live# secrets or pick up a stale value from them.#-# The browser lane also builds the gateway UI from gateway/ and serves it with-# `vite preview` on :4173, whose proxy sends its /gateway requests to :8000 as-# nginx does in a deployment, and installs Playwright's Chromium.+# The browser and explore lanes also build the gateway UI from gateway/ and+# serve it with `vite preview` on :4173, whose proxy sends its /gateway requests+# to :8000 as nginx does in a deployment, and install Playwright's Chromium.+# SWITCH_E2E_EXPLORE_TIME_LIMIT sets how long the explore lane explores (5m).## Ports 4173, 5432, 8000, 8065 and 8081 must be free: stop a local stack first.# Logs and the JUnit report go to build/e2e/ephemeral/ (or $E2E_OUT); the env@@ -236,7 +238,7 @@ if [ "$NEEDS_CONTROLLER" = 1 ]; thenfiBROWSER_ENV=()-if [ "$LANE" = browser ]; then+if [ "$LANE" = browser ] || [ "$LANE" = explore ]; thenecho "Building the gateway UI…"(cd "$REPO/gateway" && npm ci --no-audit --no-fund && npm run build) >"$OUT/gateway-build.log" 2>&1 \|| { echo "gateway UI build failed; see $OUT/gateway-build.log" >&2; exit 1; }@@ -283,5 +285,6 @@ if [ "$status" != 0 ]; thenecho "The $LANE lane failed. switch-core log: $OUT/switch-core.log" >&2[ "$NEEDS_CONTROLLER" = 1 ] && echo "Agent controller log: $OUT/controller.log" >&2[ "$LANE" = browser ] && echo "Screenshots of failed pages: $OUT/screenshots" >&2+ [ "$LANE" = explore ] && echo "The explorer's trace: pnpm --dir $HARNESS exec bombadil browser inspect $OUT/explore" >&2fiexit "$status"diff --git a/console/tooling-e2e/src/core/env.ts b/console/tooling-e2e/src/core/env.tsindex 158e99390..23dfac5b4 100644--- a/console/tooling-e2e/src/core/env.ts+++ b/console/tooling-e2e/src/core/env.ts@@ -20,12 +20,14 @@ import { fileURLToPath } from 'node:url';* Chromium against the same server: a smoke test of what an operator sees,* which no HTTP-level scenario can see break. `agentic` runs the same real* session as `provider`, but a model plays the human from a scenario file and- * another model judges the conversation.+ * another model judges the conversation. `explore` drives the same gateway UI+ * as `browser`, but at random for a fixed time (Bombadil), checking properties+ * every page must keep rather than following a script.*/-export type Lane = 'fast' | 'provider' | 'browser' | 'agentic';+export type Lane = 'fast' | 'provider' | 'browser' | 'agentic' | 'explore';/** Every lane: the one list anything that takes a lane by name checks against. */-export const LANES: readonly Lane[] = ['fast', 'provider', 'browser', 'agentic'];+export const LANES: readonly Lane[] = ['fast', 'provider', 'browser', 'agentic', 'explore'];export function isLane(raw: string): raw is Lane {return (LANES as readonly string[]).includes(raw);@@ -132,11 +134,19 @@ export interface HarnessEnv {serverLog: string | null;/*** The gateway UI, served against `switch.apiUrl` (its `/gateway` requests- * reach that server). Only the browser lane reads it.+ * reach that server). Only the browser and explore lanes read it.*/gatewayUiUrl: string | null;- /** Where the browser lane saves a screenshot of a page that failed, if anywhere. */+ /**+ * Where the browser lane saves a screenshot of a page that failed, and the+ * explore lane its trace, if anywhere.+ */outputDir: string | null;+ /**+ * How long the explore lane explores, as Bombadil's `--time-limit` takes it+ * (`90s`, `5m`, `1h`). `SWITCH_E2E_EXPLORE_TIME_LIMIT`, default `5m`.+ */+ exploreTimeLimit: string;switch: SwitchEnv;/** `null` when no Mattermost is configured; scenarios that need one then skip or fail. */mattermost: MattermostEnv | null;@@ -236,6 +246,33 @@ export function strictFrom(return reader(source, file)('SWITCH_E2E_STRICT') === '1';}+/**+ * The explore lane's time limit. Read on its own as well as by `loadEnv`,+ * because the lane sizes its case's timeout from it before the rest of the+ * configuration is loaded.+ */+export function exploreTimeLimitFrom(+ source: Source = process.env,+ file: Record<string, string> = readDotEnvFile()+): string {+ const raw = reader(source, file)('SWITCH_E2E_EXPLORE_TIME_LIMIT') ?? '5m';+ durationMs(raw);+ return raw;+}++const DURATION_UNITS_MS = { s: 1_000, m: 60_000, h: 3_600_000, d: 86_400_000 } as const;++/** A duration as Bombadil's `--time-limit` takes it (`90s`, `5m`, `1h`, `1d`), in milliseconds. */+export function durationMs(raw: string): number {+ const match = /^([1-9][0-9]*)([smhd])$/.exec(raw);+ if (match === null) {+ throw new Error(+ `SWITCH_E2E_EXPLORE_TIME_LIMIT must be a whole number with a unit, s, m, h or d (e.g. 5m) — got '${raw}'.`+ );+ }+ return Number(match[1]) * DURATION_UNITS_MS[match[2] as keyof typeof DURATION_UNITS_MS];+}+/*** Resolve the harness configuration from `process.env` layered over `.env`.* Throws {@link MissingConfigError} naming every absent variable at once rather@@ -288,6 +325,7 @@ export function loadEnv(serverLog: get('SWITCH_E2E_SERVER_LOG') ?? null,gatewayUiUrl: optionalUrl(get('SWITCH_GATEWAY_UI_URL')),outputDir: get('SWITCH_E2E_OUTPUT_DIR') ?? null,+ exploreTimeLimit: exploreTimeLimitFrom(source, file),switch: {apiUrl: stripTrailingSlash(required.SWITCH_API_URL as string),gatewayAdminEmail: required.SWITCH_GATEWAY_ADMIN_EMAIL as string,@@ -383,6 +421,6 @@ export function describeEnv(env: HarnessEnv): string {const mattermost = env.mattermost? `mattermost=${env.mattermost.url} team=${env.mattermost.team}`: 'mattermost=none';- const ui = env.lane === 'browser' ? ` gateway-ui=${env.gatewayUiUrl ?? 'none'}` : '';+ const ui = env.lane === 'browser' || env.lane === 'explore' ? ` gateway-ui=${env.gatewayUiUrl ?? 'none'}` : '';return `lane=${env.lane} environment=${env.environment} switch=${env.switch.apiUrl} ${mattermost}${ui}`;}diff --git a/console/tooling-e2e/src/explore/explore.e2e.test.ts b/console/tooling-e2e/src/explore/explore.e2e.test.tsnew file mode 100644index 000000000..947860aa5--- /dev/null+++ b/console/tooling-e2e/src/explore/explore.e2e.test.ts@@ -0,0 +1,267 @@+import { spawn } from 'node:child_process';+import {+ copyFileSync,+ createWriteStream,+ existsSync,+ mkdirSync,+ mkdtempSync,+ rmSync,+ writeFileSync,+} from 'node:fs';+import { tmpdir } from 'node:os';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { chromium } from 'playwright';+import { afterAll, beforeAll, describe, expect, it } from 'vitest';+import { SwitchHttpError, type SwitchClient } from '../clients/switch-client.ts';+import { describeEnv, durationMs, exploreTimeLimitFrom, type HarnessEnv } from '../core/env.ts';+import { RunContext, type NameScope } from '../core/run.ts';+import {+ describeServerLogCheck,+ expectNoServerErrors,+ ServerLogWatch,+ type ScenarioObserver,+ type ServerLogError,+} from '../core/server-log.ts';+import { resolveLane, type LaneResolution } from '../core/suite.ts';+import { Fixtures } from '../fixtures/fixtures.ts';+import { PROPERTIES } from './properties.ts';+import { readTrace, type TraceSummary } from './trace.ts';++/**+ * The explore lane: Bombadil drives the gateway UI at random for+ * `SWITCH_E2E_EXPLORE_TIME_LIMIT`, checking the properties in `spec/gateway.ts`+ * in every state it reaches. Runs when `SWITCH_E2E_LANE=explore`, with+ * `SWITCH_GATEWAY_UI_URL` naming where the UI is served.+ *+ * The explorer deletes, archives and renames whatever it finds, so the lane+ * runs only on an ephemeral stack. It starts on the login page and signs in+ * from there (the spec's `signIn`), so no session cookie is ever passed on+ * Bombadil's command line, which it prints back. The admin's credentials go+ * beside a copy of the spec in a private temporary directory, never into the+ * output.+ *+ * One case explores; then one case per property reports what the trace says+ * about it, with the first violating state's page and screenshot; then the+ * server log is checked for the whole exploration, as the browser lane checks+ * it per scenario. Bombadil's own log and trace go to+ * `$SWITCH_E2E_OUTPUT_DIR/explore`, readable with `bombadil browser inspect`.+ */++const SPEC_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), 'spec');++/** Bombadil's exit codes (`bombadil browser test`). */+const EXIT_CLEAN = 0;+const EXIT_STALLED = 1;+const EXIT_VIOLATIONS = 2;++/** Over the time limit, what the explorer may take to start Chromium and write its trace. */+const SLACK_MS = 3 * 60_000;++const run = new RunContext();+const timeLimitMs = durationMs(exploreTimeLimitFrom());+let lane: LaneResolution;+let watch: ServerLogWatch;+let fixtures: Fixtures | null = null;+let names: NameScope;+let observer: ScenarioObserver | null = null;+let outputDir: string;+let trace: TraceSummary | null = null;+let serverErrors: ServerLogError[] | null = null;++beforeAll(async () => {+ lane = await resolveLane('explore');+ if (!lane.target) return;+ const { env, switchClient, mattermost } = lane.target;+ if (env.environment !== 'ephemeral') {+ // The explorer deletes what it finds; on any other stack that is someone's data.+ const reason = `the explore lane runs only on an ephemeral stack, not '${env.environment}'`;+ if (env.strict) throw new Error(`The explore lane cannot run (SWITCH_E2E_STRICT=1): ${reason}`);+ lane = { target: null, skip: reason };+ return;+ }+ if (env.gatewayUiUrl === null) {+ throw new Error('The explore lane needs SWITCH_GATEWAY_UI_URL: where the gateway UI is served.');+ }+ console.log(`explore lane run ${run.runId}: ${describeEnv(env)}`);+ console.log(` ${describeServerLogCheck(env.serverLog)}`);+ watch = new ServerLogWatch(env.serverLog);+ outputDir =+ env.outputDir === null+ ? mkdtempSync(path.join(tmpdir(), 'switch-explore-'))+ : path.join(env.outputDir, 'explore');+ names = run.scope();+ fixtures = new Fixtures(env, names, switchClient, mattermost);+ await seed(fixtures, switchClient, names);+}, 120_000);++afterAll(async () => {+ observer?.stop();+ if (fixtures === null) return;+ const leftovers = await fixtures.teardown();+ for (const left of leftovers) console.log(`[explore] teardown: ${left}`);+});++describe('explore lane', () => {+ it(`explores the gateway UI`, { timeout: timeLimitMs + SLACK_MS }, async (context) => {+ if (!lane.target) return context.skip(lane.skip);+ const { env, switchClient } = lane.target;+ observer = watch.begin(switchClient, names.tag, 'explore');+ const exit = await explore(env, names.tag);+ serverErrors = observer.errors(null, (message) => console.log(`[explore] ${message}`));+ observer.stop();+ observer = null;++ const traceFile = path.join(outputDir, 'trace.jsonl');+ if (!existsSync(traceFile)) {+ throw new Error(`Bombadil exited ${exit} and wrote no trace; see ${logFile()}.`);+ }+ trace = readTrace(traceFile);+ console.log(+ `[explore] ${trace.states} states, ${trace.violations.size} properties violated; ` ++ `inspect with: pnpm exec bombadil browser inspect ${outputDir}`+ );+ if (exit === EXIT_STALLED) {+ throw new Error(`The browser stalled: Bombadil stopped getting new states; see ${logFile()}.`);+ }+ if (exit !== EXIT_CLEAN && exit !== EXIT_VIOLATIONS) {+ throw new Error(`Bombadil exited ${exit}; see ${logFile()}.`);+ }+ const unknown = [...trace.violations.keys()].filter(+ (name) => !(PROPERTIES as readonly string[]).includes(name)+ );+ expect(unknown, 'violated properties missing from PROPERTIES').toEqual([]);+ expect(exit === EXIT_VIOLATIONS, 'Bombadil exit code agrees with the trace').toBe(+ trace.violations.size > 0+ );+ });++ for (const property of PROPERTIES) {+ it(property, (context) => {+ if (!lane.target) return context.skip(lane.skip);+ if (trace === null) throw new Error('The exploration did not finish, so nothing was checked.');+ const violation = trace.violations.get(property);+ if (violation === undefined) return;+ throw new Error(+ `${property} was violated ${(violation.firstAtMicros / 1e6).toFixed(1)} s in, on ${violation.url}, ` ++ `and in ${violation.states} of ${trace.states} states: ${violation.detail}\n` ++ ` screenshot: ${violation.screenshot}`+ );+ });+ }++ it('the server logged no errors while it explored', (context) => {+ if (!lane.target) return context.skip(lane.skip);+ if (trace === null) throw new Error('The exploration did not finish, so nothing was checked.');+ expectNoServerErrors(serverErrors, names.tag, 'while the explorer ran');+ });++ it('the server logged no errors outside the exploration', async (context) => {+ if (!lane.target) return context.skip(lane.skip);+ expectNoServerErrors(await watch.sweep(), null, 'outside the exploration');+ });+});++/**+ * Something on the pages: an agent, in two rooms (an internal room needs one).+ * The explorer may delete them before teardown does, so one already gone is+ * not a leftover.+ */+async function seed(fixtures: Fixtures, client: SwitchClient, names: NameScope): Promise<void> {+ const agentName = names.name('agent');+ await client.registerKnownAgent({+ agentType: 'opencode',+ name: agentName,+ description: `Switch end-to-end harness agent (run ${names.runId}). Safe to delete.`,+ options: { auto_session: false },+ });+ fixtures.defer(`agent ${agentName}`, () => unlessGone(client.deleteAgentByName(agentName)));+ for (const role of ['room', 'other-room']) {+ const name = names.name(role);+ const room = await client.createRoom({+ name,+ description: `Switch end-to-end harness (run ${names.runId}). Safe to delete.`,+ bridgeId: null,+ agentNames: [agentName],+ userNames: [],+ });+ fixtures.defer(`room ${name} (${room.id})`, () => unlessGone(client.deleteRoom(room.id)));+ }+}++function logFile(): string {+ return path.join(outputDir, 'bombadil.log');+}++async function unlessGone(removal: Promise<void>): Promise<void> {+ try {+ await removal;+ } catch (error) {+ if (error instanceof SwitchHttpError && error.status === 404) return;+ throw error;+ }+}++/**+ * Run Bombadil against the gateway UI, and return its exit code.+ *+ * Its output goes to a log file rather than this process's: it prints a line+ * per action, thousands in a run. Every request the page makes carries the+ * scope's tag, so an error the server logs serving one is attributed here.+ */+async function explore(env: HarnessEnv, tag: string): Promise<number> {+ rmSync(outputDir, { recursive: true, force: true });+ mkdirSync(outputDir, { recursive: true });+ const specDir = mkdtempSync(path.join(tmpdir(), 'switch-explore-spec-'));+ try {+ copyFileSync(path.join(SPEC_DIR, 'gateway.ts'), path.join(specDir, 'gateway.ts'));+ writeFileSync(+ path.join(specDir, 'credentials.json'),+ JSON.stringify({+ email: env.switch.gatewayAdminEmail,+ password: env.switch.gatewayAdminPassword,+ }),+ { mode: 0o600 }+ );+ const args = [+ 'exec',+ 'bombadil',+ 'browser',+ 'test',+ '--headless',+ '--no-sandbox',+ '--time-limit',+ env.exploreTimeLimit,+ '--output-path',+ outputDir,+ '--width',+ '1440',+ '--height',+ '900',+ '--header',+ `X-Request-Id=${tag}-ui`,+ `${env.gatewayUiUrl}/login`,+ path.join(specDir, 'gateway.ts'),+ ];+ console.log(`[explore] exploring for ${env.exploreTimeLimit}; Bombadil's log: ${logFile()}`);+ const log = createWriteStream(logFile());+ const child = spawn('pnpm', args, {+ // Bombadil finds no browser of its own; Playwright's is installed for this lane.+ env: { ...process.env, CHROME: chromium.executablePath() },+ stdio: ['ignore', 'pipe', 'pipe'],+ });+ child.stdout.pipe(log);+ child.stderr.pipe(log);+ const code = await new Promise<number>((resolve, reject) => {+ child.on('error', reject);+ child.on('close', (exitCode, signal) =>+ exitCode === null ? reject(new Error(`Bombadil was killed by ${signal}.`)) : resolve(exitCode)+ );+ });+ await new Promise((resolve) => log.end(resolve));+ return code;+ } finally {+ rmSync(specDir, { recursive: true, force: true });+ }+}+diff --git a/console/tooling-e2e/src/explore/explore.test.ts b/console/tooling-e2e/src/explore/explore.test.tsnew file mode 100644index 000000000..fd77d534e--- /dev/null+++ b/console/tooling-e2e/src/explore/explore.test.ts@@ -0,0 +1,97 @@+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';+import { ACTION_GENERATORS, PROPERTIES } from './properties.ts';+import { describeViolation, summarizeTrace } from './trace.ts';++const SPEC = path.join(path.dirname(fileURLToPath(import.meta.url)), 'spec', 'gateway.ts');++/** Every name the spec exports: its own `export const`s and what it re-exports. */+function specExports(source: string): string[] {+ const own = [...source.matchAll(/^export const (\w+)/gm)].map((match) => match[1]!);+ const reexported = [...source.matchAll(/^export \{([^}]*)\} from/gm)].flatMap((match) =>+ match[1]!+ .split(',')+ .map((name) => name.trim())+ .filter((name) => name !== '')+ );+ return [...own, ...reexported];+}++const banner = (condition: string, value: unknown) => ({+ Always: {+ violation: {+ False: { time: 2, condition, snapshots: [{ index: 1, name: 'errorBanners', value, time: 2 }] },+ },+ subformula: { Thunk: { function: condition, negated: false } },+ start: 1,+ end: null,+ time: 2,+ },+});++const line = (timestamp: number, url: string, violations: unknown[]) =>+ JSON.stringify({+ timestamp,+ action: null,+ state: { url, screenshot: `/out/screenshots/${timestamp}.jpeg`, resources: {} },+ snapshots: [],+ violations,+ });++describe('explore lane', () => {+ it("lists exactly the spec's properties", () => {+ const exported = specExports(readFileSync(SPEC, 'utf8'));+ expect([...exported].sort()).toEqual([...PROPERTIES, ...ACTION_GENERATORS].sort());+ });++ it('reports each property by the first state it was violated in', () => {+ const condition = '() => errorBanners.current.length === 0';+ const trace = [+ line(1_000_000, 'http://ui/rooms', []),+ line(1_250_000, 'http://ui/agents', [{ name: 'noErrorBanner', violation: banner(condition, ['Boom']) }]),+ line(1_500_000, 'http://ui/rooms', [{ name: 'noErrorBanner', violation: banner(condition, ['Boom']) }]),+ '',+ ].join('\n');+ const summary = summarizeTrace(trace);+ expect(summary.states).toBe(3);+ expect([...summary.violations.values()]).toEqual([+ {+ name: 'noErrorBanner',+ states: 2,+ firstAtMicros: 250_000,+ url: 'http://ui/agents',+ screenshot: '/out/screenshots/1250000.jpeg',+ detail: `${condition}, with errorBanners = ["Boom"]`,+ },+ ]);+ });++ it('says which formula an eventually was still waiting for', () => {+ const violation = {+ Eventually: {+ subformula: { Thunk: { function: '() => !onLoginPage.current', negated: false } },+ reason: { TimedOut: 30_000_000 },+ },+ };+ expect(describeViolation(violation)).toBe(+ 'never held within its time limit: () => !onLoginPage.current'+ );+ });++ it('describes the consequent of an implication', () => {+ const violation = {+ Implies: {+ left: { Thunk: { function: '() => a', negated: false } },+ right: { False: { time: 1, condition: '() => b', snapshots: [] } },+ antecedent_snapshots: [],+ },+ };+ expect(describeViolation(violation)).toBe('() => b');+ });++ it('refuses a trace line that is not JSON', () => {+ expect(() => summarizeTrace('{"timestamp":')).toThrow(/trace line 1 is not JSON/);+ });+});diff --git a/console/tooling-e2e/src/explore/properties.ts b/console/tooling-e2e/src/explore/properties.tsnew file mode 100644index 000000000..f8f4040c4--- /dev/null+++ b/console/tooling-e2e/src/explore/properties.ts@@ -0,0 +1,18 @@+/**+ * The properties `spec/gateway.ts` exports, by export name, which is the name+ * Bombadil reports a violation under. The lane reports one case per property,+ * and vitest needs the cases before the run that finds the violations, so the+ * list is written out here; a unit test holds it to the spec's exports.+ */+export const PROPERTIES = [+ 'noConsoleErrors',+ 'noUncaughtExceptions',+ 'noUnhandledPromiseRejections',+ 'signsIn',+ 'staysSignedIn',+ 'noErrorBanner',+ 'noServerErrors',+] as const;++/** The spec's other exports: its action generators. */+export const ACTION_GENERATORS = ['signingIn', 'exploring'] as const;diff --git a/console/tooling-e2e/src/explore/spec/credentials.json b/console/tooling-e2e/src/explore/spec/credentials.jsonnew file mode 100644index 000000000..dd78002c7--- /dev/null+++ b/console/tooling-e2e/src/explore/spec/credentials.json@@ -0,0 +1 @@+{ "email": "", "password": "" }diff --git a/console/tooling-e2e/src/explore/spec/gateway.ts b/console/tooling-e2e/src/explore/spec/gateway.tsnew file mode 100644index 000000000..86f3c7df8--- /dev/null+++ b/console/tooling-e2e/src/explore/spec/gateway.ts@@ -0,0 +1,188 @@+/**+ * The explore lane's Bombadil specification: what every gateway page must keep+ * to, and what the explorer may do, while it drives the UI at random.+ *+ * This runs inside Bombadil, not Node: it may import only Bombadil's own+ * modules and JSON. Every property and action generator exported here is used,+ * so nothing else may be exported.+ *+ * The properties port the browser lane's rules for a page that renders but is+ * broken (`src/browser/gateway-ui.ts`), checked in every state the explorer+ * reaches instead of on a scripted path. A 4xx is not one of them: random input+ * is meant to be refused.+ */+import { always, eventually, now } from '@antithesishq/bombadil';+import { isAction } from '@antithesishq/bombadil/actions';+import {+ actions,+ extract,+ registerCustomAction,+ weighted,+ type ActionTemplate,+} from '@antithesishq/bombadil/browser';+import {+ clicks,+ inputs,+ navigation,+ scroll,+} from '@antithesishq/bombadil/browser/defaults/actions';+// The admin's credentials, written by the lane beside a copy of this file for+// the run; the copy in the repository is an empty placeholder.+import credentials from './credentials.json';++export {+ noConsoleErrors,+ noUncaughtExceptions,+ noUnhandledPromiseRejections,+} from '@antithesishq/bombadil/browser/defaults/properties';++/** The pages in the navigation rail, and the form for a new room. */+const ROUTES = [+ '/rooms',+ '/rooms/new',+ '/agents',+ '/resources',+ '/collaborations',+ '/registration-keys',+ '/workspace',+ '/usage',+ '/ecosystem',+];++/**+ * What would end the session the explorer is testing. Signing out and back in+ * proves nothing the browser lane does not, and a changed password locks the+ * lane out for the rest of the run.+ */+const SESSION_ENDING = ['Sign out', 'Change password'];++const path = extract((state) => state.window.location.pathname);++/**+ * Whether the login form is shown. Not the path: going back to `/login` while+ * signed in shows that path for a moment, until the app redirects to `/`.+ */+const onLoginPage = extract(+ (state) =>+ state.document.querySelector('form input[type="password"]') !== null &&+ state.document.querySelector('nav[aria-label="Main"]') === null+);++/** Whether the app's navigation rail is shown: only to someone signed in. */+const signedIn = extract(+ (state) => state.document.querySelector('nav[aria-label="Main"]') !== null+);++/**+ * Whether the last action opened a page: a link, back, forward, reload or a+ * jump. Read loosely because Bombadil hands the action over in its trace shape+ * (snake case, empty fields left out), not the one its types declare.+ */+const openedPage = extract((state) => {+ const action = state.lastAction as unknown;+ if (action === null || typeof action === 'string') return action !== 'Wait';+ const { Click, Custom } = action as {+ Click?: { fingerprint: { tag: string; href?: string } };+ Custom?: { name: string };+ };+ if (Click !== undefined) return Click.fingerprint.tag === 'a' && Click.fingerprint.href !== undefined;+ return Custom !== undefined && (Custom.name === 'goTo' || Custom.name === 'signIn');+});++const errorBanners = extract((state) =>+ Array.from(state.document.querySelectorAll('[role="alert"].MuiAlert-colorError'))+ .filter((element) => element.getClientRects().length > 0)+ .map((element) => (element.textContent ?? '').trim().slice(0, 200))+);++/**+ * Every `/gateway/` response since the page last loaded that the server+ * failed. Resource timing keeps 250 entries by default, fewer than a long+ * session on one page makes, so the buffer is raised before it is read.+ */+const serverErrors = extract((state) => {+ const performance = state.window.performance;+ performance.setResourceTimingBufferSize(10_000);+ return performance+ .getEntriesByType('resource')+ .filter(+ (entry): entry is PerformanceResourceTiming =>+ entry instanceof PerformanceResourceTiming &&+ new URL(entry.name).pathname.startsWith('/gateway/') &&+ entry.responseStatus >= 500+ )+ .map((entry) => `${entry.responseStatus} ${new URL(entry.name).pathname}`);+});++/** The explorer signs in from the login page it starts on. */+export const signsIn = eventually(() => signedIn.current).within(30, 'seconds');++/**+ * Once signed in, the explorer stays signed in. It cannot sign out (see+ * `SESSION_ENDING`), so the login page again means the session was lost.+ */+export const staysSignedIn = always(+ now(() => !onLoginPage.current).implies(always(() => !onLoginPage.current))+);++/**+ * No page opens with an error banner: MUI renders each as an `alert` with error+ * severity. A banner after a click is how the gateway refuses what random input+ * asked for (removing a member who owns agents, say), so only a banner that+ * comes with opening a page counts.+ */+export const noErrorBanner = always(+ now(() => openedPage.current).implies(() => errorBanners.current.length === 0)+);++/** The server answered no `/gateway` request with a 5xx. */+export const noServerErrors = always(() => serverErrors.current.length === 0);++/**+ * Sign in as the gateway admin, from the login page.+ *+ * Over `fetch` rather than the form, and with no arguments, because an action's+ * arguments are written to the trace the lane uploads: typed into the form, the+ * password would be too.+ */+const signIn = registerCustomAction('signIn', async (_document, window) => {+ if (credentials.email === '' || credentials.password === '') {+ throw new Error(+ 'credentials.json is the empty placeholder: run this spec through the explore lane.'+ );+ }+ const response = await window.fetch('/gateway/auth/login', {+ method: 'POST',+ headers: { 'Content-Type': 'application/json' },+ body: JSON.stringify(credentials),+ credentials: 'same-origin',+ });+ if (!response.ok) throw new Error(`Signing in answered ${response.status}.`);+ // Replacing the login page in history, so going back does not restore it.+ window.location.replace('/rooms');+});++const goTo = registerCustomAction('goTo', async (_document, window, route: string) => {+ window.location.assign(route);+});++export const signingIn = actions(() => [signIn()]).when(() => onLoginPage.current);++function endsSession(action: ActionTemplate): boolean {+ if (!isAction(action, 'Click')) return false;+ const { textContent, accessibleName } = action.Click.fingerprint;+ return SESSION_ENDING.some((label) => textContent === label || accessibleName === label);+}++/**+ * Bombadil's default exploration, without what ends the session, plus a jump to+ * any page in the rail so a dead end (a dialog with nothing to click, a page+ * scrolled away from its links) does not hold the explorer.+ */+export const exploring = weighted([+ [100, clicks.filter((action) => !endsSession(action))],+ [100, inputs],+ [50, scroll],+ [10, navigation],+ [5, actions(() => ROUTES.filter((route) => route !== path.current).map((route) => goTo(route)))],+]).when(() => !onLoginPage.current);diff --git a/console/tooling-e2e/src/explore/spec/tsconfig.json b/console/tooling-e2e/src/explore/spec/tsconfig.jsonnew file mode 100644index 000000000..25fda6aaa--- /dev/null+++ b/console/tooling-e2e/src/explore/spec/tsconfig.json@@ -0,0 +1,16 @@+{+ "compilerOptions": {+ "target": "ES2023",+ "module": "ESNext",+ "moduleResolution": "bundler",+ "lib": ["ES2023", "DOM", "DOM.Iterable"],+ "types": [],+ "strict": true,+ "noUncheckedIndexedAccess": true,+ "noEmit": true,+ "skipLibCheck": true,+ "resolveJsonModule": true,+ "verbatimModuleSyntax": true+ },+ "include": ["*.ts"]+}diff --git a/console/tooling-e2e/src/explore/trace.ts b/console/tooling-e2e/src/explore/trace.tsnew file mode 100644index 000000000..dfd286027--- /dev/null+++ b/console/tooling-e2e/src/explore/trace.ts@@ -0,0 +1,129 @@+import { readFileSync } from 'node:fs';++/**+ * What Bombadil's `trace.jsonl` says about the properties: one line per state+ * it captured, each listing the properties violated in that state. A property+ * stays violated once it is, so a violation is reported by its first state.+ */++/** The parts of a trace line the lane reads. */+interface TraceLine {+ timestamp: number;+ state: { url: string; screenshot: string };+ violations: { name: string; violation: unknown }[];+}++export interface PropertyViolation {+ name: string;+ /** How many states it was reported in. */+ states: number;+ /** Microseconds since the first state captured. */+ firstAtMicros: number;+ url: string;+ screenshot: string;+ /** The condition that failed, and the extracted values it failed on. */+ detail: string;+}++export interface TraceSummary {+ states: number;+ violations: Map<string, PropertyViolation>;+}++export function readTrace(file: string): TraceSummary {+ return summarizeTrace(readFileSync(file, 'utf8'));+}++export function summarizeTrace(raw: string): TraceSummary {+ const violations = new Map<string, PropertyViolation>();+ let states = 0;+ let start: number | null = null;+ for (const [index, text] of raw.split('\n').entries()) {+ if (text.trim() === '') continue;+ let line: TraceLine;+ try {+ line = JSON.parse(text) as TraceLine;+ } catch (error) {+ throw new Error(+ `trace line ${index + 1} is not JSON: ${error instanceof Error ? error.message : String(error)}`+ );+ }+ states += 1;+ start ??= line.timestamp;+ for (const { name, violation } of line.violations) {+ const seen = violations.get(name);+ if (seen !== undefined) {+ seen.states += 1;+ continue;+ }+ violations.set(name, {+ name,+ states: 1,+ firstAtMicros: line.timestamp - start,+ url: line.state.url,+ screenshot: line.state.screenshot,+ detail: describeViolation(violation),+ });+ }+ }+ return { states, violations };+}++/** A violation as Bombadil serializes it: one variant per formula that can fail. */+type Violation =+ | { False: { condition: string; snapshots: { name: string | null; value: unknown }[] } }+ | { Always: { violation: Violation } }+ | { Eventually: { subformula: unknown; reason: 'TestEnded' | { TimedOut: number } } }+ | { Implies: { right: Violation } }+ | { And: { left: Violation; right: Violation } }+ | { Or: { left: Violation; right: Violation } };++/**+ * The condition that was false, with the values it was false on, or the+ * formula an `eventually` was still waiting for. Bombadil wraps a violation+ * once per operator around it, and both sides of an `and` or `or`.+ */+export function describeViolation(raw: unknown): string {+ const violation = raw as Violation;+ if ('False' in violation) {+ const { condition, snapshots } = violation.False;+ const values = snapshots+ .map((snapshot) => `${snapshot.name ?? '?'} = ${JSON.stringify(snapshot.value)}`)+ .join('; ');+ return values === '' ? condition : `${condition}, with ${values}`;+ }+ if ('Always' in violation) return describeViolation(violation.Always.violation);+ if ('Implies' in violation) return describeViolation(violation.Implies.right);+ if ('And' in violation) {+ return `${describeViolation(violation.And.left)} and ${describeViolation(violation.And.right)}`;+ }+ if ('Or' in violation) {+ return `${describeViolation(violation.Or.left)} and ${describeViolation(violation.Or.right)}`;+ }+ if ('Eventually' in violation) {+ const { subformula, reason } = violation.Eventually;+ const why = reason === 'TestEnded' ? 'by the end of the run' : 'within its time limit';+ return `never held ${why}: ${formulaText(subformula)}`;+ }+ throw new Error(`Unknown violation in the trace: ${JSON.stringify(raw).slice(0, 300)}`);+}++/** A formula as Bombadil serializes it, written back the way a specification writes it. */+function formulaText(raw: unknown): string {+ const formula = raw as Record<string, unknown>;+ if ('Thunk' in formula) return (formula.Thunk as { function: string }).function;+ if ('Pure' in formula) return (formula.Pure as { pretty: string }).pretty;+ for (const operator of ['And', 'Or', 'Implies'] as const) {+ if (operator in formula) {+ const [left, right] = formula[operator] as [unknown, unknown];+ return `(${formulaText(left)}) ${operator.toLowerCase()} (${formulaText(right)})`;+ }+ }+ for (const operator of ['Next', 'Always', 'Eventually'] as const) {+ if (operator in formula) {+ const inner = formula[operator];+ return `${operator.toLowerCase()}(${formulaText(Array.isArray(inner) ? inner[0] : inner)})`;+ }+ }+ throw new Error(`Unknown formula in the trace: ${JSON.stringify(raw).slice(0, 300)}`);+}diff --git a/console/tooling-e2e/tsconfig.json b/console/tooling-e2e/tsconfig.jsonindex 641ae596e..a1866a376 100644--- a/console/tooling-e2e/tsconfig.json+++ b/console/tooling-e2e/tsconfig.json@@ -22,5 +22,8 @@"include": ["src/**/*.ts","vitest.config.ts"+ ],+ "exclude": [+ "src/explore/spec"]}diff --git a/justfile b/justfileindex 85963459e..cba829484 100644--- a/justfile+++ b/justfile@@ -190,6 +190,14 @@ e2e-provider *args:e2e-browser *args:SWITCH_E2E_LANE=browser console/tooling-e2e/scripts/ephemeral-stack.sh {{ args }}+# ── End-to-end harness, explore lane, on a throwaway stack ─────────────────────+# As e2e-browser, but Bombadil drives the gateway UI at random for+# SWITCH_E2E_EXPLORE_TIME_LIMIT (default 5m), checking the properties in+# console/tooling-e2e/src/explore/spec/gateway.ts in every state. Its trace:+# build/e2e/ephemeral/explore, read with `bombadil browser inspect`.+e2e-explore *args:+ SWITCH_E2E_LANE=explore console/tooling-e2e/scripts/ephemeral-stack.sh {{ args }}+# ── End-to-end harness, agentic lane, on a throwaway stack ─────────────────────# As e2e-provider, then the scenario files in console/tooling-e2e/agentic: a# driver model plays the human, a judge model grades the conversation. Needs