Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript
8.3 KB
Raw Blame

Control UI — headful journey tests

Status snapshot of the optional Playwright QA recorder journeys for the Clerum Control UI (Next.js operator/admin web app). These drive a real headful Chromium window (recorded WebM + PNG) against a local control-ui dev server pointed at a example-dev control-api port-forward. They are separate from the CI Playwright suite and never run in CI by default.

Recorder runs are local and opt-in. Configure the exact QA identity in .env.qa-recorder; target only an approved loopback or disposable development environment; and set QA_RECORDER_CONFIRM_MUTATIONS=1 only for journeys whose side effects you intend to run. Recordings stay under the git-ignored .local-notes/qa-recorder/ directory and may contain sensitive UI data.

Coverage

Every common Control UI operator journey has at least one headful spec. Each signs in with the exact admin identity from .env.qa-recorder, shares launch/login/proof plumbing in e2e/qa-recorder-helpers.ts, and records a WebM + PNG per test under .local-notes/qa-recorder/runs/control-ui/ (git-ignored).

Inventory & navigation (read-only, plus the create-agent wizard)

Journey Spec Tests Confirm flag Status
Login & shell qa-recorder-login.spec.ts 2 — sign-in page, authenticated shell — ✅
Navigation qa-recorder-navigation.spec.ts 1 — sidebar across main sections — ✅
Agents (hosts) qa-recorder-agents.spec.ts 1 — list + detail — ✅
Agent-centric vocabulary sweep qa-recorder-context-vocab-sweep.spec.ts 1 — current operator surfaces — ✳️
Connectors qa-recorder-connectors.spec.ts 1 — inventory — ✅
External channels qa-recorder-external-channels.spec.ts 1 — Telegram/Email/Slack — ✅
LLM models & secrets qa-recorder-llm-models.spec.ts 1 — models catalog + secrets — ✅
Users & teams qa-recorder-users-teams.spec.ts 1 — directory — ✅
Cost & usage qa-recorder-cost-usage.spec.ts 1 — usage dashboard — ✅
Traces qa-recorder-traces.spec.ts 2 — dashboard, state — ✅
Settings qa-recorder-settings.spec.ts 1 — read-only settings — ✅
Marketplace & plugins qa-recorder-marketplace.spec.ts 2 — connectors, plugins — ✅
Marketplace browse qa-recorder-marketplace-browse.spec.ts 1 — search, filter, plugins tab, entry detail (skips if empty) — ✳️
Create agent qa-recorder-agent-create.spec.ts 1 — full create wizard QA_RECORDER_CONFIRM_MUTATIONS (required) ✅

The two recorder filenames that still contain context cover migration-only compatibility behavior: legacy-link redirects and the assertion that current operator surfaces use Agent-centric vocabulary. They do not restore Context management journeys.

Workflow & combo journeys (create / edit / delete / grant)

Full mutating user journeys. Each is self-contained — it creates its own resources with uniqueE2EName(...) and tears them down via the Control API in a finally, so none depend on pre-seeded data or real external credentials (dummy qa-recorder-* values are used where a form wants a value but no real credential). Run with npm run qa:recorder:<journey>.

Journey Spec Tests Confirm flag Status
Legacy scope links → owning Agent qa-recorder-context-redirects.spec.ts 1 CONFIRM_MUTATIONS ✳️
Agent connector inventory + detail qa-recorder-agent-connectors.spec.ts 1 CONFIRM_MUTATIONS ✳️
Connector create + Agent access qa-recorder-connector-create-access.spec.ts 1 CONFIRM_MUTATIONS ✳️
Installed Connectors Agent access qa-recorder-connectors-agent-access.spec.ts 1 CONFIRM_MUTATIONS ✳️
Connector edit egress qa-recorder-connector-edit.spec.ts 1 CONFIRM_MUTATIONS ✳️
LLM model add → disable → remove qa-recorder-llm-model-lifecycle.spec.ts 1 CONFIRM_MUTATIONS ✳️
Recipe secret (shared, dummy) qa-recorder-secret-recipe.spec.ts 1 CONFIRM_MUTATIONS ✳️
Connector secret (dummy) qa-recorder-secret-connector.spec.ts 1 CONFIRM_MUTATIONS ✳️
SharedFileSystem provision + upload + delete qa-recorder-shared-fs-upload.spec.ts 1 CONFIRM_MUTATIONS ✳️
SharedFileSystem folder create + rename qa-recorder-shared-fs-folders.spec.ts 1 CONFIRM_MUTATIONS ✳️
Global file system folder + upload + rename qa-recorder-global-fs.spec.ts 1 CONFIRM_MUTATIONS ✳️
Team create + detail tour qa-recorder-team-create.spec.ts 1 CONFIRM_MUTATIONS ✳️
Create member + cancel invite (sends email) qa-recorder-member-invite.spec.ts 1 CONFIRM_MUTATIONS ✳️
Admin invite (no Desktop) + cancel (sends email) qa-recorder-admin-invite.spec.ts 1 CONFIRM_MUTATIONS ✳️
Marketplace API key create + reveal + revoke qa-recorder-marketplace-api-keys.spec.ts 1 CONFIRM_MUTATIONS ✳️
Token budget (global) create + edit qa-recorder-token-budget.spec.ts 1 CONFIRM_MUTATIONS ✳️
Token budget scoped to a team qa-recorder-token-budget-team.spec.ts 1 CONFIRM_MUTATIONS ✳️
LLM price add → edit → delete qa-recorder-llm-price.spec.ts 1 CONFIRM_MUTATIONS ✳️
Create-form availability smoke (15 forms, no submit) qa-recorder-create-form-smoke.spec.ts 15 — ✳️
Settings — username + theme + reset alerts qa-recorder-settings-account.spec.ts 1 CONFIRM_MUTATIONS ✳️
Settings — change password + revert qa-recorder-settings-password.spec.ts 1 CONFIRM_MUTATIONS ✳️
Settings — email change + resend + revert (sends email) qa-recorder-settings-email.spec.ts 1 CONFIRM_MUTATIONS ✳️

Persona combos (multi-resource, end-to-end)

Journey Spec Tests Confirm flag Status
New-team onboarding (team → member → Agent grant) qa-recorder-onboarding-combo.spec.ts 1 CONFIRM_MUTATIONS ✳️
Cost governance (model → price → model-scoped budget) qa-recorder-cost-governance.spec.ts 1 CONFIRM_MUTATIONS ✳️
Agent-scoped token budget qa-recorder-token-budget-scope.spec.ts 1 CONFIRM_MUTATIONS ✳️

Totals: 50 specs, 67 tests. Read-only journeys need no confirmation flag; every journey still calls the loopback reachability guard on both URLs. Every mutating journey requires QA_RECORDER_CONFIRM_MUTATIONS=1, creates only temporary qa-recorder-* resources, and cleans them up via the Control API in a finally (reverse order, each deletion best-effort). Invite and email/password journeys have external side effects (an email is sent / the admin password is changed) and revert to the original state on completion.

Legend: ✅ verified against a live example-dev environment; ✳️ implemented and compile-verified (npx playwright test --config=playwright.qa-recorder.config.ts --list) but not yet exercised against a live environment — re-run to refresh.

Run

cd control-ui
npm run qa:recorder:install   # one-time: recorder Chromium into .local-notes/qa-recorder/browsers
# control-ui dev server on :3000 + control-api forward on :8090 must be up
npm run qa:recorder:login     # one journey
npm run qa:recorder:all       # all 50 specs / 67 tests

Proof artifacts (local only, git-ignored)

find .local-notes/qa-recorder/runs/control-ui -name '*.webm'   # WebM recordings
find .local-notes/qa-recorder/runs/control-ui -name '*.png'    # full-page screenshots

Last verified

  • Inventory & create-agent (✅): 2026-07-17 against example-dev (GKE) — the original 13 specs / 17 tests passed, exit 0, with WebM + PNG proof for every test. Point-in-time snapshot; re-run to refresh.
  • Workflow, combo & smoke journeys (✳️): implemented and compile-verified via npx playwright test --config=playwright.qa-recorder.config.ts --list (67/67 tests discovered, zero compile errors). Not yet exercised against a live environment — run npm run qa:recorder:all (with QA_RECORDER_CONFIRM_MUTATIONS=1) to produce WebM + PNG proof and flip these to ✅.