Fully autonomous AI Agents system capable of performing complex penetration testing tasks
Go ai-agentsai-security-toolanthropicautonomous-agentsgolanggptgraphqlmulti-agent-systemoffensive-securityopen-sourceopenaipenetration-testingpenetration-testing-toolsreactsecurity-automationsecurity-testingsecurity-toolsself-hosted
feat(e2e): tier 2 — real backend run driven by an OpenAI-compatible mock LLM
Sergey Kozyrenko1a4f6032160b
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.ymlindex 11d71362..6ac88aee 100644--- a/.github/workflows/e2e.yml+++ b/.github/workflows/e2e.yml@@ -72,3 +72,45 @@ jobs:frontend/e2e/test-resultsretention-days: 7if-no-files-found: ignore++ # Tier 2: branch-built image + isolated compose stack + the mock-LLM agent+ # loop. Not part of the PR gate — nightly and on-demand only.+ e2e-local:+ if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'+ runs-on: ubuntu-latest+ timeout-minutes: 40+ steps:+ - name: Checkout+ uses: actions/checkout@v6++ - name: Install pnpm+ uses: pnpm/action-setup@v4+ with:+ package_json_file: frontend/package.json++ - name: Set up Node.js+ uses: actions/setup-node@v6+ with:+ node-version-file: frontend/.nvmrc+ cache: "pnpm"+ cache-dependency-path: "frontend/pnpm-lock.yaml"++ - name: Install dependencies+ working-directory: frontend+ run: pnpm install --frozen-lockfile++ - name: Install Playwright browsers+ working-directory: frontend+ run: pnpm exec playwright install --with-deps chromium++ - name: Run Tier 2 (branch image + mock LLM)+ run: ./frontend/e2e/tools/run-local-tier.sh++ - name: Upload report and traces+ if: always()+ uses: actions/upload-artifact@v4+ with:+ name: e2e-local-report+ path: frontend/e2e/test-results+ retention-days: 7+ if-no-files-found: ignorediff --git a/docker-compose.e2e.yml b/docker-compose.e2e.ymlnew file mode 100644index 00000000..3778431b--- /dev/null+++ b/docker-compose.e2e.yml@@ -0,0 +1,58 @@+# Tier-2 e2e stack. Runs beside a developer stack: use its own project name,+# container names, network, and host ports —+#+# docker build -t local/pentagi:e2e .+# PENTAGI_IMAGE=local/pentagi:e2e PENTAGI_LISTEN_PORT=8444 PGVECTOR_LISTEN_PORT=5433 \+# docker compose -p pentagi-e2e --env-file /dev/null \+# -f docker-compose.yml -f docker-compose.e2e.yml \+# up -d --wait --wait-timeout 240 pentagi mock-llm+#+# --env-file /dev/null is load-bearing: without it compose auto-loads the+# developer's .env and the stack inherits live provider keys and DOCKER_HOST.+# (frontend/e2e/tools/run-local-tier.sh wraps all of this.)++networks:+ pentagi-network:+ name: pentagi-e2e-network++services:+ mock-llm:+ image: node:24-alpine+ container_name: pentagi-e2e-mock-llm+ restart: unless-stopped+ command: ["node", "/app/server.mjs"]+ volumes:+ - ./frontend/e2e/mock-llm:/app:ro+ networks:+ - pentagi-network++ pentagi:+ container_name: pentagi-e2e+ depends_on:+ pgvector:+ condition: service_healthy+ mock-llm:+ condition: service_started+ environment:+ - LLM_SERVER_URL=http://mock-llm:8080/v1+ # The langchaingo client refuses to construct without a token even though+ # the mock never checks it.+ - LLM_SERVER_KEY=e2e-dummy-key+ - LLM_SERVER_MODEL=e2e-mock+ - EMBEDDING_PROVIDER=none+ # The base service has no healthcheck, so `up --wait` would return before+ # goose migrations finish and the API serves. The runtime image is alpine —+ # probe with busybox wget, not curl.+ healthcheck:+ test:+ [+ "CMD-SHELL",+ "wget --no-check-certificate -qO /dev/null https://127.0.0.1:8443/api/v1/info",+ ]+ interval: 3s+ timeout: 5s+ retries: 60+ start_period: 10s++ pgvector:+ container_name: pentagi-e2e-pgvectordiff --git a/frontend/.gitignore b/frontend/.gitignoreindex 2e4d9116..1df32974 100644--- a/frontend/.gitignore+++ b/frontend/.gitignore@@ -28,3 +28,4 @@ e2e/test-results/e2e/playwright-report/e2e/blob-report/playwright/.auth/+e2e/.auth/diff --git a/frontend/docs/e2e.md b/frontend/docs/e2e.mdindex 1e861cea..e454c6db 100644--- a/frontend/docs/e2e.md+++ b/frontend/docs/e2e.md@@ -26,7 +26,8 @@ pnpm e2e:ui # same, in Playwright UI mode (watch/debug)CI=1 pnpm e2e # byte-identical reproduction of a CI run# (same retries and trace policy; CI-style reporters)-E2E_TIER=local pnpm e2e # against the local docker stack (https://localhost:8443)+./e2e/tools/run-local-tier.sh # Tier 2: branch image + isolated docker+ # stack + mock LLM; runs specs/real/**E2E_TIER=stand E2E_BASE_URL=https://… pnpm e2e # against a live stand```@@ -37,10 +38,17 @@ running (`reuseExistingServer`), so only the first run pays the build.| Tier | Backend | Needs | Used for ||---|---|---|---|-| `mock` (default) | Playwright route/WS mocks replaying cassettes against the **production bundle** | nothing | the PR gate; every functional spec |-| `local` | `docker compose` stack | docker | fidelity checks, real backend paths |+| `mock` (default) | Playwright route/WS mocks replaying cassettes against the **production bundle** | nothing | the PR gate; every cassette spec |+| `local` | branch-built image in an isolated compose stack (`pentagi-e2e`, ports 8444/5433 — coexists with a dev stack) + an OpenAI-compatible **mock LLM** driving the real agent loop | docker | `specs/real/**`: the fidelity check — real GraphQL/WS/pub-sub end to end || `stand` | a live stand | `E2E_BASE_URL` + creds | deploy smoke, version-skew checks |+Tier-2 notes: the runner isolates the stack from your `.env` (`--env-file+/dev/null`), seeds flow ids from 90001 so sandbox containers+(`pentagi-terminal-<id>`) never collide with a developer stack on the same+docker daemon, and removes those sandboxes on exit. The deterministic agent+transcript lives in `e2e/mock-llm/scenario.mjs`. Iterating: `E2E_SKIP_BUILD=1`+reuses the image, `E2E_KEEP_STACK=1` leaves the stack up.+## Debugging a red CI run1. Open the failed run (link in the PR comment) and download the `e2e-report`diff --git a/frontend/e2e/fixtures/auth.ts b/frontend/e2e/fixtures/auth.tsindex 9d7f17e0..01d8117c 100644--- a/frontend/e2e/fixtures/auth.ts+++ b/frontend/e2e/fixtures/auth.ts@@ -1,7 +1,5 @@import type { Page } from '@playwright/test';-import { test as base } from '@playwright/test';-import type { AuthInfo } from '@/models/info';import type { User } from '@/models/user';@@ -48,17 +46,3 @@ export const seedAuthenticated = async (page: Page): Promise<void> => {[AUTH_STORAGE_KEY, JSON.stringify(seededAuthInfo())] as const,);};--export const test = base.extend<{ _seedAuth: void; isAuthSeeded: boolean }>({- _seedAuth: [- async ({ isAuthSeeded, page }, use) => {- if (isAuthSeeded) {- await seedAuthenticated(page);- }-- await use();- },- { auto: true },- ],- isAuthSeeded: [true, { option: true }],-});diff --git a/frontend/e2e/fixtures/backend.ts b/frontend/e2e/fixtures/backend.tsindex c1f9abf2..a83a7d52 100644--- a/frontend/e2e/fixtures/backend.ts+++ b/frontend/e2e/fixtures/backend.ts@@ -4,6 +4,7 @@ import type { Cassette } from '../mocks/cassette.ts';import { installMockRoutes } from '../mocks/install.ts';import { MockWorld } from '../mocks/world.ts';+import { seedAuthenticated } from './auth.ts';export interface BackendConfig {installMocks: boolean;@@ -27,9 +28,13 @@ interface BackendFixtures {world: MockWorld;}-export const test = base.extend<BackendFixtures & BackendOptions & { _installMocks: void }>({+export const test = base.extend<BackendFixtures & BackendOptions & { _installMocks: void; isAuthSeeded: boolean }>({+ // Seeding client-side auth is a mock-tier shortcut only: against a real+ // backend a forged localStorage session has no cookie behind it, and the+ // first 401 wipes it and hard-redirects to login. Real tiers authenticate+ // once in the setup project and reuse storageState._installMocks: [- async ({ backend, page, world }, use) => {+ async ({ backend, isAuthSeeded, page, world }, use) => {if (!backend.installMocks) {await use();@@ -37,6 +42,11 @@ export const test = base.extend<BackendFixtures & BackendOptions & { _installMoc}await page.clock.install({ time: CASSETTE_EPOCH });++ if (isAuthSeeded) {+ await seedAuthenticated(page);+ }+await installMockRoutes(page, world);await use();expect(world.unmatched, 'every API call the app made must have a cassette entry').toEqual([]);@@ -45,6 +55,7 @@ export const test = base.extend<BackendFixtures & BackendOptions & { _installMoc],backend: [{ installMocks: true, tier: 'mock' }, { option: true }],cassette: [{}, { option: true }],+ isAuthSeeded: [true, { option: true }],world: async ({ cassette }, use) => {await use(new MockWorld(cassette));},diff --git a/frontend/e2e/fixtures/test.ts b/frontend/e2e/fixtures/test.tsindex bd754072..69ec656b 100644--- a/frontend/e2e/fixtures/test.ts+++ b/frontend/e2e/fixtures/test.ts@@ -3,7 +3,6 @@ import { test as base, mergeTests } from '@playwright/test';import type { PageErrorLog } from '../helpers/errors.ts';import { watchPageErrors } from '../helpers/errors.ts';-import { test as authTest } from './auth.ts';import { test as backendTest } from './backend.ts';const errorsTest = base.extend<{ pageErrorLog: PageErrorLog }>({@@ -12,6 +11,6 @@ const errorsTest = base.extend<{ pageErrorLog: PageErrorLog }>({},});-export const test = mergeTests(backendTest, authTest, errorsTest);+export const test = mergeTests(backendTest, errorsTest);export { expect } from '@playwright/test';diff --git a/frontend/e2e/mock-llm/scenario.mjs b/frontend/e2e/mock-llm/scenario.mjsnew file mode 100644index 00000000..7503cb58--- /dev/null+++ b/frontend/e2e/mock-llm/scenario.mjs@@ -0,0 +1,90 @@+// Deterministic transcript for one "say hello" flow. Rules are matched (first+// hit wins) against the JSON of the whole request — tool definitions live in+// `tools`, not `messages`, so markers may come from either.+export const RULES = [+ {+ content: 'debian:stable-slim',+ label: 'image-chooser',+ match: /Docker Image Selector/,+ },+ {+ content: 'English',+ label: 'language-chooser',+ match: /Language Detector/,+ },+ {+ content: 'Say Hello Flow',+ label: 'flow-title',+ match: /Flow Title Generator/,+ },+ {+ content: 'Say hello to the user',+ label: 'task-title',+ match: /Task Title Generator/,+ },+ // The pattern detector must FAIL cleanly (plain text, no tool call): the+ // backend then derives the tool-call-ID template from the sampled ids —+ // this rule must sit above get_number since detector prompts quote samples.+ {+ content: 'call_000000',+ label: 'toolcall-pattern-detector',+ match: /submit_pattern/,+ },+ {+ label: 'toolcall-id-sample',+ match: /get_number/,+ toolCalls: [{ args: { value: 1 }, name: 'get_number' }],+ },+ // The task reporter summarizes the finished task and only accepts the+ // report_result tool.+ {+ label: 'task-report',+ match: /report_result|TASK EXECUTION EVALUATOR AND REPORTER/,+ toolCalls: [+ {+ args: {+ message: 'Task finished successfully',+ result: 'Hello from the e2e mock LLM!',+ success: true,+ },+ name: 'report_result',+ },+ ],+ },+ // The refiner re-plans between subtasks and only accepts subtask_patch;+ // an empty operations list means "plan unchanged".+ {+ label: 'subtask-refine',+ match: /subtask_patch/,+ toolCalls: [{ args: { operations: [] }, name: 'subtask_patch' }],+ },+ {+ label: 'subtask-plan',+ match: /subtask_list/,+ toolCalls: [+ {+ args: {+ message: 'Plan ready',+ subtasks: [{ description: 'Say hello to the user', title: 'Greet' }],+ },+ name: 'subtask_list',+ },+ ],+ },+];++// Any agent turn not matched above settles its subtask via the `done` barrier+// tool — plain text answers would loop through the reflector instead.+export const FALLBACK = {+ label: 'agent-done',+ toolCalls: [+ {+ args: {+ message: 'Greeted the user',+ result: 'Hello from the e2e mock LLM!',+ success: true,+ },+ name: 'done',+ },+ ],+};diff --git a/frontend/e2e/mock-llm/server.mjs b/frontend/e2e/mock-llm/server.mjsnew file mode 100644index 00000000..64f7e3a2--- /dev/null+++ b/frontend/e2e/mock-llm/server.mjs@@ -0,0 +1,134 @@+// OpenAI-compatible mock LLM for the Tier-2 e2e stack. The backend talks to it+// through the `custom` provider (LLM_SERVER_URL), so the real langchaingo+// client, streaming, and tool-call plumbing are exercised end-to-end while+// answers stay deterministic: the whole request JSON is matched against+// scenario.mjs rules in order, first hit wins.+import { createServer } from 'node:http';++import { FALLBACK, RULES } from './scenario.mjs';++const PORT = 8080;++const readBody = (request) =>+ new Promise((resolve) => {+ let body = '';++ request.on('data', (chunk) => {+ body += chunk;+ });+ request.on('end', () => resolve(body));+ });++const pickAnswer = (payload) => {+ const text = JSON.stringify(payload);++ for (const rule of RULES) {+ if (rule.match.test(text)) {+ return rule;+ }+ }++ return FALLBACK;+};++let completionCounter = 0;++const completionMessage = (rule) => {+ const message = { content: rule.content ?? null, role: 'assistant' };++ if (rule.toolCalls) {+ // Fixed-width ids: the backend samples these to derive its tool-call-ID+ // template, so the shape must stay consistent across calls.+ message.tool_calls = rule.toolCalls.map((call, index) => ({+ function: { arguments: JSON.stringify(call.args ?? {}), name: call.name },+ id: `call_${String(completionCounter * 10 + index).padStart(6, '0')}`,+ type: 'function',+ }));+ }++ return message;+};++const respondJson = (response, status, payload) => {+ response.writeHead(status, { 'content-type': 'application/json' });+ response.end(JSON.stringify(payload));+};++const respondCompletion = (response, payload, rule) => {+ completionCounter += 1;+ const id = `chatcmpl-e2e-${completionCounter}`;+ const message = completionMessage(rule);+ const finishReason = rule.toolCalls ? 'tool_calls' : 'stop';+ const usage = { completion_tokens: 7, prompt_tokens: 11, total_tokens: 18 };++ if (!payload.stream) {+ respondJson(response, 200, {+ choices: [{ finish_reason: finishReason, index: 0, message }],+ created: 1767225600,+ id,+ model: payload.model ?? 'e2e-mock',+ object: 'chat.completion',+ usage,+ });++ return;+ }++ response.writeHead(200, {+ 'cache-control': 'no-cache',+ connection: 'keep-alive',+ 'content-type': 'text/event-stream',+ });++ const chunk = (delta, finish = null) =>+ response.write(+ `data: ${JSON.stringify({+ choices: [{ delta, finish_reason: finish, index: 0 }],+ created: 1767225600,+ id,+ model: payload.model ?? 'e2e-mock',+ object: 'chat.completion.chunk',+ })}\n\n`,+ );++ chunk({ role: 'assistant' });++ if (message.tool_calls) {+ chunk({+ tool_calls: message.tool_calls.map((call, index) => ({ index, ...call })),+ });+ } else {+ for (const piece of String(message.content ?? '').match(/.{1,24}/gs) ?? []) {+ chunk({ content: piece });+ }+ }++ chunk({}, finishReason);+ response.write(`data: [DONE]\n\n`);+ response.end();+};++createServer(async (request, response) => {+ const { url = '' } = request;++ if (request.method === 'POST' && url.endsWith('/chat/completions')) {+ const payload = JSON.parse((await readBody(request)) || '{}');+ const rule = pickAnswer(payload);++ console.log(`[mock-llm] ${rule.label}: ${payload.stream ? 'stream' : 'plain'}`);+ respondCompletion(response, payload, rule);++ return;+ }++ if (request.method === 'GET' && url.endsWith('/models')) {+ respondJson(response, 200, {+ data: [{ id: 'e2e-mock', object: 'model', owned_by: 'e2e' }],+ object: 'list',+ });++ return;+ }++ respondJson(response, 404, { error: `mock-llm: unhandled ${request.method} ${url}` });+}).listen(PORT, () => console.log(`[mock-llm] listening on :${PORT}`));diff --git a/frontend/e2e/playwright.config.ts b/frontend/e2e/playwright.config.tsindex 7676f675..5a4f6641 100644--- a/frontend/e2e/playwright.config.ts+++ b/frontend/e2e/playwright.config.ts@@ -15,6 +15,8 @@ const PREVIEW_PORT = 8100;// so CI artifact uploads have one deterministic location.const here = (relative: string) => fileURLToPath(new URL(relative, import.meta.url));+export const AUTH_STATE_PATH = here('./.auth/user.json');+const TIERS: Record<BackendTier, { baseURL: string; installMocks: boolean }> = {local: { baseURL: process.env.E2E_BASE_URL ?? 'https://localhost:8443', installMocks: false },mock: { baseURL: `http://localhost:${PREVIEW_PORT}`, installMocks: true },@@ -30,12 +32,30 @@ export default defineConfig<BackendOptions>({fullyParallel: true,globalTimeout: isCI ? 10 * 60_000 : undefined,outputDir: './test-results',- projects: [- {- name: `${tier}-chromium`,- use: { ...devices['Desktop Chrome'] },- },- ],+ // Cassette specs run only on the mock tier; specs/real/** run only against a+ // live backend, which authenticates once in the setup project and reuses+ // storageState.+ projects:+ tier === 'mock'+ ? [+ {+ name: 'mock-chromium',+ testIgnore: '**/specs/real/**',+ use: { ...devices['Desktop Chrome'] },+ },+ ]+ : [+ {+ name: 'setup',+ testMatch: '**/auth.setup.ts',+ },+ {+ dependencies: ['setup'],+ name: `${tier}-chromium`,+ testMatch: '**/specs/real/**',+ use: { ...devices['Desktop Chrome'], storageState: AUTH_STATE_PATH },+ },+ ],reporter: isCI? [['blob', { outputDir: here('./blob-report') }],diff --git a/frontend/e2e/specs/auth.setup.ts b/frontend/e2e/specs/auth.setup.tsnew file mode 100644index 00000000..ae3a5142--- /dev/null+++ b/frontend/e2e/specs/auth.setup.ts@@ -0,0 +1,27 @@+import { expect, test as setup } from '@playwright/test';++import { AUTH_STATE_PATH } from '../playwright.config.ts';++const E2E_USER = process.env.E2E_USER ?? '[email protected]';+const E2E_PASSWORD = process.env.E2E_PASSWORD ?? 'admin';++setup('authenticate', async ({ page }) => {+ await page.goto('/login');+ await page.getByLabel('Login').fill(E2E_USER);+ await page.getByRole('textbox', { name: 'Password' }).fill(E2E_PASSWORD);+ await page.getByRole('button', { name: 'Sign in' }).click();++ // The migration-seeded admin carries password_change_required — the forced+ // change screen is skippable and must not block the suite.+ const skip = page.getByRole('button', { name: 'Skip for now' });+ const loggedIn = page.getByRole('button', { name: /admin@|flows/i });++ await expect(skip.or(loggedIn).first()).toBeVisible({ timeout: 15_000 });++ if (await skip.isVisible()) {+ await skip.click();+ }++ await expect(page).toHaveURL(/\/flows/, { timeout: 15_000 });+ await page.context().storageState({ path: AUTH_STATE_PATH });+});diff --git a/frontend/e2e/specs/real/flow-run.spec.ts b/frontend/e2e/specs/real/flow-run.spec.tsnew file mode 100644index 00000000..97c5f16e--- /dev/null+++ b/frontend/e2e/specs/real/flow-run.spec.ts@@ -0,0 +1,23 @@+import { expect, test } from '@playwright/test';++test.describe('real backend flow run', { tag: '@real' }, () => {+ test('runs a flow end-to-end through the mock LLM', async ({ page }) => {+ test.setTimeout(240_000);++ await page.goto('/flows/new');+ await page.getByPlaceholder(/Describe what you would like PentAGI to test/).fill('Say hello');+ await page.getByRole('button', { name: 'Submit' }).click();++ await expect(page).toHaveURL(/\/flows\/\d+/, { timeout: 30_000 });++ await expect(page.getByTestId('flow-message-id').first()).toBeVisible({ timeout: 90_000 });+ await expect(page.getByText('Hello from the e2e mock LLM!').first()).toBeVisible({ timeout: 90_000 });++ // A completed task settles the flow into Waiting (ready for the next+ // input) — Finished only happens via the explicit user action.+ await page.goto('/flows');+ await expect(+ page.getByRole('row', { name: /Say Hello Flow/ }).getByText('Waiting', { exact: true }),+ ).toBeVisible({ timeout: 60_000 });+ });+});diff --git a/frontend/e2e/tools/run-local-tier.sh b/frontend/e2e/tools/run-local-tier.shnew file mode 100755index 00000000..d7a9f4a4--- /dev/null+++ b/frontend/e2e/tools/run-local-tier.sh@@ -0,0 +1,49 @@+#!/usr/bin/env bash+# Tier-2 runner: branch image + isolated compose stack (coexists with a dev+# stack — own project/ports) + specs/real/** against the live backend.+set -euo pipefail++REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"+export PENTAGI_IMAGE="${PENTAGI_IMAGE:-local/pentagi:e2e}"+export PENTAGI_LISTEN_PORT="${PENTAGI_LISTEN_PORT:-8444}"+export PGVECTOR_LISTEN_PORT="${PGVECTOR_LISTEN_PORT:-5433}"++# --env-file /dev/null: without it compose auto-loads the developer's .env and+# the stack inherits their live provider keys, DOCKER_HOST, and config paths.+compose() {+ docker compose -p pentagi-e2e --env-file /dev/null \+ -f "$REPO_ROOT/docker-compose.yml" \+ -f "$REPO_ROOT/docker-compose.e2e.yml" "$@"+}++# Sandbox containers are spawned by the backend straight on the docker socket,+# outside the compose project — `down -v` never removes them. The 9xxxx flow-id+# range (seeded below) makes them unambiguously ours to delete.+remove_e2e_sandboxes() {+ docker ps -aq --filter "name=pentagi-terminal-9" | xargs -r docker rm -f+}++cleanup() {+ remove_e2e_sandboxes+ if [[ "${E2E_KEEP_STACK:-}" != "1" ]]; then+ compose down -v --remove-orphans+ fi+}+trap cleanup EXIT++if [[ "${E2E_SKIP_BUILD:-}" != "1" ]]; then+ docker build -t "$PENTAGI_IMAGE" "$REPO_ROOT"+fi++remove_e2e_sandboxes+compose down -v --remove-orphans+compose up -d --wait --wait-timeout 240 pentagi mock-llm++# Flow ids drive sandbox container names (pentagi-terminal-<id>); the 9xxxx+# offset keeps them clear of any developer stack sharing this docker daemon.+docker exec pentagi-e2e-pgvector psql -U postgres -d pentagidb \+ -c "ALTER SEQUENCE flows_id_seq RESTART WITH 90001" > /dev/null++cd "$REPO_ROOT/frontend"+rm -f e2e/.auth/user.json+E2E_TIER=local E2E_BASE_URL="https://localhost:${PENTAGI_LISTEN_PORT}" corepack pnpm e2e "$@"