Fully autonomous AI Agents system capable of performing complex penetration testing tasks
Go ai-agentsai-security-toolanthropicautonomous-agentsgolanggptgraphqlmulti-agent-systemoffensive-securityopen-sourceopenaipenetration-testingpenetration-testing-toolsreactsecurity-automationsecurity-testingsecurity-toolsself-hosted
chore(e2e): scope Tier 2 networks and gate pentagi on a healthy mock LLM
Sergey Kozyrenko179c814bb339
diff --git a/docker-compose.e2e.yml b/docker-compose.e2e.ymlindex 3778431b..daacaade 100644--- a/docker-compose.e2e.yml+++ b/docker-compose.e2e.yml@@ -11,9 +11,16 @@# developer's .env and the stack inherits live provider keys and DOCKER_HOST.# (frontend/e2e/tools/run-local-tier.sh wraps all of this.)+# All three base networks get e2e names — the base file gives them fixed+# `name:`s, so leaving any un-overridden would join this stack to the+# developer's live observability/langfuse networks.networks:pentagi-network:name: pentagi-e2e-network+ observability-network:+ name: pentagi-e2e-observability-network+ langfuse-network:+ name: pentagi-e2e-langfuse-networkservices:mock-llm:@@ -25,6 +32,13 @@ services:- ./frontend/e2e/mock-llm:/app:ronetworks:- pentagi-network+ # Keeps a crash-looping mock (restart would silently mask it) from ever+ # reading as a started dependency.+ healthcheck:+ test: ["CMD-SHELL", "wget -qO /dev/null http://127.0.0.1:8080/v1/models"]+ interval: 3s+ timeout: 5s+ retries: 20pentagi:container_name: pentagi-e2e@@ -32,7 +46,7 @@ services:pgvector:condition: service_healthymock-llm:- condition: service_started+ condition: service_healthyenvironment:- LLM_SERVER_URL=http://mock-llm:8080/v1# The langchaingo client refuses to construct without a token even thoughdiff --git a/frontend/docs/e2e.md b/frontend/docs/e2e.mdindex ce19145c..f5f8c3ac 100644--- a/frontend/docs/e2e.md+++ b/frontend/docs/e2e.md@@ -28,14 +28,18 @@ CI=1 pnpm e2e # byte-identical reproduction of a CI run./e2e/tools/run-local-tier.sh # Tier 2: branch image + isolated docker# stack + mock LLM; runs specs/real/**-E2E_TIER=stand E2E_BASE_URL=https://… pnpm e2e # against a live stand+E2E_TIER=stand E2E_BASE_URL=https://… pnpm e2e # live stand: @stand smoke only+ # (flow-run drives a real paid agent run+ # and is scoped to the local tier)pnpm e2e:visual # visual snapshots (pinned container)pnpm e2e:visual:update # regenerate baselines after a UI change```-Tips for the mock tier: `vite preview` is reused between runs if you keep it-running (`reuseExistingServer`), so only the first run pays the build.+Every mock-tier run rebuilds the bundle and starts its own `vite preview` —+deliberately: reusing a server already listening on the port would silently+test a previous commit's dist. A stray listener on 8100 fails the run loudly;+kill it and rerun.## Tiers@@ -105,7 +109,7 @@ Key conventions:## Stand tier (Tier 3)Runs the LLM-independent `@stand` smoke against a real deployment. Label a PR-`e2e:stand` (or dispatch the workflow); the job runs in a protected+`e2e:stand` (or dispatch the workflow with `tier: stand`); the job runs in a protected`e2e-stand` Environment whose required reviewers approve before any secret isexposed, so even a mislabeled run blocks on a human. Fork PRs never reach it.@@ -119,10 +123,11 @@ prefix `NODE_TLS_REJECT_UNAUTHORIZED=0`; a real stand has a valid cert).## Trends and selective runs-- **Trend:** `node e2e/tools/trend.mjs` turns a run's `results.json` into one- JSONL record (p50/p95 spec duration, slowest three, pass/flaky/fail counts).- CI appends it to the retained `e2e-trend` artifact so duration creep and flake- are visible, not just green/red.+- **Trend:** `node e2e/tools/trend.mjs` turns a run's `results.json` (written+ under `CI=1`) into one JSONL record (p50/p95 spec duration, slowest three,+ pass/flaky/fail counts). Each CI run uploads its own `e2e-trend` artifact;+ aggregate across runs offline (`gh run download`) to see duration creep and+ flake, not just green/red.- **Affected routes:** `pnpm exec tsx e2e/tools/affected.ts <base>` prints themanifest routes a diff touches (backed by each route's owning `sources` in`e2e/routes.ts`). Empty output = no frontend route changed. This is the@@ -153,6 +158,3 @@ bespoke bot:This section is the recipe, not yet wired — the deterministic tiers above arethe foundation it plugs into.--The full design (tiers, CI topology, mocking contract, roadmap) lives in the-team's E2E testing plan; scenario coverage maps 1:1 to the scenario catalog IDs.