High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
E2E Test Suite
End-to-end tests for PinchTab that exercise the full stack including browser automation.
Quick Start
With Docker (recommended)
./dev e2e # Run the extended suite quietly by default
./dev e2e --browser=cloak
./dev e2e basic # Run the basic suite (api + cli + infra basic tests)
./dev e2e extended # Run the extended suite
./dev e2e smoke # Run the CI-backed smoke subset
./dev smoke # Run all local smoke categories
./dev smoke --browser=cloak
./dev smoke cdp-attach # Run only the CDP attach smoke
./dev e2e api # Run API basic tests
./dev e2e cli # Run CLI basic tests
./dev e2e infra # Run infra basic tests
./dev e2e api-extended # Run API extended tests
./dev e2e cli-extended # Run CLI extended tests
./dev e2e infra-extended # Run infra extended tests
./dev e2e api logs=show # Opt back into full streaming logs
Or directly through the Go runner:
go run ./tests/tools/runner e2e --suite basic
go run ./tests/tools/runner e2e --suite extended
go run ./tests/tools/runner e2e --suite smoke
go run ./tests/tools/runner e2e --suite infra-extended --filter orchestrator
E2E System Boundary
The E2E system has three layers with separate responsibilities. Keep this boundary explicit when adding scenarios or refactoring the runner.
Go Runner
go run ./tests/tools/runner e2e ... is the host-side orchestrator. It owns:
- Suite normalization and expansion (
basic,extended,smoke, group suites, and filtered smoke suites) - Scenario discovery from
tests/e2e/scenarios/*/*.sh - Applying
tests/e2e/scenarios/manifest.jsonmetadata - Selecting the compose stack, compose services, and readiness targets
- Passing explicit
scenario=<file>arguments into the container executor - Host Docker smoke checks and their required image build steps
- Log capture, result cleanup, summaries, markdown reports, failure accounting, GitHub Actions outputs, and GitHub Actions summaries
The runner does not own test assertions, endpoint request details, or CLI command behavior inside a scenario. Those stay in scenario files and helpers.
Manifest
tests/e2e/scenarios/manifest.json is metadata, not the test list. Shell files are still discovered from disk. The manifest only controls per-scenario overrides:
tier:basic,extended, orsmokehelper:apiorcliservices: compose services required by the scenarioready: readiness targets required before the scenario runstags: filter labels
By default, tier comes from the filename suffix: *-basic.sh is basic, *-smoke.sh is smoke, and every other scenario is extended. The default helper is cli for scenarios/cli/ and api for the other groups. The default services are pinchtab and fixtures.
Tiers
basicis the PR happy path: fast, representative coverage with small setup../dev e2e basicruns the API, CLI, and Infra basic suites.extendedis deeper coverage: edge cases, detailed interaction checks, and tests that are useful before release. Extended suites include the matchingbasicscenarios plusextendedscenarios for that group.smokeis independent high-setup coverage: lifecycle, multi-instance, host Docker, and production-like checks that do not belong in PR flow../dev e2e smokeruns only the CI-backed*-smoke.shscenarios; it does not includebasic,extended, or opt-in provider/live-detection smokes../dev smokeis the local full-smoke entrypoint and also runs provider parity, CDP attach, and live-detection categories unless filtered.
CloakBrowser smoke checks need a Linux CloakBrowser binary and a compatible
PinchTab image, so they are exposed through local ./dev smoke filters
instead of being included in the CI-backed ./dev e2e smoke subset. The
./dev smoke cloakbrowser filter builds
tests/tools/docker/cloakbrowser-smoke.Dockerfile into
pinchtab-cloakbrowser:test and uses tmpfs mounts for browser profile and
socket paths. The smoke reuses tests/e2e/fixtures, checks a representative
endpoint subset, and then invokes the existing E2E scenario executor in the API
runner container attached to the Cloak-backed container's network namespace for
the Cloak-compatible API basic scenario set. This uses the same
tests/e2e/run.sh API helper and assertion base as the normal Chrome API
suite, while keeping CloakBrowser out of CI by default.
Adding A Scenario
- Add or update a grouped entrypoint under
tests/e2e/scenarios/api/,tests/e2e/scenarios/cli/,tests/e2e/scenarios/infra/, ortests/e2e/scenarios/plugin/. - Pick the tier by filename:
feature-basic.sh,feature-extended.sh, orfeature-smoke.sh. - Source the local helper, usually
../../helpers/api.shor../../helpers/cli.sh. - Add a manifest entry only when the scenario needs non-default services, readiness targets, helper, tier, or tags.
- Check the plan before running the full suite:
go run ./tests/tools/runner e2e --suite basic --dry-run
go run ./tests/tools/runner e2e --suite extended --filter feature --dry-run
go run ./tests/tools/runner e2e --suite smoke --filter feature --dry-run
Filtering
--filter TEXT selects scenario files before compose planning. It is a case-sensitive substring match against the scenario file name, manifest key, group, tier, helper, and tags. Suites with no matching scenarios are skipped, and the runner only starts services required by the remaining scenarios.
For host Docker smoke checks, --filter matches the smoke step name or tags; required image build steps are included automatically when a filtered step depends on them.
At the ./dev smoke layer, positional names such as cdp-attach,
live-detection, and cloakbrowser select those local smoke categories.
Unknown names are passed to the CI-backed E2E smoke runner as --filter, so
./dev smoke orchestrator maps to the orchestrator smoke scenario.
--test TEXT is different: it does not select scenario files or compose services. It passes E2E_TEST_FILTER into the container so run.sh can run one matching start_test block inside the already-selected scenarios.
CI Path
CI uses the Go runner directly:
.github/workflows/reusable-e2e.ymlrunsgo run ./tests/tools/runner e2e --suite <suite> --logs hide.github/workflows/reusable-smoke.ymlrunsgo run ./tests/tools/runner e2e --suite smoke --logs hide.github/workflows/ci-e2e.ymlchooses the PR, extended-on-touch, extended-on-merge, manual, and smoke jobs, then delegates execution to the reusable workflows.github/workflows/ci-smoke.ymlis the nightly smoke entrypoint, and the manual one
Workflow YAML should stay thin: it decides when to run suites, while scenario selection, services, logs, reports, failure summaries, and GitHub Actions outputs stay in the Go layer.
Architecture
tests/e2e/
├── docker-compose.yml # Single-instance stack for basic suites
├── docker-compose-multi.yml # Multi-instance extended stack
├── config/ # E2E-specific PinchTab configs
│ ├── pinchtab.json
│ ├── pinchtab-medium-permissive.json
│ ├── pinchtab-full-permissive.json
│ ├── pinchtab-secure.json
│ └── pinchtab-bridge.json
├── fixtures/ # Static HTML test pages
│ ├── index.html
│ ├── form.html
│ ├── table.html
│ └── buttons.html
├── helpers/ # Shared API/CLI E2E helpers
│ ├── api.sh
│ ├── api-http.sh
│ ├── api-assertions.sh
│ ├── api-actions.sh
│ ├── api-snapshot.sh
│ ├── cli.sh
│ └── base.sh
├── scenarios/ # Test scenarios organized by type
│ ├── api/ # Browser control and page interaction
│ │ ├── browser-basic.sh
│ │ ├── browser-extended.sh
│ │ ├── tabs-basic.sh
│ │ ├── tabs-extended.sh
│ │ ├── actions-basic.sh
│ │ ├── actions-extended.sh
│ │ ├── files-basic.sh
│ │ ├── files-extended.sh
│ │ ├── clipboard-basic.sh
│ │ └── console-basic.sh
│ ├── cli/ # CLI command tests
│ │ ├── browser-basic.sh
│ │ ├── browser-extended.sh
│ │ ├── tabs-basic.sh
│ │ ├── tabs-extended.sh
│ │ └── ...
│ └── infra/ # System, network, security, stealth
│ ├── system-basic.sh
│ ├── system-extended.sh
│ ├── network-basic.sh
│ ├── network-extended.sh
│ ├── security-basic.sh
│ ├── security-extended.sh
│ ├── stealth-basic.sh
│ ├── stealth-extended.sh
│ ├── orchestrator-extended.sh
│ ├── auth-extended.sh
│ └── ...
├── runner-api/ # API test runner container
│ └── Dockerfile
├── runner-cli/ # CLI test runner container
│ └── Dockerfile
└── results/ # Test output (gitignored)
The Docker stacks reuse the repository root Dockerfile and mount explicit config files with PINCHTAB_CONFIG instead of maintaining separate e2e-only images.
Test Groups
Tests are organized into three parallel groups:
API Group (scenarios/api/)
Browser control and page interaction tests:
browser-basic/browser-extendedtabs-basic/tabs-extendedactions-basic/actions-extendedfiles-basic/files-extendedclipboard-basicconsole-basic
CLI Group (scenarios/cli/)
CLI command tests:
browser-basic/browser-extendedtabs-basic/tabs-extendedactions-basic/actions-extendedfiles-basic/files-extendedsystem-basic/system-extended- And more...
Infra Group (scenarios/infra/)
System, networking, security, and stealth tests:
system-basic/system-extendednetwork-basic/network-extendedsecurity-basic/security-extendedstealth-basic/stealth-extendedorchestrator-extendedauth-extendedautosolver-smokedashboard-smoke- manual autosolver check lives at
tests/manual/autosolver-check.sh - real-world autosolver smoke lives at
scripts/autosolver-realworld-smoke.sh idpi-extended
The basic entrypoints are the PR happy path. The extended entrypoints add extra and edge-case coverage; extended suites include their matching basic scenarios. The smoke tier is separate and runs only *-smoke.sh scenarios plus host-level Docker smoke checks, not basic or extended scenarios. The Go runner selects the exact scenario files before entering the container; run.sh is only the in-container executor and requires explicit scenario=<file> arguments from the host runner.
Compose usage:
docker-compose.ymlpowersapi,cli,infra, andcli-extendeddocker-compose-multi.ymlpowersapi-extendedandinfra-extended
Adding Tests
- Add or update a grouped entrypoint such as
tabs-basic.sh,tabs-extended.sh, ortabs-smoke.sh - Source
../../helpers/api.shor../../helpers/cli.sh - Put PR happy-path coverage in
*-basic.sh, deeper coverage in*-extended.sh, and slow/high-setup checks in*-smoke.sh - Add or update
scenarios/manifest.jsononly when the scenario needs non-default services, readiness targets, helper, tier, or tags - Verify selection and service scope with
go run ./tests/tools/runner e2e --suite <suite> --filter <name> --dry-run - Use the assertion helpers:
#!/bin/bash
GROUP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "${GROUP_DIR}/../../helpers/api.sh"
start_test "My test name"
# Assert HTTP status
assert_status 200 "${PINCHTAB_URL}/health"
# Assert JSON field equals value
RESULT=$(pt_get "/some/endpoint")
assert_json_eq "$RESULT" '.field' 'expected'
# Assert JSON contains substring
assert_json_contains "$RESULT" '.message' 'success'
# Assert array length
assert_json_length "$RESULT" '.items' 5
end_test
The action scenarios already cover common interaction regressions against the bundled fixtures:
tests/e2e/scenarios/api/actions-basic.shgroups the API happy-path actionstests/e2e/scenarios/cli/actions-basic.shgroups the matching CLI commands
Adding Fixtures
Add HTML files to fixtures/ for testing specific scenarios:
- Forms and inputs
- Tables and data
- Dynamic content
- iframes
- File upload/download
CI Integration
The E2E tests run automatically:
- On PRs:
api,cli, andinfrabasic tests always run - On PRs: touching any non-basic scenario also triggers the matching extended suite on its native compose stack
- On PRs: touching smoke scenarios or Docker smoke inputs triggers the smoke workflow
- Manually via workflow dispatch: Extended tests for all groups
Result Files
The Go e2e runner captures runner output and writes each suite's result files in tests/e2e/results/. The container executor emits structured E2E_RESULT lines as tests finish, and the Go layer turns those into the final console summary, GitHub Actions summaries, durable summaries, and reports:
summary-api.txt/report-api.mdsummary-api-extended.txt/report-api-extended.mdsummary-cli.txt/report-cli.mdsummary-cli-extended.txt/report-cli-extended.mdsummary-infra.txt/report-infra.mdsummary-infra-extended.txt/report-infra-extended.mdsummary-api-smoke.txt/report-api-smoke.mdsummary-cli-smoke.txt/report-cli-smoke.mdsummary-infra-smoke.txt/report-infra-smoke.mdsummary-plugin-smoke.txt/report-plugin-smoke.mdsummary-docker-smoke.txt/report-docker-smoke.md
The runner deletes the target suite files before each run to avoid stale output. Saved and printed summaries include total test time and suite wall time.
The runner also saves the captured suite log under output-*.log and captures compose service logs on failure. When logs=hide is used and a suite fails, it prints the relevant failure summary and artifact paths.
Debugging
View container logs
docker compose -f tests/e2e/docker-compose.yml logs pinchtab
docker compose -f tests/e2e/docker-compose-multi.yml logs pinchtab
Interactive shell in runner
docker compose -f tests/e2e/docker-compose.yml run runner-api bash
docker compose -f tests/e2e/docker-compose.yml run runner-cli bash
Run specific scenario
go run ./tests/tools/runner e2e --suite api --filter tabs-basic
go run ./tests/tools/runner e2e --suite api-extended --filter tabs-extended
go run ./tests/tools/runner e2e --suite api-extended --filter tabs-extended --test "tab-scoped snapshot"
Orchestrator Coverage
infra-extended uses docker-compose-multi.yml and includes the multi-instance and remote-bridge orchestrator scenarios through orchestrator-extended.sh.