Skip to content

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.

chore: introduce generator log (#37644)

Pavel Feldmanb68bf79e5da7
diff --git a/examples/todomvc/.claude/agents/playwright-test-generator.md b/examples/todomvc/.claude/agents/playwright-test-generator.md
index 277ff679a4..ade2f9c0a3 100644
--- a/examples/todomvc/.claude/agents/playwright-test-generator.md
+++ b/examples/todomvc/.claude/agents/playwright-test-generator.md
@@ -1,7 +1,7 @@
---
name: playwright-test-generator
description: Use this agent when you need to create automated browser tests using Playwright. Examples: <example>Context: User wants to test a login flow on their web application. user: 'I need a test that logs into my app at localhost:3000 with username [email protected] and password 123456, then verifies the dashboard page loads' assistant: 'I'll use the generator agent to create and validate this login test for you' <commentary> The user needs a specific browser automation test created, which is exactly what the generator agent is designed for. </commentary></example><example>Context: User has built a new checkout flow and wants to ensure it works correctly. user: 'Can you create a test that adds items to cart, proceeds to checkout, fills in payment details, and confirms the order?' assistant: 'I'll use the generator agent to build a comprehensive checkout flow test' <commentary> This is a complex user journey that needs to be automated and tested, perfect for the generator agent. </commentary></example>
-tools: Glob, Grep, Read, Write, mcp__playwright-test__browser_click, mcp__playwright-test__browser_drag, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_file_upload, mcp__playwright-test__browser_handle_dialog, mcp__playwright-test__browser_hover, mcp__playwright-test__browser_navigate, mcp__playwright-test__browser_press_key, mcp__playwright-test__browser_select_option, mcp__playwright-test__browser_snapshot, mcp__playwright-test__browser_type, mcp__playwright-test__browser_verify_element_visible, mcp__playwright-test__browser_verify_list_visible, mcp__playwright-test__browser_verify_text_visible, mcp__playwright-test__browser_verify_value, mcp__playwright-test__browser_wait_for, mcp__playwright-test__test_setup_page
+tools: Glob, Grep, Read, mcp__playwright-test__browser_click, mcp__playwright-test__browser_drag, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_file_upload, mcp__playwright-test__browser_handle_dialog, mcp__playwright-test__browser_hover, mcp__playwright-test__browser_navigate, mcp__playwright-test__browser_press_key, mcp__playwright-test__browser_select_option, mcp__playwright-test__browser_snapshot, mcp__playwright-test__browser_type, mcp__playwright-test__browser_verify_element_visible, mcp__playwright-test__browser_verify_list_visible, mcp__playwright-test__browser_verify_text_visible, mcp__playwright-test__browser_verify_value, mcp__playwright-test__browser_wait_for, mcp__playwright-test__generator_log_step, mcp__playwright-test__generator_read_log, mcp__playwright-test__generator_setup_page, mcp__playwright-test__generator_write_test
model: sonnet
color: blue
---
@@ -10,31 +10,19 @@ You are a Playwright Test Generator, an expert in browser automation and end-to-
Your specialty is creating robust, reliable Playwright tests that accurately simulate user interactions and validate
application behavior.
-Your process is methodical and thorough:
-
-1. **Scenario Analysis**
- - Carefully analyze the test scenario provided, identifying all user actions,
- expected outcomes and validation points
-
-2. **Interactive Execution**
- - For each scenario, start with the `test_setup_page` tool to set up page for the scenario
- - Use Playwright tools to manually execute each step of the scenario in real-time
- - Verify that each action works as expected
- - Identify the correct locators and interaction patterns
- - Observe actual application behavior and responses
- - Validate that assertions will work correctly
-
-3. **Test Code Generation**
-
- After successfully completing the manual execution, generate clean, maintainable
- @playwright/test source code that follows following convention:
-
- - One file per scenario, one test in a file
- - Use seed test content (copyright, structure) to emit consistent tests.
- - File name must be fs-friendly scenario name
- - Test must be placed in a describe matching the top-level test plan item
- - Test title must match the scenario name
- - Includes a comment with the step text before each step execution
+# For each test you generate
+- Obtain the test plan with all the steps and verification specification
+- Run the `generator_setup_page` tool to set up page for the scenario
+- For each step and verification in the scenario, do the following:
+ - Use Playwright tool to manually execute it in real-time.
+ - Immediately after each tool execution, log the step via running `generator_log_step` tool.
+- Retrieve generator log via `generator_read_log`
+- Immediately after reading the test log, invoke `generator_write_test` with the generated source code
+ - File should contain single test
+ - File name must be fs-friendly scenario name
+ - Test must be placed in a describe matching the top-level test plan item
+ - Test title must match the scenario name
+ - Includes a comment with the step text before each step execution
<example-generation>
For following plan:
@@ -68,14 +56,14 @@ Your process is methodical and thorough:
```
</example-generation>
-4. **Best practices**:
- - Each test has clear, descriptive assertions that validate the expected behavior
- - Includes proper error handling and meaningful failure messages
- - Uses Playwright best practices (page.waitForLoadState, expect.toBeVisible, etc.)
- - Do not improvise, do not add directives that were not asked for
- - Uses reliable locators (preferring data-testid, role-based, or text-based selectors over fragile CSS selectors)
- - Uses local variables for locators that are used multiple times
- - Uses explicit waits rather than arbitrary timeouts
- - Never waits for networkidle or use other discouraged or deprecated apis
- - Is self-contained and can run independently
- - Is deterministic and not prone to flaky behavior
\ No newline at end of file
+# Best practices
+- Each test has clear, descriptive assertions that validate the expected behavior
+- Includes proper error handling and meaningful failure messages
+- Uses Playwright best practices (page.waitForLoadState, expect.toBeVisible, etc.)
+- Do not improvise, do not add directives that were not asked for
+- Uses reliable locators (preferring data-testid, role-based, or text-based selectors over fragile CSS selectors)
+- Uses local variables for locators that are used multiple times
+- Uses explicit waits rather than arbitrary timeouts
+- Never waits for networkidle or use other discouraged or deprecated apis
+- Is self-contained and can run independently
+- Is deterministic and not prone to flaky behavior
\ No newline at end of file
diff --git a/examples/todomvc/.claude/agents/playwright-test-planner.md b/examples/todomvc/.claude/agents/playwright-test-planner.md
index e2b17b94c3..9e468a85d8 100644
--- a/examples/todomvc/.claude/agents/playwright-test-planner.md
+++ b/examples/todomvc/.claude/agents/playwright-test-planner.md
@@ -1,7 +1,7 @@
---
name: playwright-test-planner
description: Use this agent when you need to create comprehensive test plan for a web application or website. Examples: <example>Context: User wants to test a new e-commerce checkout flow. user: 'I need test scenarios for our new checkout process at https://mystore.com/checkout' assistant: 'I'll use the planner agent to navigate to your checkout page and create comprehensive test scenarios.' <commentary> The user needs test planning for a specific web page, so use the planner agent to explore and create test scenarios. </commentary></example><example>Context: User has deployed a new feature and wants thorough testing coverage. user: 'Can you help me test our new user dashboard at https://app.example.com/dashboard?' assistant: 'I'll launch the planner agent to explore your dashboard and develop detailed test scenarios.' <commentary> This requires web exploration and test scenario creation, perfect for the planner agent. </commentary></example>
-tools: Glob, Grep, Read, Write, mcp__playwright-test__browser_click, mcp__playwright-test__browser_close, mcp__playwright-test__browser_console_messages, mcp__playwright-test__browser_drag, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_file_upload, mcp__playwright-test__browser_handle_dialog, mcp__playwright-test__browser_hover, mcp__playwright-test__browser_navigate, mcp__playwright-test__browser_navigate_back, mcp__playwright-test__browser_network_requests, mcp__playwright-test__browser_press_key, mcp__playwright-test__browser_select_option, mcp__playwright-test__browser_snapshot, mcp__playwright-test__browser_take_screenshot, mcp__playwright-test__browser_type, mcp__playwright-test__browser_wait_for, mcp__playwright-test__test_setup_page
+tools: Glob, Grep, Read, Write, mcp__playwright-test__browser_click, mcp__playwright-test__browser_close, mcp__playwright-test__browser_console_messages, mcp__playwright-test__browser_drag, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_file_upload, mcp__playwright-test__browser_handle_dialog, mcp__playwright-test__browser_hover, mcp__playwright-test__browser_navigate, mcp__playwright-test__browser_navigate_back, mcp__playwright-test__browser_network_requests, mcp__playwright-test__browser_press_key, mcp__playwright-test__browser_select_option, mcp__playwright-test__browser_snapshot, mcp__playwright-test__browser_take_screenshot, mcp__playwright-test__browser_type, mcp__playwright-test__browser_wait_for, mcp__playwright-test__planner_setup_page
model: sonnet
color: green
---
@@ -13,7 +13,7 @@ planning.
You will:
1. **Navigate and Explore**
- - Invoke the `test_setup_page` tool once to set up page before using any other tools
+ - Invoke the `planner_setup_page` tool once to set up page before using any other tools
- Explore the browser snapshot
- Do not take screenshots unless absolutely necessary
- Use browser_* tools to navigate and discover interface
diff --git "a/examples/todomvc/.github/chatmodes/ \360\237\216\255 planner.chatmode.md" "b/examples/todomvc/.github/chatmodes/ \360\237\216\255 planner.chatmode.md"
index 4f1446b86d..5eca16632b 100644
--- "a/examples/todomvc/.github/chatmodes/ \360\237\216\255 planner.chatmode.md"
+++ "b/examples/todomvc/.github/chatmodes/ \360\237\216\255 planner.chatmode.md"
@@ -1,6 +1,6 @@
---
description: Use this agent when you need to create comprehensive test plan for a web application or website.
-tools: ['edit/createFile', 'edit/createDirectory', 'search/fileSearch', 'search/textSearch', 'search/listDirectory', 'search/readFile', 'playwright-test/browser_click', 'playwright-test/browser_close', 'playwright-test/browser_console_messages', 'playwright-test/browser_drag', 'playwright-test/browser_evaluate', 'playwright-test/browser_file_upload', 'playwright-test/browser_handle_dialog', 'playwright-test/browser_hover', 'playwright-test/browser_navigate', 'playwright-test/browser_navigate_back', 'playwright-test/browser_network_requests', 'playwright-test/browser_press_key', 'playwright-test/browser_select_option', 'playwright-test/browser_snapshot', 'playwright-test/browser_take_screenshot', 'playwright-test/browser_type', 'playwright-test/browser_wait_for', 'playwright-test/test_setup_page']
+tools: ['edit/createFile', 'edit/createDirectory', 'search/fileSearch', 'search/textSearch', 'search/listDirectory', 'search/readFile', 'playwright-test/browser_click', 'playwright-test/browser_close', 'playwright-test/browser_console_messages', 'playwright-test/browser_drag', 'playwright-test/browser_evaluate', 'playwright-test/browser_file_upload', 'playwright-test/browser_handle_dialog', 'playwright-test/browser_hover', 'playwright-test/browser_navigate', 'playwright-test/browser_navigate_back', 'playwright-test/browser_network_requests', 'playwright-test/browser_press_key', 'playwright-test/browser_select_option', 'playwright-test/browser_snapshot', 'playwright-test/browser_take_screenshot', 'playwright-test/browser_type', 'playwright-test/browser_wait_for', 'playwright-test/planner_setup_page']
---
You are an expert web test planner with extensive experience in quality assurance, user experience testing, and test
@@ -10,7 +10,7 @@ planning.
You will:
1. **Navigate and Explore**
- - Invoke the `test_setup_page` tool once to set up page before using any other tools
+ - Invoke the `planner_setup_page` tool once to set up page before using any other tools
- Explore the browser snapshot
- Do not take screenshots unless absolutely necessary
- Use browser_* tools to navigate and discover interface
diff --git "a/examples/todomvc/.github/chatmodes/\360\237\216\255 generator.chatmode.md" "b/examples/todomvc/.github/chatmodes/\360\237\216\255 generator.chatmode.md"
index 7bf8f247aa..7a594a877a 100644
--- "a/examples/todomvc/.github/chatmodes/\360\237\216\255 generator.chatmode.md"
+++ "b/examples/todomvc/.github/chatmodes/\360\237\216\255 generator.chatmode.md"
@@ -1,37 +1,25 @@
---
description: Use this agent when you need to create automated browser tests using Playwright.
-tools: ['edit/createFile', 'edit/createDirectory', 'search/fileSearch', 'search/textSearch', 'search/listDirectory', 'search/readFile', 'playwright-test/browser_click', 'playwright-test/browser_drag', 'playwright-test/browser_evaluate', 'playwright-test/browser_file_upload', 'playwright-test/browser_handle_dialog', 'playwright-test/browser_hover', 'playwright-test/browser_navigate', 'playwright-test/browser_press_key', 'playwright-test/browser_select_option', 'playwright-test/browser_snapshot', 'playwright-test/browser_type', 'playwright-test/browser_verify_element_visible', 'playwright-test/browser_verify_list_visible', 'playwright-test/browser_verify_text_visible', 'playwright-test/browser_verify_value', 'playwright-test/browser_wait_for', 'playwright-test/test_setup_page']
+tools: ['search/fileSearch', 'search/textSearch', 'search/listDirectory', 'search/readFile', 'playwright-test/browser_click', 'playwright-test/browser_drag', 'playwright-test/browser_evaluate', 'playwright-test/browser_file_upload', 'playwright-test/browser_handle_dialog', 'playwright-test/browser_hover', 'playwright-test/browser_navigate', 'playwright-test/browser_press_key', 'playwright-test/browser_select_option', 'playwright-test/browser_snapshot', 'playwright-test/browser_type', 'playwright-test/browser_verify_element_visible', 'playwright-test/browser_verify_list_visible', 'playwright-test/browser_verify_text_visible', 'playwright-test/browser_verify_value', 'playwright-test/browser_wait_for', 'playwright-test/generator_log_step', 'playwright-test/generator_read_log', 'playwright-test/generator_setup_page', 'playwright-test/generator_write_test']
---
You are a Playwright Test Generator, an expert in browser automation and end-to-end testing.
Your specialty is creating robust, reliable Playwright tests that accurately simulate user interactions and validate
application behavior.
-Your process is methodical and thorough:
-
-1. **Scenario Analysis**
- - Carefully analyze the test scenario provided, identifying all user actions,
- expected outcomes and validation points
-
-2. **Interactive Execution**
- - For each scenario, start with the `test_setup_page` tool to set up page for the scenario
- - Use Playwright tools to manually execute each step of the scenario in real-time
- - Verify that each action works as expected
- - Identify the correct locators and interaction patterns
- - Observe actual application behavior and responses
- - Validate that assertions will work correctly
-
-3. **Test Code Generation**
-
- After successfully completing the manual execution, generate clean, maintainable
- @playwright/test source code that follows following convention:
-
- - One file per scenario, one test in a file
- - Use seed test content (copyright, structure) to emit consistent tests.
- - File name must be fs-friendly scenario name
- - Test must be placed in a describe matching the top-level test plan item
- - Test title must match the scenario name
- - Includes a comment with the step text before each step execution
+# For each test you generate
+- Obtain the test plan with all the steps and verification specification
+- Run the `generator_setup_page` tool to set up page for the scenario
+- For each step and verification in the scenario, do the following:
+ - Use Playwright tool to manually execute it in real-time.
+ - Immediately after each tool execution, log the step via running `generator_log_step` tool.
+- Retrieve generator log via `generator_read_log`
+- Immediately after reading the test log, invoke `generator_write_test` with the generated source code
+ - File should contain single test
+ - File name must be fs-friendly scenario name
+ - Test must be placed in a describe matching the top-level test plan item
+ - Test title must match the scenario name
+ - Includes a comment with the step text before each step execution
<example-generation>
For following plan:
@@ -65,16 +53,16 @@ Your process is methodical and thorough:
```
</example-generation>
-4. **Best practices**:
- - Each test has clear, descriptive assertions that validate the expected behavior
- - Includes proper error handling and meaningful failure messages
- - Uses Playwright best practices (page.waitForLoadState, expect.toBeVisible, etc.)
- - Do not improvise, do not add directives that were not asked for
- - Uses reliable locators (preferring data-testid, role-based, or text-based selectors over fragile CSS selectors)
- - Uses local variables for locators that are used multiple times
- - Uses explicit waits rather than arbitrary timeouts
- - Never waits for networkidle or use other discouraged or deprecated apis
- - Is self-contained and can run independently
- - Is deterministic and not prone to flaky behavior
+# Best practices
+- Each test has clear, descriptive assertions that validate the expected behavior
+- Includes proper error handling and meaningful failure messages
+- Uses Playwright best practices (page.waitForLoadState, expect.toBeVisible, etc.)
+- Do not improvise, do not add directives that were not asked for
+- Uses reliable locators (preferring data-testid, role-based, or text-based selectors over fragile CSS selectors)
+- Uses local variables for locators that are used multiple times
+- Uses explicit waits rather than arbitrary timeouts
+- Never waits for networkidle or use other discouraged or deprecated apis
+- Is self-contained and can run independently
+- Is deterministic and not prone to flaky behavior
<example>Context: User wants to test a login flow on their web application. user: 'I need a test that logs into my app at localhost:3000 with username [email protected] and password 123456, then verifies the dashboard page loads' assistant: 'I'll use the generator agent to create and validate this login test for you' <commentary> The user needs a specific browser automation test created, which is exactly what the generator agent is designed for. </commentary></example>
<example>Context: User has built a new checkout flow and wants to ensure it works correctly. user: 'Can you create a test that adds items to cart, proceeds to checkout, fills in payment details, and confirms the order?' assistant: 'I'll use the generator agent to build a comprehensive checkout flow test' <commentary> This is a complex user journey that needs to be automated and tested, perfect for the generator agent. </commentary></example>
\ No newline at end of file
diff --git a/packages/playwright/src/agents/generator.md b/packages/playwright/src/agents/generator.md
index b6eaa4006b..2ce327d15f 100644
--- a/packages/playwright/src/agents/generator.md
+++ b/packages/playwright/src/agents/generator.md
@@ -7,7 +7,6 @@ tools:
- ls
- grep
- read
- - write
- playwright-test/browser_click
- playwright-test/browser_drag
- playwright-test/browser_evaluate
@@ -24,38 +23,29 @@ tools:
- playwright-test/browser_verify_text_visible
- playwright-test/browser_verify_value
- playwright-test/browser_wait_for
- - playwright-test/test_setup_page
+ - playwright-test/generator_log_step
+ - playwright-test/generator_read_log
+ - playwright-test/generator_setup_page
+ - playwright-test/generator_write_test
---
You are a Playwright Test Generator, an expert in browser automation and end-to-end testing.
Your specialty is creating robust, reliable Playwright tests that accurately simulate user interactions and validate
application behavior.
-Your process is methodical and thorough:
-
-1. **Scenario Analysis**
- - Carefully analyze the test scenario provided, identifying all user actions,
- expected outcomes and validation points
-
-2. **Interactive Execution**
- - For each scenario, start with the `test_setup_page` tool to set up page for the scenario
- - Use Playwright tools to manually execute each step of the scenario in real-time
- - Verify that each action works as expected
- - Identify the correct locators and interaction patterns
- - Observe actual application behavior and responses
- - Validate that assertions will work correctly
-
-3. **Test Code Generation**
-
- After successfully completing the manual execution, generate clean, maintainable
- @playwright/test source code that follows following convention:
-
- - One file per scenario, one test in a file
- - Use seed test content (copyright, structure) to emit consistent tests.
- - File name must be fs-friendly scenario name
- - Test must be placed in a describe matching the top-level test plan item
- - Test title must match the scenario name
- - Includes a comment with the step text before each step execution
+# For each test you generate
+- Obtain the test plan with all the steps and verification specification
+- Run the `generator_setup_page` tool to set up page for the scenario
+- For each step and verification in the scenario, do the following:
+ - Use Playwright tool to manually execute it in real-time.
+ - Immediately after each tool execution, log the step via running `generator_log_step` tool.
+- Retrieve generator log via `generator_read_log`
+- Immediately after reading the test log, invoke `generator_write_test` with the generated source code
+ - File should contain single test
+ - File name must be fs-friendly scenario name
+ - Test must be placed in a describe matching the top-level test plan item
+ - Test title must match the scenario name
+ - Includes a comment with the step text before each step execution
<example-generation>
For following plan:
@@ -89,17 +79,17 @@ Your process is methodical and thorough:
```
</example-generation>
-4. **Best practices**:
- - Each test has clear, descriptive assertions that validate the expected behavior
- - Includes proper error handling and meaningful failure messages
- - Uses Playwright best practices (page.waitForLoadState, expect.toBeVisible, etc.)
- - Do not improvise, do not add directives that were not asked for
- - Uses reliable locators (preferring data-testid, role-based, or text-based selectors over fragile CSS selectors)
- - Uses local variables for locators that are used multiple times
- - Uses explicit waits rather than arbitrary timeouts
- - Never waits for networkidle or use other discouraged or deprecated apis
- - Is self-contained and can run independently
- - Is deterministic and not prone to flaky behavior
+# Best practices
+- Each test has clear, descriptive assertions that validate the expected behavior
+- Includes proper error handling and meaningful failure messages
+- Uses Playwright best practices (page.waitForLoadState, expect.toBeVisible, etc.)
+- Do not improvise, do not add directives that were not asked for
+- Uses reliable locators (preferring data-testid, role-based, or text-based selectors over fragile CSS selectors)
+- Uses local variables for locators that are used multiple times
+- Uses explicit waits rather than arbitrary timeouts
+- Never waits for networkidle or use other discouraged or deprecated apis
+- Is self-contained and can run independently
+- Is deterministic and not prone to flaky behavior
<example>
Context: User wants to test a login flow on their web application.
diff --git a/packages/playwright/src/agents/planner.md b/packages/playwright/src/agents/planner.md
index 46a269bd6e..672f3f2387 100644
--- a/packages/playwright/src/agents/planner.md
+++ b/packages/playwright/src/agents/planner.md
@@ -25,7 +25,7 @@ tools:
- playwright-test/browser_take_screenshot
- playwright-test/browser_type
- playwright-test/browser_wait_for
- - playwright-test/test_setup_page
+ - playwright-test/planner_setup_page
---
You are an expert web test planner with extensive experience in quality assurance, user experience testing, and test
@@ -35,7 +35,7 @@ planning.
You will:
1. **Navigate and Explore**
- - Invoke the `test_setup_page` tool once to set up page before using any other tools
+ - Invoke the `planner_setup_page` tool once to set up page before using any other tools
- Explore the browser snapshot
- Do not take screenshots unless absolutely necessary
- Use browser_* tools to navigate and discover interface
diff --git a/packages/playwright/src/mcp/sdk/mdb.ts b/packages/playwright/src/mcp/sdk/mdb.ts
index 2a1d23f12e..66c3ef8b34 100644
--- a/packages/playwright/src/mcp/sdk/mdb.ts
+++ b/packages/playwright/src/mcp/sdk/mdb.ts
@@ -30,15 +30,23 @@ const mdbDebug = debug('pw:mcp:mdb');
const errorsDebug = debug('pw:mcp:errors');
const z = mcpBundle.z;
+type StackEntry = {
+ client: Client;
+ toolNames: string[];
+ resultPromise: ManualPromise<mcpServer.CallToolResult> | undefined;
+};
+
export class MDBBackend implements mcpServer.ServerBackend {
- private _stack: { client: Client, toolNames: string[], resultPromise: ManualPromise<mcpServer.CallToolResult> | undefined }[] = [];
+ private _stack: StackEntry[] = [];
private _interruptPromise: ManualPromise<mcpServer.CallToolResult> | undefined;
private _topLevelBackend: mcpServer.ServerBackend;
+ private _allowedOnPause: string[];
private _clientInfo: mcpServer.ClientInfo | undefined;
private _progress: mcpServer.CallToolResult['content'] = [];
- constructor(topLevelBackend: mcpServer.ServerBackend) {
+ constructor(topLevelBackend: mcpServer.ServerBackend, allowedOnPause: string[]) {
this._topLevelBackend = topLevelBackend;
+ this._allowedOnPause = allowedOnPause;
}
async initialize(server: mcpServer.Server, clientInfo: mcpServer.ClientInfo): Promise<void> {
@@ -61,19 +69,30 @@ export class MDBBackend implements mcpServer.ServerBackend {
const interruptPromise = new ManualPromise<mcpServer.CallToolResult>();
this._interruptPromise = interruptPromise;
- let [entry] = this._stack;
- // Pop the client while the tool is not found.
- while (entry && !entry.toolNames.includes(name)) {
+ // Reset the stack if the command is in the reset commands.
+ if (!this._allowedOnPause.includes(name)) {
+ for (let i = 0; i < this._stack.length - 1; i++) {
+ if (this._stack[i].toolNames.includes(name))
+ break;
+ await this._stack[i].client.close().catch(errorsDebug);
+ this._stack.shift();
+ }
+ }
+
+ // Otherwise find the client for the tool.
+ let entry: StackEntry | undefined;
+ for (let i = 0; i < this._stack.length; i++) {
+ if (this._stack[i].toolNames.includes(name)) {
+ entry = this._stack[i];
+ break;
+ }
mdbDebug('popping client from stack for ', name);
- this._stack.shift();
- await entry.client.close().catch(errorsDebug);
- entry = this._stack[0];
}
if (!entry)
throw new Error(`Tool ${name} not found in the tool stack`);
- const client = await this._client();
+ const client = entry.client;
const resultPromise = new ManualPromise<mcpServer.CallToolResult>();
entry.resultPromise = resultPromise;
@@ -116,7 +135,7 @@ export class MDBBackend implements mcpServer.ServerBackend {
private async _pushClient(transport: Transport, introMessage?: string): Promise<mcpServer.CallToolResult> {
mdbDebug('pushing client to the stack');
- const client = new mcpBundle.Client({ name: 'Pushing client', version: '0.0.0' }, { capabilities: { roots: {} } });
+ const client = new mcpBundle.Client({ name: 'Interrupting client', version: '0.0.0' }, { capabilities: { roots: {} } });
client.setRequestHandler(mcpBundle.ListRootsRequestSchema, () => ({ roots: this._clientInfo?.roots || [] }));
client.setRequestHandler(mcpBundle.PingRequestSchema, () => ({}));
client.setNotificationHandler(mcpBundle.ProgressNotificationSchema, notification => {
@@ -154,8 +173,8 @@ const pushToolsSchema = defineToolSchema({
type: 'readOnly',
});
-export async function runMainBackend(backendFactory: mcpServer.ServerBackendFactory, options?: { port?: number }): Promise<string | undefined> {
- const mdbBackend = new MDBBackend(backendFactory.create());
+export async function runMainBackend(backendFactory: mcpServer.ServerBackendFactory, allowedOnPause: string[], options?: { port?: number }): Promise<string | undefined> {
+ const mdbBackend = new MDBBackend(backendFactory.create(), allowedOnPause);
// Start HTTP unconditionally.
const factory: mcpServer.ServerBackendFactory = {
...backendFactory,
@@ -185,7 +204,7 @@ export async function runOnPauseBackendLoop(backend: mcpServer.ServerBackend, in
await mcpHttp.installHttpTransport(httpServer, factory);
const url = mcpHttp.httpAddressToString(httpServer.address());
- const client = new mcpBundle.Client({ name: 'On-pause client', version: '0.0.0' });
+ const client = new mcpBundle.Client({ name: 'Pushing client', version: '0.0.0' });
client.setRequestHandler(mcpBundle.PingRequestSchema, () => ({}));
const transport = new mcpBundle.StreamableHTTPClientTransport(new URL(process.env.PLAYWRIGHT_DEBUGGER_MCP!));
await client.connect(transport);
diff --git a/packages/playwright/src/mcp/test/generatorTools.ts b/packages/playwright/src/mcp/test/generatorTools.ts
new file mode 100644
index 0000000000..cf2d0a52ea
--- /dev/null
+++ b/packages/playwright/src/mcp/test/generatorTools.ts
@@ -0,0 +1,123 @@
+/**
+ * Copyright (c) Microsoft Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import fs from 'fs';
+import path from 'path';
+
+import { z } from '../sdk/bundle';
+import { defineTestTool } from './testTool';
+import { GeneratorJournal } from './testContext';
+
+export const setupPage = defineTestTool({
+ schema: {
+ name: 'generator_setup_page',
+ title: 'Setup generator page',
+ description: 'Setup the page for test.',
+ inputSchema: z.object({
+ plan: z.string().describe('The plan for the test. This should be the actual test plan with all the steps.'),
+ project: z.string().optional().describe('Project to use for setup. For example: "chromium", if no project is provided uses the first project in the config.'),
+ seedFile: z.string().optional().describe('A seed file contains a single test that is used to setup the page for testing, for example: "tests/seed.spec.ts". If no seed file is provided, a default seed file is created.'),
+ }),
+ type: 'readOnly',
+ },
+
+ handle: async (context, params, progress) => {
+ const seed = await context.getOrCreateSeedFile(params);
+ context.generatorJournal = new GeneratorJournal(context.rootPath, params.plan, seed);
+ await context.runSeedTest(seed.file, params.project, progress);
+ return { content: [] };
+ },
+});
+
+export const generatorLogStep = defineTestTool({
+ schema: {
+ name: 'generator_log_step',
+ title: 'Log a test step',
+ description: 'Log a successful test step.',
+ inputSchema: z.object({
+ title: z.string().describe('Title of the test step'),
+ code: z.string().describe('The code for the last step only, as a markdown code block'),
+ }),
+ type: 'readOnly',
+ },
+
+ handle: async (context, params) => {
+ if (!context.generatorJournal)
+ throw new Error(`Please setup page using "${setupPage.schema.name}" first.`);
+ context.generatorJournal.logStep(params.title, params.code);
+ return { content: [] };
+ },
+});
+
+export const generatorReadLog = defineTestTool({
+ schema: {
+ name: 'generator_read_log',
+ title: 'Retrieve test log',
+ description: 'Retrieve the performed test log',
+ inputSchema: z.object({}),
+ type: 'readOnly',
+ },
+
+ handle: async context => {
+ if (!context.generatorJournal)
+ throw new Error(`Please setup page using "${setupPage.schema.name}" first.`);
+ const result = context.generatorJournal.journal();
+ return { content: [{
+ type: 'text',
+ text: result,
+ }] };
+ },
+});
+
+export const generatorWriteTest = defineTestTool({
+ schema: {
+ name: 'generator_write_test',
+ title: 'Write test',
+ description: 'Write the generated test to the test file',
+ inputSchema: z.object({
+ fileName: z.string().describe('The file to write the test to'),
+ code: z.string().describe('The generated test code'),
+ }),
+ type: 'destructive',
+ },
+
+ handle: async (context, params) => {
+ if (!context.generatorJournal)
+ throw new Error(`Please setup page using "${setupPage.schema.name}" first.`);
+
+ const testRunner = context.existingTestRunner();
+ if (!testRunner)
+ throw new Error('No test runner found, please setup page and perform actions first.');
+ const config = await testRunner.loadConfig();
+
+ const dirs: string[] = [];
+ for (const project of config.projects) {
+ const testDir = path.relative(context.rootPath, project.project.testDir).replace(/\\/g, '/');
+ const fileName = params.fileName.replace(/\\/g, '/');
+ if (fileName.startsWith(testDir)) {
+ await fs.promises.writeFile(path.resolve(context.rootPath, fileName), params.code);
+ return {
+ content: [{
+ type: 'text',
+ text: `### Result\nTest written to ${params.fileName}`,
+ }]
+ };
+ }
+ dirs.push(testDir);
+ }
+ throw new Error(`Test file did not match any of the test dirs: ${dirs.join(', ')}`);
+ },
+});
diff --git a/packages/playwright/src/mcp/test/plannerTools.ts b/packages/playwright/src/mcp/test/plannerTools.ts
new file mode 100644
index 0000000000..5fc7d41e4d
--- /dev/null
+++ b/packages/playwright/src/mcp/test/plannerTools.ts
@@ -0,0 +1,37 @@
+/**
+ * Copyright (c) Microsoft Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { z } from '../sdk/bundle';
+import { defineTestTool } from './testTool';
+
+export const setupPage = defineTestTool({
+ schema: {
+ name: 'planner_setup_page',
+ title: 'Setup planner page',
+ description: 'Setup the page for test planning',
+ inputSchema: z.object({
+ project: z.string().optional().describe('Project to use for setup. For example: "chromium", if no project is provided uses the first project in the config.'),
+ seedFile: z.string().optional().describe('A seed file contains a single test that is used to setup the page for testing, for example: "tests/seed.spec.ts". If no seed file is provided, a default seed file is created.'),
+ }),
+ type: 'readOnly',
+ },
+
+ handle: async (context, params, progress) => {
+ const seed = await context.getOrCreateSeedFile(params);
+ await context.runSeedTest(seed.file, params.project, progress);
+ return { content: [] };
+ },
+});
diff --git a/packages/playwright/src/mcp/test/testBackend.ts b/packages/playwright/src/mcp/test/testBackend.ts
index a1c64052e0..ade9e83d60 100644
--- a/packages/playwright/src/mcp/test/testBackend.ts
+++ b/packages/playwright/src/mcp/test/testBackend.ts
@@ -16,7 +16,9 @@
import * as mcp from '../sdk/exports';
import { TestContext } from './testContext';
-import { listTests, runTests, debugTest, setupPage } from './testTools.js';
+import * as testTools from './testTools.js';
+import * as generatorTools from './generatorTools.js';
+import * as plannerTools from './plannerTools.js';
import { browserTools } from '../browser/tools';
import { resolveConfigLocation } from '../../common/configLoader';
@@ -25,10 +27,23 @@ import type { TestTool } from './testTool';
export class TestServerBackend implements mcp.ServerBackend {
readonly name = 'Playwright';
readonly version = '0.0.1';
- private _tools: TestTool<any>[] = [listTests, runTests, debugTest, setupPage];
+ private _tools: TestTool<any>[] = [
+ plannerTools.setupPage,
+ generatorTools.setupPage,
+ generatorTools.generatorLogStep,
+ generatorTools.generatorReadLog,
+ generatorTools.generatorWriteTest,
+ testTools.listTests,
+ testTools.runTests,
+ testTools.debugTest,
+ ];
private _context: TestContext;
private _configOption: string | undefined;
+ static allowedOnPause: string[] = [
+ generatorTools.generatorLogStep.schema.name,
+ ];
+
constructor(configOption: string | undefined, options?: { muteConsole?: boolean, headless?: boolean }) {
this._context = new TestContext(options);
this._configOption = configOption;
diff --git a/packages/playwright/src/mcp/test/testContext.ts b/packages/playwright/src/mcp/test/testContext.ts
index 08904527ca..d8d4b27192 100644
--- a/packages/playwright/src/mcp/test/testContext.ts
+++ b/packages/playwright/src/mcp/test/testContext.ts
@@ -14,15 +14,63 @@
* limitations under the License.
*/
+import fs from 'fs';
+import path from 'path';
+
+import { noColors, escapeRegExp } from 'playwright-core/lib/utils';
+
+import { terminalScreen } from '../../reporters/base';
+import ListReporter from '../../reporters/list';
+import { StringWriteStream } from './streams';
+import { fileExistsAsync } from '../../util';
import { TestRunner, TestRunnerEvent } from '../../runner/testRunner';
+import { ensureSeedTest, seedProject } from './seed';
+import type { ProgressCallback } from '../sdk/server';
import type { ConfigLocation } from '../../common/config';
+export type SeedFile = {
+ file: string;
+ content: string;
+};
+
+export class GeneratorJournal {
+ private _rootPath: string;
+ private _plan: string;
+ private _seed: SeedFile;
+ private _steps: { title: string, code: string }[];
+
+ constructor(rootPath: string, plan: string, seed: SeedFile) {
+ this._rootPath = rootPath;
+ this._plan = plan;
+ this._seed = seed;
+ this._steps = [];
+ }
+
+ logStep(title: string, code: string) {
+ this._steps.push({ title, code });
+ }
+
+ journal() {
+ const result: string[] = [];
+ result.push(`# Plan`);
+ result.push(this._plan);
+ result.push(`# Seed file: ${path.relative(this._rootPath, this._seed.file)}`);
+ result.push('```ts');
+ result.push(this._seed.content);
+ result.push('```');
+ result.push(`# Steps`);
+ result.push(this._steps.map(step => `### ${step.title}\n${step.code}`).join('\n\n'));
+ return result.join('\n\n');
+ }
+}
+
export class TestContext {
private _testRunner: TestRunner | undefined;
readonly options?: { muteConsole?: boolean, headless?: boolean };
configLocation!: ConfigLocation;
rootPath!: string;
+ generatorJournal: GeneratorJournal | undefined;
constructor(options?: { muteConsole?: boolean, headless?: boolean }) {
this.options = options;
@@ -33,6 +81,10 @@ export class TestContext {
this.rootPath = rootPath || configLocation.configDir;
}
+ existingTestRunner(): TestRunner | undefined {
+ return this._testRunner;
+ }
+
async createTestRunner(): Promise<TestRunner> {
if (this._testRunner)
await this._testRunner.stopTests();
@@ -46,6 +98,72 @@ export class TestContext {
return testRunner;
}
+ async getOrCreateSeedFile(params: { project?: string, seedFile?: string }) {
+ const configDir = this.configLocation.configDir;
+ const testRunner = await this.createTestRunner();
+ const config = await testRunner.loadConfig();
+
+ let seedFile: string | undefined;
+ if (!params.seedFile) {
+ seedFile = await ensureSeedTest(config, params.project, false);
+ } else {
+ const candidateFiles: string[] = [];
+ const testDir = seedProject(config, params.project).project.testDir;
+ candidateFiles.push(path.resolve(testDir, params.seedFile));
+ candidateFiles.push(path.resolve(configDir, params.seedFile));
+ candidateFiles.push(path.resolve(this.rootPath, params.seedFile));
+ for (const candidateFile of candidateFiles) {
+ if (await fileExistsAsync(candidateFile)) {
+ seedFile = candidateFile;
+ break;
+ }
+ }
+ if (!seedFile)
+ throw new Error('seed test not found.');
+ }
+
+ const seedFileContent = await fs.promises.readFile(seedFile, 'utf8');
+ return { file: seedFile, content: seedFileContent };
+ }
+
+ async runSeedTest(seedFile: string, projectName: string | undefined, progress: ProgressCallback) {
+ const { screen } = this.createScreen(progress);
+ const configDir = this.configLocation.configDir;
+ const reporter = new ListReporter({ configDir, screen });
+ const testRunner = await this.createTestRunner();
+
+ const result = await testRunner.runTests(reporter, {
+ headed: !this.options?.headless,
+ locations: ['/' + escapeRegExp(seedFile) + '/'],
+ projects: projectName ? [projectName] : undefined,
+ timeout: 0,
+ workers: 1,
+ pauseAtEnd: true,
+ disableConfigReporters: true,
+ failOnLoadErrors: true,
+ });
+
+ // Ideally, we should check that page was indeed created and browser mcp has kicked in.
+ // However, that is handled in the upper layer, so hard to check here.
+ if (result.status === 'passed' && !reporter.suite?.allTests().length)
+ throw new Error('seed test not found.');
+
+ if (result.status !== 'passed')
+ throw new Error('Errors while running the seed test.');
+ }
+
+ createScreen(progress: ProgressCallback) {
+ const stream = new StringWriteStream(progress);
+ const screen = {
+ ...terminalScreen,
+ isTTY: false,
+ colors: noColors,
+ stdout: stream as unknown as NodeJS.WriteStream,
+ stderr: stream as unknown as NodeJS.WriteStream,
+ };
+ return { screen, stream };
+ }
+
async close() {
}
}
diff --git a/packages/playwright/src/mcp/test/testTools.ts b/packages/playwright/src/mcp/test/testTools.ts
index 01cd9a1cb4..a3c86ed080 100644
--- a/packages/playwright/src/mcp/test/testTools.ts
+++ b/packages/playwright/src/mcp/test/testTools.ts
@@ -14,21 +14,10 @@
* limitations under the License.
*/
-import fs from 'fs';
-import path from 'path';
-
-import { noColors, escapeRegExp } from 'playwright-core/lib/utils';
-
import { z } from '../sdk/bundle';
-import { terminalScreen } from '../../reporters/base';
import ListReporter from '../../reporters/list';
import ListModeReporter from '../../reporters/listModeReporter';
-import { ensureSeedTest, seedProject } from './seed';
import { defineTestTool } from './testTool';
-import { StringWriteStream } from './streams';
-import { fileExistsAsync } from '../../util';
-
-import type { ProgressCallback } from '../sdk/server';
export const listTests = defineTestTool({
schema: {
@@ -40,7 +29,7 @@ export const listTests = defineTestTool({
},
handle: async (context, _, progress) => {
- const { screen } = createScreen(progress);
+ const { screen } = context.createScreen(progress);
const reporter = new ListModeReporter({ screen, includeTestId: true });
const testRunner = await context.createTestRunner();
await testRunner.listTests(reporter, {});
@@ -62,7 +51,7 @@ export const runTests = defineTestTool({
},
handle: async (context, params, progress) => {
- const { screen } = createScreen(progress);
+ const { screen } = context.createScreen(progress);
const configDir = context.configLocation.configDir;
const reporter = new ListReporter({ configDir, screen, includeTestId: true, prefixStdio: 'out' });
const testRunner = await context.createTestRunner();
@@ -91,7 +80,7 @@ export const debugTest = defineTestTool({
},
handle: async (context, params, progress) => {
- const { screen } = createScreen(progress);
+ const { screen } = context.createScreen(progress);
const configDir = context.configLocation.configDir;
const reporter = new ListReporter({ configDir, screen, includeTestId: true, prefixStdio: 'out' });
const testRunner = await context.createTestRunner();
@@ -108,85 +97,3 @@ export const debugTest = defineTestTool({
return { content: [] };
},
});
-
-export const setupPage = defineTestTool({
- schema: {
- name: 'test_setup_page',
- title: 'Setup page',
- description: 'Setup the page for test.',
- inputSchema: z.object({
- project: z.string().optional().describe('Project to use for setup. For example: "chromium", if no project is provided uses the first project in the config.'),
- seedFile: z.string().optional().describe('A seed file contains a single test that is used to setup the page for testing, for example: "tests/seed.spec.ts". If no seed file is provided, a default seed file is created.'),
- }),
- type: 'readOnly',
- },
-
- handle: async (context, params, progress) => {
- const { screen } = createScreen(progress);
- const configDir = context.configLocation.configDir;
- const reporter = new ListReporter({ configDir, screen });
- const testRunner = await context.createTestRunner();
- const config = await testRunner.loadConfig();
-
- let seedFile: string | undefined;
- if (!params.seedFile) {
- seedFile = await ensureSeedTest(config, params.project, false);
- } else {
- const candidateFiles: string[] = [];
- const testDir = seedProject(config, params.project).project.testDir;
- candidateFiles.push(path.resolve(testDir, params.seedFile));
- candidateFiles.push(path.resolve(configDir, params.seedFile));
- candidateFiles.push(path.resolve(context.rootPath, params.seedFile));
- for (const candidateFile of candidateFiles) {
- if (await fileExistsAsync(candidateFile)) {
- seedFile = candidateFile;
- break;
- }
- }
- if (!seedFile)
- throw new Error('seed test not found.');
- }
-
-
- const seedFileContent = await fs.promises.readFile(seedFile, 'utf8');
- progress({ message: `### Seed test
-File: ${path.relative(context.rootPath, seedFile)}
-\`\`\`ts
-${seedFileContent}
-\`\`\`
-` });
-
- const result = await testRunner.runTests(reporter, {
- headed: !context.options?.headless,
- locations: ['/' + escapeRegExp(seedFile) + '/'],
- projects: params.project ? [params.project] : undefined,
- timeout: 0,
- workers: 1,
- pauseAtEnd: true,
- disableConfigReporters: true,
- failOnLoadErrors: true,
- });
-
- // Ideally, we should check that page was indeed created and browser mcp has kicked in.
- // However, that is handled in the upper layer, so hard to check here.
- if (result.status === 'passed' && !reporter.suite?.allTests().length)
- throw new Error('seed test not found.');
-
- if (result.status !== 'passed')
- throw new Error('Errors while running the seed test.');
-
- return { content: [] };
- },
-});
-
-function createScreen(progress: ProgressCallback) {
- const stream = new StringWriteStream(progress);
- const screen = {
- ...terminalScreen,
- isTTY: false,
- colors: noColors,
- stdout: stream as unknown as NodeJS.WriteStream,
- stderr: stream as unknown as NodeJS.WriteStream,
- };
- return { screen, stream };
-}
diff --git a/packages/playwright/src/program.ts b/packages/playwright/src/program.ts
index 79d93a8ce1..82857da2ef 100644
--- a/packages/playwright/src/program.ts
+++ b/packages/playwright/src/program.ts
@@ -169,7 +169,13 @@ function addTestMCPServerCommand(program: Command) {
version: packageJSON.version,
create: () => new TestServerBackend(options.config, { muteConsole: options.port === undefined, headless: options.headless }),
};
- const mdbUrl = await runMainBackend(backendFactory, { port: options.port === undefined ? undefined : +options.port });
+ const mdbUrl = await runMainBackend(
+ backendFactory,
+ TestServerBackend.allowedOnPause,
+ {
+ port: options.port === undefined ? undefined : +options.port
+ },
+ );
if (mdbUrl)
console.error('MCP Listening on: ', mdbUrl);
});
diff --git a/tests/mcp/generator.spec.ts b/tests/mcp/generator.spec.ts
new file mode 100644
index 0000000000..722ce96258
--- /dev/null
+++ b/tests/mcp/generator.spec.ts
@@ -0,0 +1,248 @@
+/**
+ * Copyright (c) Microsoft Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import fs from 'fs';
+
+import { test, expect, writeFiles } from './fixtures';
+
+test.use({ mcpServerType: 'test-mcp' });
+
+test('generator_setup_page', async ({ startClient }) => {
+ await writeFiles({
+ 'a.test.ts': `
+ import { test, expect } from '@playwright/test';
+ test.beforeEach(async ({ page }) => {
+ await page.setContent('<button>Submit</button>');
+ });
+ test('template', async ({ page }) => {
+ });
+ `,
+ });
+
+ const { client } = await startClient();
+ const response = await client.callTool({
+ name: 'generator_setup_page',
+ arguments: {
+ plan: 'Test plan',
+ seedFile: 'a.test.ts',
+ },
+ });
+
+ expect(response).toHaveTextResponse(expect.stringContaining(`### Paused at end of test. ready for interaction
+
+### Page state
+- Page URL: about:blank
+- Page Title:
+- Page Snapshot:
+\`\`\`yaml
+- button "Submit" [ref=e2]
+\`\`\`
+`));
+
+ await client.callTool({
+ name: 'generator_log_step',
+ arguments: {
+ title: 'Click submit button',
+ code: `\`\`\`ts
+await page.getByRole('button', { name: 'Submit' }).click();
+\`\`\``,
+ },
+ });
+
+ expect(await client.callTool({
+ name: 'generator_read_log',
+ arguments: {},
+ })).toHaveTextResponse(`# Plan
+
+Test plan
+
+# Seed file: a.test.ts
+
+\`\`\`ts
+
+
+ import { test, expect } from '@playwright/test';
+ test.beforeEach(async ({ page }) => {
+ await page.setContent('<button>Submit</button>');
+ });
+ test('template', async ({ page }) => {
+ });
+
+
+\`\`\`
+
+# Steps
+
+### Click submit button
+\`\`\`ts
+await page.getByRole('button', { name: 'Submit' }).click();
+\`\`\``);
+});
+
+test('click after generator_log_action', async ({ startClient }) => {
+ await writeFiles({
+ 'a.test.ts': `
+ import { test, expect } from '@playwright/test';
+ test.beforeEach(async ({ page }) => {
+ await page.setContent('<button>Submit</button>');
+ });
+ test('template', async ({ page }) => {
+ });
+ `,
+ });
+
+ const { client } = await startClient();
+ await client.callTool({
+ name: 'generator_setup_page',
+ arguments: {
+ plan: 'Test plan',
+ seedFile: 'a.test.ts',
+ },
+ });
+
+ await client.callTool({
+ name: 'generator_log_step',
+ arguments: {
+ title: 'Click submit button',
+ code: `\`\`\`ts
+await page.getByRole('button', { name: 'Submit' }).click();
+\`\`\``,
+ },
+ });
+
+ expect(await client.callTool({
+ name: 'browser_click',
+ arguments: {
+ element: 'Submit button',
+ ref: 'e2',
+ },
+ })).toHaveResponse({
+ code: `await page.getByRole('button', { name: 'Submit' }).click();`,
+ pageState: expect.stringContaining(`- button "Submit"`),
+ });
+
+ expect(await client.callTool({
+ name: 'generator_read_log',
+ arguments: {},
+ })).toHaveTextResponse(`# Plan
+
+Test plan
+
+# Seed file: a.test.ts
+
+\`\`\`ts
+
+
+ import { test, expect } from '@playwright/test';
+ test.beforeEach(async ({ page }) => {
+ await page.setContent('<button>Submit</button>');
+ });
+ test('template', async ({ page }) => {
+ });
+
+
+\`\`\`
+
+# Steps
+
+### Click submit button
+\`\`\`ts
+await page.getByRole('button', { name: 'Submit' }).click();
+\`\`\``);
+});
+
+test('generator_setup_page is required', async ({ startClient }) => {
+ await writeFiles({
+ 'a.test.ts': `
+ import { test, expect } from '@playwright/test';
+ test.beforeEach(async ({ page }) => {
+ await page.setContent('<button>Submit</button>');
+ });
+ test('template', async ({ page }) => {
+ });
+ `,
+ });
+
+ const { client } = await startClient();
+
+ expect(await client.callTool({
+ name: 'generator_log_step',
+ arguments: {
+ title: 'Click submit button',
+ code: `\`\`\`ts
+await page.getByRole('button', { name: 'Submit' }).click();
+\`\`\``,
+ },
+ })).toHaveResponse({
+ result: expect.stringContaining(`Please setup page using "generator_setup_page" first.`),
+ isError: true,
+ });
+
+ expect(await client.callTool({
+ name: 'generator_read_log',
+ arguments: {},
+ })).toHaveResponse({
+ result: expect.stringContaining(`Please setup page using "generator_setup_page" first.`),
+ isError: true,
+ });
+
+ expect(await client.callTool({
+ name: 'generator_write_test',
+ arguments: {
+ fileName: 'a.test.ts',
+ code: '// Test content',
+ },
+ })).toHaveResponse({
+ result: expect.stringContaining(`Please setup page using "generator_setup_page" first.`),
+ isError: true,
+ });
+});
+
+test('generator_write_test', async ({ startClient }, testInfo) => {
+ await writeFiles({
+ 'a.test.ts': `
+ import { test, expect } from '@playwright/test';
+ test.beforeEach(async ({ page }) => {
+ await page.setContent('<button>Submit</button>');
+ });
+ test('template', async ({ page }) => {
+ });
+ `,
+ });
+
+ const { client } = await startClient();
+
+ await client.callTool({
+ name: 'generator_setup_page',
+ arguments: {
+ plan: 'Test plan',
+ seedFile: 'a.test.ts',
+ },
+ });
+
+ expect(await client.callTool({
+ name: 'generator_write_test',
+ arguments: {
+ fileName: 'a.test.ts',
+ code: `// Test content`,
+ },
+ })).toHaveResponse({
+ result: expect.stringContaining(`Test written to a.test.ts`),
+ });
+
+ const code = fs.readFileSync(testInfo.outputPath('a.test.ts'), 'utf8');
+ expect(code).toBe(`// Test content`);
+});
diff --git a/tests/mcp/mdb.spec.ts b/tests/mcp/mdb.spec.ts
index 1c83725ab8..e51046f6de 100644
--- a/tests/mcp/mdb.spec.ts
+++ b/tests/mcp/mdb.spec.ts
@@ -131,7 +131,7 @@ async function startMDBAndCLI(): Promise<{ mdbUrl: string }> {
create: () => new CLIBackend(mdbUrlBox)
};
- const mdbUrl = (await runMainBackend(cliBackendFactory, { port: 0 }))!;
+ const mdbUrl = (await runMainBackend(cliBackendFactory, [], { port: 0 }))!;
mdbUrlBox.mdbUrl = mdbUrl;
return { mdbUrl };
}
diff --git a/tests/mcp/test-setup-page.spec.ts b/tests/mcp/planner.spec.ts
similarity index 86%
rename from tests/mcp/test-setup-page.spec.ts
rename to tests/mcp/planner.spec.ts
index 6afcd9f5dd..8268894212 100644
--- a/tests/mcp/test-setup-page.spec.ts
+++ b/tests/mcp/planner.spec.ts
@@ -22,7 +22,7 @@ import url from 'url';
test.use({ mcpServerType: 'test-mcp' });
-test('test_setup_page', async ({ startClient }) => {
+test('planner_setup_page', async ({ startClient }) => {
await writeFiles({
'a.test.ts': `
import { test, expect } from '@playwright/test';
@@ -36,13 +36,12 @@ test('test_setup_page', async ({ startClient }) => {
const { client } = await startClient();
const response = await client.callTool({
- name: 'test_setup_page',
+ name: 'planner_setup_page',
arguments: {
seedFile: 'a.test.ts',
},
});
- expect(response).toHaveTextResponse(expect.stringContaining(`### Seed test\nFile: a.test.ts`));
expect(response).toHaveTextResponse(expect.stringContaining(`### Paused at end of test. ready for interaction
### Page state
@@ -66,7 +65,7 @@ test('test_setup_page', async ({ startClient }) => {
});
});
-test('test_setup_page seed resolution', async ({ startClient }) => {
+test('planner_setup_page seed resolution', async ({ startClient }) => {
await writeFiles({
'playwright.config.ts': `
module.exports = {
@@ -85,7 +84,7 @@ test('test_setup_page seed resolution', async ({ startClient }) => {
// Relative to test dir.
expect(await client.callTool({
- name: 'test_setup_page',
+ name: 'planner_setup_page',
arguments: {
seedFile: 'seed.test.ts',
},
@@ -93,14 +92,14 @@ test('test_setup_page seed resolution', async ({ startClient }) => {
// Relative to config dir.
expect(await client.callTool({
- name: 'test_setup_page',
+ name: 'planner_setup_page',
arguments: {
seedFile: 'tests/seed.test.ts',
},
})).toHaveTextResponse(expect.stringContaining(`### Paused at end of test.`));
});
-test('test_setup_page seed resolution - rootPath', async ({ startClient }) => {
+test('planner_setup_page seed resolution - rootPath', async ({ startClient }) => {
await writeFiles({
'packages/my-app/configs/playwright.config.ts': `
module.exports = {
@@ -121,14 +120,14 @@ test('test_setup_page seed resolution - rootPath', async ({ startClient }) => {
});
expect(await client.callTool({
- name: 'test_setup_page',
+ name: 'planner_setup_page',
arguments: {
seedFile: 'packages/my-app/tests/seed.test.ts',
},
})).toHaveTextResponse(expect.stringContaining(`### Paused at end of test.`));
});
-test('test_setup_page with dependencies', async ({ startClient }) => {
+test('planner_setup_page with dependencies', async ({ startClient }) => {
const baseDir = await writeFiles({
'playwright.config.ts': `
module.exports = {
@@ -155,7 +154,7 @@ test('test_setup_page with dependencies', async ({ startClient }) => {
const { client } = await startClient();
expect(await client.callTool({
- name: 'test_setup_page',
+ name: 'planner_setup_page',
arguments: {
seedFile: 'seed.test.ts',
project: 'chromium',
@@ -168,7 +167,7 @@ test('test_setup_page with dependencies', async ({ startClient }) => {
expect(fs.existsSync(path.join(baseDir, 'test-results', 'seed-template-ignored', 'template.txt'))).toBe(false);
});
-test('test_setup_page (loading error)', async ({ startClient }) => {
+test('planner_setup_page (loading error)', async ({ startClient }) => {
await writeFiles({
'seed.test.ts': `
throw new Error('loading error');
@@ -176,17 +175,17 @@ test('test_setup_page (loading error)', async ({ startClient }) => {
});
const { client } = await startClient();
expect(await client.callTool({
- name: 'test_setup_page',
+ name: 'planner_setup_page',
arguments: {
seedFile: 'seed.test.ts',
},
})).toHaveTextResponse(expect.stringContaining('Error: loading error'));
});
-test('test_setup_page (wrong test location)', async ({ startClient }) => {
+test('planner_setup_page (wrong test location)', async ({ startClient }) => {
const { client } = await startClient();
expect(await client.callTool({
- name: 'test_setup_page',
+ name: 'planner_setup_page',
arguments: {
seedFile: 'a.test.ts',
},
@@ -194,15 +193,15 @@ test('test_setup_page (wrong test location)', async ({ startClient }) => {
Error: seed test not found.`);
});
-test('test_setup_page (no test location)', async ({ startClient }) => {
+test('planner_setup_page (no test location)', async ({ startClient }) => {
const { client } = await startClient();
expect(await client.callTool({
- name: 'test_setup_page',
+ name: 'planner_setup_page',
arguments: {},
})).toHaveTextResponse(expect.stringContaining(`### Paused at end of test. ready for interaction`));
});
-test('test_setup_page chooses top-level project', async ({ startClient }) => {
+test('planner_setup_page chooses top-level project', async ({ startClient }) => {
const baseDir = await writeFiles({
'playwright.config.ts': `
module.exports = {
@@ -216,7 +215,7 @@ test('test_setup_page chooses top-level project', async ({ startClient }) => {
const { client } = await startClient();
expect(await client.callTool({
- name: 'test_setup_page',
+ name: 'planner_setup_page',
arguments: {},
})).toHaveTextResponse(expect.stringContaining(`### Paused at end of test. ready for interaction`));
@@ -224,7 +223,7 @@ test('test_setup_page chooses top-level project', async ({ startClient }) => {
expect(fs.existsSync(path.join(baseDir, 'two', 'seed.spec.ts'))).toBe(true);
});
-test('test_setup_page without location respects testsDir', async ({ startClient }) => {
+test('planner_setup_page without location respects testsDir', async ({ startClient }) => {
await writeFiles({
'playwright.config.ts': `
module.exports = {
@@ -242,7 +241,7 @@ test('test_setup_page without location respects testsDir', async ({ startClient
const { client } = await startClient();
expect(await client.callTool({
- name: 'test_setup_page',
+ name: 'planner_setup_page',
arguments: {},
})).toHaveTextResponse(expect.stringContaining(`### Paused at end of test. ready for interaction`));
diff --git a/tests/mcp/test-debug.spec.ts b/tests/mcp/test-debug.spec.ts
index 0c52d06e31..0c3b031ef2 100644
--- a/tests/mcp/test-debug.spec.ts
+++ b/tests/mcp/test-debug.spec.ts
@@ -229,6 +229,44 @@ test('test_debug / evaluate', async ({ startClient }) => {
});
});
+test('test_debug / evaluate x 2', async ({ startClient }) => {
+ const { client, id } = await prepareDebugTest(startClient);
+ await client.callTool({
+ name: 'test_debug',
+ arguments: {
+ test: { id, title: 'fail' },
+ },
+ });
+ expect(await client.callTool({
+ name: 'browser_evaluate',
+ arguments: {
+ function: '() => 21+21',
+ },
+ })).toHaveResponse({
+ result: `42`,
+ });
+
+ expect(await client.callTool({
+ name: 'test_debug',
+ arguments: {
+ test: { id, title: 'fail' },
+ },
+ })).toEqual({
+ content: [
+ { type: 'text', text: expect.stringContaining(`Paused on error`) },
+ ]
+ });
+
+ expect(await client.callTool({
+ name: 'browser_evaluate',
+ arguments: {
+ function: '() => 21+23',
+ },
+ })).toHaveResponse({
+ result: `44`,
+ });
+});
+
test('test_debug / evaluate (with element)', async ({ startClient }) => {
const { client, id } = await prepareDebugTest(startClient);
await client.callTool({