Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
Skip DataDragon ledger DDL when the schema is current so concurrent runs stop deadlocking (SKY-18309) (#8920)
Shuchang Zheng010331f5985b
diff --git a/.env.example b/.env.exampleindex 256e99f60..40d7bf77f 100644--- a/.env.example+++ b/.env.example@@ -167,13 +167,6 @@ PORT=8000# ANALYTICS_ID: Distinct analytics ID (a UUID is generated if left blank).ANALYTICS_ID="anonymous"-# LAMINAR-# Skyvern's backend runs on port 8000 by default. Consider updating your self-hosted laminar to env vars to avoid conflicts-# LMNR_HTTP_PORT=8010-# LMNR_GRPC_PORT=8011-# LMNR_BASE_URL=http://localhost-# LMNR_PROJECT_API_KEY=<your-laminar-project-api-key>-# 1Password Integration# OP_SERVICE_ACCOUNT_TOKEN: API token for 1Password integrationOP_SERVICE_ACCOUNT_TOKEN=""diff --git a/docs/browsers.mdx b/docs/browsers.mdxindex cb7eab3b9..45a0b8b12 100644--- a/docs/browsers.mdx+++ b/docs/browsers.mdx@@ -134,7 +134,7 @@ Use a session when you need a browser right now. Use a profile when you want to| Parameter | Values ||---|---|| `timeout` | Minutes, 5 to 240. Default 60. Larger values are capped at 240. |- | `proxy_location` | A country, such as `RESIDENTIAL` (US). See the [API reference](/api-reference) for the list. |+ | `proxy_location` | A country, such as `RESIDENTIAL` (US). See [Proxies and geo-targeting](/developers/production#proxies-and-geo-targeting) for the list. || `browser_type` | `chrome`, `msedge` or `stealth-chromium`. If you omit it, Skyvern picks a managed default that may not be Chrome. || `extensions` | `ad-blocker`, `captcha-solver` (Cloud only) || `browser_profile_id` | Start from a saved [profile](#browser-profiles) |diff --git a/docs/cloud/account.mdx b/docs/cloud/account.mdxindex 8ee07754f..cb6f4f41f 100644--- a/docs/cloud/account.mdx+++ b/docs/cloud/account.mdx@@ -257,18 +257,18 @@ Specific prompts use fewer actions. "Extract the monthly price from the pricing### Plans-| Plan | Price | Credits | About | Concurrent runs |-|---|---|---|---|---|-| **Free** | $0 | 5,000, one time | 165 actions | 1 |-| **Hobby** | $29/month | 30,000/month | 1,000 actions | 10 |-| **Pro** | $149/month | 150,000/month | 5,000 actions | 25 |-| **Enterprise** | Custom | Unlimited | Unlimited | Unlimited |+| Plan | Price | Credits | Concurrent runs |+|---|---|---|---|+| **Free** | $0 | 5,000 one time (about 165 actions) | 1 |+| **Hobby** | $29/month | 30,000/month (about 1,000 actions) | 10 |+| **Pro** | $149/month | 150,000/month (about 5,000 actions) | 25 |+| **Enterprise** | Custom | Unlimited | Unlimited |Free credits are a one-time grant and don't reset. Click **Subscribe** on the Hobby or Pro card to start a plan. Once you're subscribed, the cards read **Upgrade** or **Downgrade** and the page shows your current plan, with **Manage Subscription** to change or cancel it. For Enterprise, click **Book a Demo**. For a full feature comparison, see [skyvern.com/pricing](https://www.skyvern.com/pricing).### Extra credits and payment-On a paid plan, **Buy Extra Credits** lets you top up with a one-time purchase ($10, $25, $50, $100 or a custom amount). Some plans also offer monthly add-on credit packs under **Add more credits**. Click **Manage Payment Methods** to update your card, see invoices and change billing details.+On a paid plan, **Buy Extra Credits** lets you top up with a one-time purchase (\$10, \$25, \$50, \$100 or a custom amount). Some plans also offer monthly add-on credit packs under **Add more credits**. Click **Manage Payment Methods** to update your card, see invoices and change billing details.### Runs refused for credits@@ -277,7 +277,7 @@ Skyvern checks credits before it starts a run and returns `402` if they don't co| Message | Why | Fix ||---|---|---|| `Credits exhausted. Upgrade your plan in Billing.` | No credits left this period | Buy extra credits or upgrade |-| `Insufficient credit balance` | Older accounts without a plan. A run needs $0.10 per step of **Max Steps Per Run** ($1.00 when it's blank), and charges on your upcoming invoice count against your balance, so you can be refused while still showing credit. | Add credit, or lower **Max Steps Per Run** |+| `Insufficient credit balance` | Older accounts without a plan. A run needs \$0.10 per step of **Max Steps Per Run** (\$1.00 when it's blank), and charges on your upcoming invoice count against your balance, so you can be refused while still showing credit. | Add credit, or lower **Max Steps Per Run** || `Not enough credit balance.` | The balance is too low at run start, at a Code block, at a cached action, at an AI step, or on an SDK browser action | Buy extra credits or upgrade |## Membersdiff --git a/docs/cloud/integrations.mdx b/docs/cloud/integrations.mdxindex f501499b1..363cee311 100644--- a/docs/cloud/integrations.mdx+++ b/docs/cloud/integrations.mdx@@ -28,7 +28,7 @@ Every integration with a **Connect**, **Configure** or **Set up →** button is|---|---|---|---|| Google Sheets | Workspace | Read and write sheets in your agents | **Connect** || Google Drive | Workspace | Upload run files to Drive folders | **Connect** |-| Gmail | Communication | Find verification codes and emails in your inbox | **Connect** |+| Gmail | Communication | Read verification codes and send email from your agents | **Connect** || Outlook | Communication | Find verification codes and emails in your inbox | **Connect** || 1Password | Credentials | Use logins from your 1Password vaults | **Configure** || Bitwarden | Credentials | Use logins from your Bitwarden vault | **Configure** |@@ -49,7 +49,7 @@ Google Sheets, Google Drive, Gmail and Outlook connect through Google or MicrosoClick **Connect** on the tile.</Step><Step title="Name the connection">- Enter a name, such as "Finance Gmail". You pick the account by this name in your agent's blocks. Click **Continue to Google** (or **Continue to Microsoft** for Outlook).+ Enter a name, such as "Finance Gmail". You pick the account by this name in your agent's blocks. For Gmail, choose what agents may do: **Read inbox** (on by default) and **Send email**. Click **Continue to Google** (or **Continue to Microsoft** for Outlook).</Step><Step title="Approve access">Sign in and approve every permission on the consent screen. Skyvern sends you back to Integrations, where the account appears under **Connected**.@@ -62,7 +62,7 @@ What each connection asks for, and where you use it:|---|---|---|| Google Sheets | Read and write spreadsheets, read Drive file metadata | **Google Sheets Read** and **Google Sheets Write** blocks || Google Drive | Full Drive access | **Cloud Storage** block with storage type **Google Drive**; **File Download** block for Drive links |-| Gmail | Read-only mail | **Email Inbox** block, and email-based 2FA on [credentials](/credentials) |+| Gmail | **Read inbox**: read-only mail. **Send email**: send mail as the account. | **Email Inbox** block and email-based 2FA on [credentials](/credentials) (read); **Send Email** block with **Send with** set to the account (send) || Outlook | Read mail (`Mail.Read`) | **Email Inbox** block, and email-based 2FA on [credentials](/credentials) |Block settings are in the [blocks reference](/reference/blocks).@@ -73,6 +73,7 @@ Each connected card lists its accounts. From there you can:- **Add another account**: click **Add another Gmail account** (or the matching name) to connect a second inbox or Google account.- **Rename** or **Disconnect**: open the **...** menu on the account row. The name you set is what agent blocks show in their account picker.+- **Turn on reading or sending** (Gmail): the account row shows **Read inbox** and **Send email**. Click the one that isn't on yet and approve the extra permission. Sending needs the account's address, so an older connection may ask you to reconnect first.- **Reconnect** (Google only): if the card shows **Needs reconnect**, the authorization expired or you didn't approve every permission. Click **Reconnect** and approve again. Agents that use the account resume without edits.<Warning>diff --git a/docs/cloud/quickstart.mdx b/docs/cloud/quickstart.mdxindex 574197bcf..a943fd406 100644--- a/docs/cloud/quickstart.mdx+++ b/docs/cloud/quickstart.mdx@@ -46,7 +46,7 @@ You describe a task in plain words. Copilot turns it into an agent, tests it inCopilot stops to ask when it needs you:- **A question.** Pick an answer or type your own in the message box.- - **A login.** The **Copilot needs to sign in** card lets you pick a saved login under **Saved logins** or add one. See [Credentials and 2FA](/credentials).+ - **A login.** The **Copilot needs to sign in** card lets you pick a saved login under **Saved logins** or add one. See [Credentials and 2FA](/credentials). **Sign in myself (this test only)** lets you sign in yourself instead; that sign-in covers the current test, and later runs usually start signed out, so connect a credential for an agent you plan to run again.<img src="/images/cloud/cloud-copilot-building.png" alt="Studio with Copilot showing its plan and working status on the left and the Browser pane on Hacker News on the right" /></Step>diff --git a/docs/cloud/runs.mdx b/docs/cloud/runs.mdxindex a6e46808e..04438e73e 100644--- a/docs/cloud/runs.mdx+++ b/docs/cloud/runs.mdx@@ -86,7 +86,7 @@ The **Run** pane has three views, plus more under **⋯**:**API & Webhooks** in the Run pane header copies this run as a cURL or PowerShell command and offers **Test Webhook**.-When a run fails, the top of the Run pane names the failing block and the reason. **Fix with Copilot** sends the failure to Copilot, which proposes a fix in the Copilot pane. **Retry** opens the run form to run it again.+When a run fails, the top of the Run pane names the failing block, the run's error codes and the reason. Each code is listed once; hover a code to read its message. Past three codes the line ends with **+N more**, and **Outputs** → **Errors** lists every code. **Fix with Copilot** sends the failure to Copilot, which proposes a fix in the Copilot pane. **Retry** opens the run form to run it again.### Re-rundiff --git a/docs/cloud/studio.mdx b/docs/cloud/studio.mdxindex ad0704b1b..bf10ca119 100644--- a/docs/cloud/studio.mdx+++ b/docs/cloud/studio.mdx@@ -206,10 +206,13 @@ The **⋯** menu on a row holds:| Action | What it does ||---|---|+| **Open in editor** | Opens the agent in Studio |+| **Run** | Starts a run |+| **Move to folder** | Moves the agent into a folder, or out of one || **Duplicate Agent** | Creates a copy of the agent |-| **Save as Template** / **Remove from Templates** | Adds or removes the agent from your organization's templates |-| **Export as...** | Downloads the agent as YAML or JSON |-| **Delete Agent** | Deletes the agent after you confirm |+| **Save as template** / **Remove from template** | Adds or removes the agent from your organization's templates |+| **Export** | Downloads the agent as **YAML** or **JSON** |+| **Delete** | Deletes the agent after you confirm |**Import** uploads one or more agents from `.yaml`, `.yml`, `.json` or `.zip` files. A `.pdf` procedure (SOP) is turned into a new agent.diff --git a/docs/developers/browser-automation.mdx b/docs/developers/browser-automation.mdxindex aba1e5585..df07c8c0e 100644--- a/docs/developers/browser-automation.mdx+++ b/docs/developers/browser-automation.mdx@@ -65,7 +65,7 @@ The snippets on this page run inside an async function; the [worked example](#wo- `launch_cloud_browser` starts a new cloud session and logs a link to watch it live.- **`timeout`**: session length in minutes, from 5 to 240, default 60. Values above 240 are capped at 240.-- **`proxy_location`**: route traffic through a proxy, for example `"RESIDENTIAL"` (US) or `"RESIDENTIAL_GB"`. See [Browsers](/browsers) for locations.+- **`proxy_location`**: route traffic through a proxy, for example `"RESIDENTIAL"` (US) or `"RESIDENTIAL_GB"`. See [Proxies and geo-targeting](/developers/production#proxies-and-geo-targeting) for locations.- **`browser_profile_id`** (Python): start with a saved profile's cookies and storage, so you skip the login.- `browser.close()` closes the browser and ends the cloud session. Always call it in `finally`.diff --git a/docs/developers/getting-started/mcp.mdx b/docs/developers/getting-started/mcp.mdxindex b77b48f48..b365b40ec 100644--- a/docs/developers/getting-started/mcp.mdx+++ b/docs/developers/getting-started/mcp.mdx@@ -106,9 +106,9 @@ grok plugin install skyvern --trustOpen `/mcps` in Grok, select **skyvern**, and sign in with your Skyvern account in the browser.-The plugin connects the `browser` scope (`https://api.skyvern.com/mcp/x/browser`, 54 tools), so it has no `skyvern_run_task`, `skyvern_extract`, `skyvern_validate`, agent or credential tools. For every tool, change the `url` in the installed plugin's `.mcp.json` to `https://api.skyvern.com/mcp/`. See [Limit the tool list with scopes](#limit-the-tool-list-with-scopes).+The plugin connects the `browser` scope (`https://api.skyvern.com/mcp/x/browser`), so it has no `skyvern_run_task`, `skyvern_extract`, `skyvern_validate`, agent or credential tools. For every tool, change the `url` in the installed plugin's `.mcp.json` to `https://api.skyvern.com/mcp/`. A plugin update may undo that edit. See [Limit the tool list with scopes](#limit-the-tool-list-with-scopes).-Grok Build also loads MCP servers from `~/.claude.json` and `.mcp.json`. If one of them already defines a server named `skyvern`, Grok may use that one instead. Run `grok mcp doctor skyvern` to see which URL is in use, and rename or remove the other entry to use the plugin's.+Grok Build also loads MCP servers from `~/.claude.json`, `.cursor/mcp.json` and the project's `.mcp.json`. If one of them already defines a server named `skyvern`, Grok may use that one instead. Run `grok mcp doctor skyvern` to see which URL is in use, and rename or remove the other entry to use the plugin's.</Tab><Tab title="ChatGPT">diff --git a/docs/developers/production.mdx b/docs/developers/production.mdxindex eb504d21b..198326414 100644--- a/docs/developers/production.mdx+++ b/docs/developers/production.mdx@@ -14,7 +14,6 @@ keywords:- GeoTarget- max_steps- run_with- - Laminar---import WebhookSignatureVerification from "/snippets/webhook-signature-verification.mdx";@@ -299,7 +298,7 @@ Each mapped error looks like `{"error_code": "login_failed", "reasoning": "...",Skyvern also adds its own codes to `errors`: `REACH_MAX_STEPS`, `REACH_MAX_RETRIES`, `OTP_ERROR`, `OTP_TIMEOUT`, `OTP_EXPIRED`, `MISSING_TOTP_SOURCE`, `CREDENTIAL_LOGIN_UNCONFIRMED`, and `CREDENTIAL_ACCOUNT_LOCKED`.-When the organization-scoped credential-verdict confirmation flag is enabled, a credential rejection can instead be reported as `CREDENTIAL_LOGIN_UNCONFIRMED` or `CREDENTIAL_ACCOUNT_LOCKED`. Handle these system-defined codes explicitly anywhere you branch, retry, or filter by the original credential code. The login-request-status check is not available yet; keep this flag disabled until that check, the follow-up evaluation, and the human audit are complete.+A rejected login with a stored credential can be reported as `CREDENTIAL_LOGIN_UNCONFIRMED` or `CREDENTIAL_ACCOUNT_LOCKED` instead of your own code. Handle both anywhere you branch, retry or filter on login failures.### Failure categories@@ -430,23 +429,6 @@ const run = await client.runWorkflow({Plans and credit prices are on [skyvern.com/pricing](https://www.skyvern.com/pricing). Self-hosted servers set the default step limit with `MAX_STEPS_PER_RUN` and choose the model with `LLM_KEY`. See [configuration](/reference/configuration).-## Observability--Artifacts explain one run (see [troubleshooting](/developers/troubleshooting#use-artifacts)). For trends across runs, such as failure rate, latency, and slow tasks, send traces to [Laminar](https://laminar.sh).--**With the SDK.** Initialize Laminar before you create the Skyvern client so it can trace your SDK calls.--```python-import os-from lmnr import Laminar # pip install 'lmnr[all]'-from skyvern import Skyvern--Laminar.initialize() # reads LMNR_PROJECT_API_KEY-client = Skyvern(api_key=os.environ["SKYVERN_API_KEY"])-```--**On a self-hosted server.** Install `lmnr` in the server's Python environment and set `LMNR_PROJECT_API_KEY`. The server then traces every LLM call (prompt, response, tokens), every browser action, and every block. For a self-hosted Laminar, also set `LMNR_BASE_URL`, `LMNR_GRPC_PORT` (default `8001`), and `LMNR_HTTP_PORT` (default `8000`).-## Next- [Troubleshooting](/developers/troubleshooting): symptoms, causes, and fixesdiff --git a/docs/docs.json b/docs/docs.jsonindex 77776f1bd..7815c097c 100644--- a/docs/docs.json+++ b/docs/docs.json@@ -653,7 +653,7 @@},{"source": "/debugging/observability-with-laminar",- "destination": "/developers/production#observability"+ "destination": "/developers/troubleshooting#use-artifacts"},{"source": "/debugging/troubleshooting-guide",@@ -1025,7 +1025,7 @@},{"source": "/developers/debugging/observability-with-laminar",- "destination": "/developers/production#observability"+ "destination": "/developers/troubleshooting#use-artifacts"},{"source": "/developers/debugging/using-artifacts",diff --git a/docs/index.mdx b/docs/index.mdxindex 464c9a648..e5755d44d 100644--- a/docs/index.mdx+++ b/docs/index.mdx@@ -34,19 +34,22 @@ mode: custom<span className="sk-dot sk-dot-green"></span><span className="sk-terminal-title">first_run.py</span></div>- <div className="sk-terminal-body">- <p className="sk-mono">import asyncio</p>- <p className="sk-mono">from skyvern import Skyvern</p>- <p className="sk-mono">async def main():</p>- <p className="sk-mono" style={{ paddingLeft: "2em" }}>skyvern = Skyvern(api_key="YOUR_API_KEY")</p>- <p className="sk-mono" style={{ paddingLeft: "2em" }}>run = await skyvern.run_task(</p>- <p className="sk-mono sk-t-teal" style={{ paddingLeft: "4em" }}>prompt="Get the title of the top post",</p>- <p className="sk-mono sk-t-teal" style={{ paddingLeft: "4em" }}>url="https://news.ycombinator.com",</p>- <p className="sk-mono sk-t-teal" style={{ paddingLeft: "4em" }}>wait_for_completion=True,</p>- <p className="sk-mono" style={{ paddingLeft: "2em" }}>)</p>- <p className="sk-mono sk-t-orange" style={{ paddingLeft: "2em" }}>print(run.output)</p>- <p className="sk-mono">asyncio.run(main())</p>- </div>+ {/* Lines are JS strings so Markdown never turns the quotes into curly quotes. */}+ <pre className="sk-terminal-body sk-terminal-code"><code>+ <span className="sk-code-line">{'import asyncio'}</span>+ <span className="sk-code-line">{'from skyvern import Skyvern'}</span>+ <span className="sk-code-line">{''}</span>+ <span className="sk-code-line">{'async def main():'}</span>+ <span className="sk-code-line">{' skyvern = Skyvern(api_key="YOUR_API_KEY")'}</span>+ <span className="sk-code-line">{' run = await skyvern.run_task('}</span>+ <span className="sk-code-line sk-t-teal">{' prompt="Get the title of the top post",'}</span>+ <span className="sk-code-line sk-t-teal">{' url="https://news.ycombinator.com",'}</span>+ <span className="sk-code-line sk-t-teal">{' wait_for_completion=True,'}</span>+ <span className="sk-code-line">{' )'}</span>+ <span className="sk-code-line sk-t-orange">{' print(run.output)'}</span>+ <span className="sk-code-line">{''}</span>+ <span className="sk-code-line">{'asyncio.run(main())'}</span>+ </code></pre></div></div></div>diff --git a/docs/reference/actions.mdx b/docs/reference/actions.mdxindex 492ea32df..791d3b883 100644--- a/docs/reference/actions.mdx+++ b/docs/reference/actions.mdx@@ -82,12 +82,12 @@ If you're building automations visually instead, see [blocks reference](/referenPerform any action described in natural language.<CodeGroup>-```python+```python Pythonawait page.act("Click the login button")await page.act("Scroll down to the pricing section")```-```typescript+```typescript TypeScriptawait page.act("Click the login button")await page.act("Scroll down to the pricing section")```@@ -110,7 +110,7 @@ await page.act("Scroll down to the pricing section")Pull structured data from the visible page.<CodeGroup>-```python+```python Pythondata = await page.extract("Extract all product names and prices",schema={@@ -126,7 +126,7 @@ data = await page.extract()```-```typescript+```typescript TypeScriptconst data = await page.extract({prompt: "Extract all product names and prices",schema: {@@ -160,11 +160,11 @@ const data = await page.extract({Assert a condition about the current page state.<CodeGroup>-```python+```python Pythonis_logged_in = await page.validate("The user is logged in")```-```typescript+```typescript TypeScriptconst isLoggedIn = await page.validate("The user is logged in")```</CodeGroup>@@ -185,14 +185,14 @@ const isLoggedIn = await page.validate("The user is logged in")Ask the LLM a question about the current page.<CodeGroup>-```python+```python Pythonresult = await page.prompt("How many items are in the navigation menu?",schema={"type": "object", "properties": {"count": {"type": "integer"}}})```-```typescript+```typescript TypeScriptconst result = await page.prompt("How many items are in the navigation menu?",{ schema: { type: "object", properties: { count: { type: "integer" } } } }@@ -217,13 +217,13 @@ const result = await page.prompt(Click an element using a selector, AI prompt, or both.<CodeGroup>-```python+```python Pythonawait page.click("#submit-button") # selectorawait page.click(prompt="Click the 'Submit' button") # AIawait page.click("#submit-button", prompt="Click the 'Submit' button") # both```-```typescript+```typescript TypeScriptawait page.click("#submit-button") // selectorawait page.click({ prompt: "Click the 'Submit' button" }) // AIawait page.click("#submit-button", { prompt: "Click the 'Submit' button" }) // both@@ -236,7 +236,7 @@ await page.click("#submit-button", { prompt: "Click the 'Submit' button" }) //| `prompt` | `str` | No | AI prompt (fallback or primary) || `ai` | `str` | No | AI mode: `"fallback"` (default) or `None` |-**Returns:** `str \| None` (resolved selector)+**Returns:** `str | None` (resolved selector)**SDK reference:** [click](/sdk-reference/browser-automation/page#page-click)@@ -247,13 +247,13 @@ await page.click("#submit-button", { prompt: "Click the 'Submit' button" }) //Fill an input field using a selector, AI prompt, or both.<CodeGroup>-```python+```python Pythonawait page.fill("#email", value="[email protected]") # selectorawait page.fill(prompt="Fill '[email protected]' in the email field") # AIawait page.fill("#email", value="[email protected]", prompt="Fill email") # both```-```typescript+```typescript TypeScriptawait page.fill("#email", "[email protected]") // selectorawait page.fill({ prompt: "Fill '[email protected]' in the email field" }) // AIawait page.fill("#email", "[email protected]", { prompt: "Fill email" }) // both@@ -280,13 +280,13 @@ await page.fill("#email", "[email protected]", { prompt: "Fill email" }) //Select a dropdown option using a selector, AI prompt, or both.<CodeGroup>-```python+```python Pythonawait page.select_option("#country", value="us") # selectorawait page.select_option(prompt="Select 'United States' from the country dropdown") # AIawait page.select_option("#country", value="us", prompt="Select United States") # both```-```typescript+```typescript TypeScriptawait page.selectOption("#country", { value: "us" }) // selectorawait page.selectOption({ prompt: "Select 'United States' from the country dropdown" }) // AIawait page.selectOption("#country", { value: "us", prompt: "Select United States" }) // both@@ -480,14 +480,14 @@ All agent methods return either a [`TaskRunResponse`](/sdk-reference/runs#run_taAuthenticate with stored credentials. Handles multi-page login flows, CAPTCHAs, and 2FA.<CodeGroup>-```python+```python Pythonawait page.agent.login(credential_type="skyvern",credential_id="cred_123")```-```typescript+```typescript TypeScriptawait page.agent.login("skyvern", {credentialId: "cred_123"})@@ -516,7 +516,7 @@ await page.agent.login("skyvern", {Run a multi-step AI task on the current page.<CodeGroup>-```python+```python Pythonresult = await page.agent.run_task("Go to the billing page and download the latest invoice",data_extraction_schema={@@ -530,7 +530,7 @@ result = await page.agent.run_task(print(result.output)```-```typescript+```typescript TypeScriptconst result = await page.agent.runTask("Go to the billing page and download the latest invoice",{@@ -552,7 +552,7 @@ console.log(result.output)| `prompt` | `str` | Yes | Multi-step goal description || `data_extraction_schema` | `dict \| str` | No | JSON Schema for structured output. See [Extract Structured Data](/developers/browser-automation#extract-structured-data). || `max_steps` | `int` | No | Cap AI steps. The run ends with status `failed` ("Reached the maximum steps") if hit. Controls cost; each step is one AI decision + action cycle. |-| `engine` | `RunEngine` | No | AI engine. Defaults to `skyvern-1.0` in the SDK and Cloud UI. `skyvern-2.0` remains available for existing V2 workflows and explicitly requested V2 task runs. Other options: `skyvern-3.0`, `openai-cua`, `anthropic-cua`, `ui-tars`, `yutori-navigator`. |+| `engine` | `RunEngine` | No | AI engine. Defaults to `skyvern-1.0`. In the app, blocks with no engine set run on Skyvern 3.0 in agents created since October 1, 2026, and the engine selector labels `skyvern-3.0` Recommended. `skyvern-2.0` remains available for existing Skyvern 2.0 agents and explicitly requested Skyvern 2.0 task runs. Other options: `skyvern-3.0`, `openai-cua`, `anthropic-cua`, `ui-tars`, `yutori-navigator`. || `model` | `dict` / `Record<string, unknown>` | No | Override LLM model configuration || `url` | `str` / `string` | No | URL to navigate to (defaults to current page URL) || `webhook_url` | `str` / `string` | No | Callback URL. Skyvern POSTs the full run result on completion or failure. See [Webhooks](/developers/production#webhooks). |@@ -574,7 +574,7 @@ console.log(result.output)Navigate and download files from the current page.<CodeGroup>-```python+```python Pythonresult = await page.agent.download_files("Download the latest invoice PDF",download_suffix="invoice",@@ -582,7 +582,7 @@ result = await page.agent.download_files()```-```typescript+```typescript TypeScriptconst result = await page.agent.downloadFiles("Download the latest invoice PDF",{@@ -612,14 +612,14 @@ const result = await page.agent.downloadFiles(Run a Cloud UI workflow on the current page.<CodeGroup>-```python+```python Pythonresult = await page.agent.run_workflow("wpid_monthly_report",parameters={"month": "2025-03"})```-```typescript+```typescript TypeScriptconst result = await page.agent.runWorkflow("wpid_monthly_report",{ parameters: { month: "2025-03" } }diff --git a/docs/reference/blocks.mdx b/docs/reference/blocks.mdxindex a6a119802..5d41c481b 100644--- a/docs/reference/blocks.mdx+++ b/docs/reference/blocks.mdx@@ -34,7 +34,7 @@ If you're writing automations in code instead, the equivalent operations are Pag| [Data and Extraction](#data-and-extraction) | Web Search, Text Prompt, File Parser || [Control Flow](#control-flow) | Loop, While Loop, Conditional, AI Validation, Code, Wait || [Files](#files) | File Download, Cloud Storage Upload |-| [Communication](#communication) | Send Email, HTTP Request, Human Interaction |+| [Communication](#communication) | Send Email, HTTP Request, Human Interaction, Email Inbox |---@@ -46,7 +46,7 @@ These fields appear on most blocks:|-------|-------------|| **Label** | Display name on the canvas. Also determines how downstream blocks reference this block's output (`{{ label_output }}`). || **Continue on Failure** | Allow the agent to continue if this block fails |-| **Next Loop on Failure** | Only inside loops. Skip to the next iteration on failure. |+| **On block failure** | Only inside loops, where it replaces **Continue on Failure**. Choose **Stop the loop**, **Continue to next block in this iteration** or **Skip to next iteration**. || **Model** | Override the default LLM model for this block (Advanced Settings on most blocks) |Browser-based blocks (Browser Task, Browser Action, Extraction, Login, File Download) share these additional fields:@@ -54,15 +54,15 @@ Browser-based blocks (Browser Task, Browser Action, Extraction, Login, File Down| Field | Description ||-------|-------------|| **URL** | Starting URL. Leave blank to continue from the previous block's page. |-| **Engine** | AI engine. Skyvern 1.0 is the default for new browser blocks. |+| **Engine** | AI engine for this block. **Default** runs on Skyvern 3.0 in agents created since October 1, 2026. The selector labels Skyvern 3.0 **Recommended** and Skyvern 1.0 **Legacy**. See [Choose an engine per block](/cloud/studio#choose-an-engine-per-block). || **Max Steps Override** | Cap the number of AI reasoning steps || **Disable Cache** | Skip cached script execution |-| **TOTP Identifier** | Link TOTP credentials for 2FA handling |-| **TOTP Verification URL** | Endpoint this block polls when it needs a 2FA code |-| **Error Code Mapping** | Map error conditions to custom error codes (JSON) |+| **2FA Identifier** | ID that links 2FA codes to this block. When empty, the block uses the selected credential's identifier, if it has one. |+| **2FA Verification URL** | Endpoint this block polls when it needs a 2FA code |+| **Error Messages** | Map error conditions to custom error codes (JSON) |<Note>-Set **TOTP Verification URL** on the block that may encounter 2FA. Top-level workflow run `totp_url` values do not automatically populate empty block fields.+Set **2FA Verification URL** on the block that may encounter 2FA. A `totp_url` set on the run does not fill empty block fields.</Note>---@@ -73,16 +73,16 @@ Set **TOTP Verification URL** on the block that may encounter 2FA. Top-level worThe primary block for browser automation. Accepts a natural-language prompt and autonomously navigates the browser to accomplish the goal.-The block uses Skyvern 1.0 by default. Existing agents that already contain V2 browser blocks remain editable and may show a reduced field set.+The block shows the same fields for every engine except Skyvern 2.0. Skyvern 2.0 is offered only on blocks that already use it; those blocks stay editable with a reduced field set.-**Skyvern 1.0:**+**Every engine except Skyvern 2.0:**| Field | Type | Description ||-------|------|-------------|| **URL** | text | Starting URL (optional) || **Prompt** | text | Natural language instructions for the AI |-**Existing V2 blocks:**+**Existing Skyvern 2.0 blocks:**| Field | Type | Description ||-------|------|-------------|@@ -91,12 +91,12 @@ The block uses Skyvern 1.0 by default. Existing agents that already contain V2 b| **Max Steps** | number | Maximum AI steps || **Disable Cache** | boolean | Skip cached script execution |-Additional fields in **Advanced Settings** (Skyvern 1.0 only):+Additional fields in **Advanced Settings** (not on Skyvern 2.0 blocks):| Field | Type | Description ||-------|------|-------------|| **Complete if...** | text | Condition that signals the block is done |-| **Include Action History in Verification** | boolean | Include prior actions when verifying completion |+| **Include Action History** | switch | Include prior actions when verifying completion || **Complete on Download** | switch | Mark the block as complete when a file download occurs || **File Name** | text | Custom filename for downloaded files |@@ -359,7 +359,7 @@ Additional fields in **Advanced Settings:**| Field | Type | Description ||-------|------|-------------|-| **Error Code Mapping** | JSON | Custom error codes for failure reasons |+| **Error Messages** | JSON | Custom error codes for failure reasons || **Input Parameters** | select | Parameters to evaluate |<Frame>@@ -396,7 +396,7 @@ values. Before string values in a successful block output or a surfaced failurereason are persisted, Skyvern masks exact registered-secret matches; registeredsecrets of at least five characters are also masked inside larger strings. Derivedor transformed values may no longer match. Do not expose credentials through output-or errors. Opaque credential containment is tracked separately in SKY-11771.+or errors.<Frame><img width="400" src="/images/cloud/code_block.png" alt="Code block configuration with Python code editor and Input Parameters field" />@@ -408,10 +408,10 @@ Pause agent execution for a specified duration.| Field | Type | Description ||-------|------|-------------|-| **Wait in Seconds** | number | Seconds to wait (0–300) |+| **Wait Time (in seconds)** | number | Seconds to wait, from 1 to 1800 |<Frame>-<img width="400" src="/images/cloud/wait_block.png" alt="Wait block configuration with Wait in Seconds field" />+<img width="400" src="/images/cloud/wait_block.png" alt="Wait block configuration with Wait Time (in seconds) field" /></Frame>---@@ -484,11 +484,25 @@ Send an email notification, optionally with file attachments from previous block| Field | Type | Description ||-------|------|-------------|-| **Recipients** | text | Comma-separated email addresses |+| **Send with** | select | `SMTP` (default) sends through Skyvern's sender or your own SMTP server. `Connected Gmail account` sends from a Gmail account connected on [Integrations](/cloud/integrations) with **Send email** turned on. |+| **Gmail account** | selector | Only with `Connected Gmail account`. The account the email is sent from. |+| **Recipients** / **To** | text | Comma-separated email addresses. The label reads **To** when sending with Gmail. |+| **Cc** | text | Only with `Connected Gmail account`. Comma-separated email addresses. |+| **Bcc** | text | Only with `Connected Gmail account`. Comma-separated email addresses. A Gmail send needs at least one address in To, Cc or Bcc. || **Subject** | text | Email subject line. Supports [parameter references](/cloud/studio#inputs) — add `{{workflow_run_id}}` to tag the subject with the run, e.g. `Your Run is Finished {{workflow_run_id}}`. || **Body** | text | Email body. Supports [parameter references](/cloud/studio#inputs). || **Body format** | select | `Text` (default) sends the body exactly as written. `HTML` renders the body as a formatted email and adds a plain-text version for clients that cannot show HTML; `<script>`, `<iframe>`, `<object>`, `<embed>`, `<svg>`, `<math>`, meta refresh directives, event-handler attributes and links whose scheme is not `http`, `https`, `mailto`, `tel` or `cid` (inline `data:image` sources are kept) are removed before sending. Mail clients strip inline SVG anyway, so use a raster image instead. `{{ }}` and `{% %}` are parameter syntax, so wrap literal braces in `{% raw %}…{% endraw %}`. |-| **File Attachments** | text | Path to files to attach |+| **File Attachments** | text | With SMTP, fixed to the run's download folder, so files that earlier blocks downloaded are attached. With Gmail, comma-separated file paths from this run, uploaded files or file URLs, up to 25 MB in total. Folders and the download folder are not attached. |++Fields in **Advanced Settings** (SMTP only):++| Field | Type | Description |+|-------|------|-------------|+| **Sender (From)** | text | The From address. With a custom SMTP server, use an address that server may send as. |+| **SMTP Host** | text | Your SMTP server. Leave blank to use Skyvern's default sender. |+| **SMTP Port** | number | Defaults to 587. Port 465 uses implicit TLS; other ports use STARTTLS. |+| **SMTP Username** | text | Login for your SMTP server |+| **SMTP Password** | password | Password for your SMTP server, or a secret input reference. For Gmail over SMTP, use an App Password. |<Frame><img width="400" src="/images/cloud/email_block.png" alt="Send Email block configuration with Recipients, Subject, Body, and File Attachments fields" />@@ -527,7 +541,7 @@ Pause the agent and request human input. Optionally sends an email notification| Field | Type | Description ||-------|------|-------------|| **Instructions For Human** | text | Instructions displayed to the reviewer |-| **Timeout (minutes)** | number | Minutes to wait before auto-continuing (default: 120 minutes / 2 hours) |+| **Timeout (minutes)** | number | Minutes to wait for a response (default: 120 minutes / 2 hours). The block times out when nobody responds in that time. |**Email notification fields:**@@ -545,6 +559,36 @@ Additional fields in **Advanced Settings:**| **Positive Button Label** | text | Label for the approval action (default: "Approve") || **Negative Button Label** | text | Label for the rejection action (default: "Reject") |+The wait also ends before the timeout when the run itself stops. Canceling the run marks the block canceled, and closing the run's [browser session](/browsers) fails the block with "Browser session was closed while waiting for human interaction".+<Frame><img width="400" src="/images/cloud/human_block.png" alt="Human Interaction block configuration with Instructions For Human and Timeout fields" /></Frame>++### Email Inbox++Read messages from a connected Gmail or Outlook account and keep the ones a prompt describes. Connect the account on [Integrations](/cloud/integrations) first.++| Field | Type | Description |+|-------|------|-------------|+| **Email Client** | select | `Gmail` or `Outlook` |+| **Google Account** / **Microsoft Account** | selector | The connected account to read from. The label follows the Email Client. |+| **Folder** | text | Gmail label or Outlook folder to read. Defaults to the inbox. |+| **Match prompt** | text | Describes which emails to keep. Leave it blank to keep every email the filters return. |++Fields under **Optional Filters:**++| Field | Type | Description |+|-------|------|-------------|+| **Sender** | text | Only include emails from this sender |+| **Subject** | text | Only include emails matching this subject |+| **Newer Than Days** | number | Only include emails newer than this many days |+| **Max Results** | number | Maximum number of emails to return (default: 25, max: 100) |+| **Include Body** | switch | Include email body text in the results (default: on). When off, **Match prompt** cannot see message bodies either. |+| **Download Attachments** | switch | Download attachments from matched emails (default: off) |++Folder, Match prompt, Sender and Subject support [parameter references](/cloud/studio#inputs).++The output holds `email_client`, `folder`, `candidate_count`, `matched_count` and `emails`.++Turn on **Download Attachments** to pass files from matched emails to later blocks. Skyvern saves each matched email's attachments as run downloads and adds `downloaded_files` and `downloaded_file_urls` to the output. A saved attachment also carries its `file_name`, `file_path` and `url` in `emails`. Inline images are skipped. Attachments count toward the same per-file size, per-run size and file-count limits as browser downloads; an attachment that exceeds a limit or cannot be saved carries a `download_error` instead of failing the block.diff --git a/docs/reference/cli.mdx b/docs/reference/cli.mdxindex ff8fc9463..2c11ad5a2 100644--- a/docs/reference/cli.mdx+++ b/docs/reference/cli.mdx@@ -135,22 +135,60 @@ skyvern workflow update --id wpid_xxx --definition @updated.jsonskyvern workflow delete --id wpid_xxx --force # --force confirms the permanent deleteskyvern workflow run --id wpid_xxx # Execute a workflowskyvern workflow run --id wpid_xxx --params '{"url": "https://example.com"}'+skyvern workflow runs --id wpid_xxx # List runs of a workflow (--status, --search-key, --error-code, --page, --page-size)skyvern workflow status --run-id wr_xxx # Check run statusskyvern workflow cancel --run-id wr_xxx # Cancel a running workflow+skyvern workflow retry --run-id wr_xxx # Retry a finished workflow run```+### Schedules++```bash+skyvern schedule list # List all schedules (--status active|paused, --search, --page, --page-size)+skyvern schedule list --workflow-id wpid_xxx # List the schedules of one workflow+skyvern schedule get --workflow-id wpid_xxx --id wfs_xxx+skyvern schedule create --workflow-id wpid_xxx --cron "0 9 * * *" --timezone America/New_York+skyvern schedule create --workflow-id wpid_xxx --interval-seconds 259200 # Every 72 hours (minimum 300)+skyvern schedule update --workflow-id wpid_xxx --id wfs_xxx --cron "0 10 * * *"+skyvern schedule enable --workflow-id wpid_xxx --id wfs_xxx+skyvern schedule disable --workflow-id wpid_xxx --id wfs_xxx+skyvern schedule delete --workflow-id wpid_xxx --id wfs_xxx --yes # --yes is required+```++`schedule create` also takes `--parameters` (JSON object), `--name`, `--description`, `--first-fire-at` and `--disabled`. The default timezone is `UTC`.++### Organization settings++```bash+skyvern config show # Show all organization settings+skyvern config get max_steps_per_run # Show one setting+skyvern config set max_steps_per_run 25 # Change one setting+```++`config set` accepts `max_steps_per_run`, `max_steps_per_workflow_run`, `max_retries_per_step`, `webhook_callback_url`, `artifact_url_expiry_seconds` (3600 to 604800), `slug`, and the reset flags `clear_max_steps_per_workflow_run` and `clear_artifact_url_expiry_seconds`.+### Credentials-Credentials are created interactively via stdin so secrets never appear in shell history.+`skyvern credentials add` prompts for secret values so they never appear in shell history. In scripts or CI, set them as environment variables instead.```bash-skyvern credentials add # Interactive credential creation (password or credit card)+skyvern credentials add --name "Vendor portal" # Password (default type); prompts for username, password, TOTP+skyvern credentials add --name "Company card" --type credit_card # Prompts for card number, CVV, expiry, brand, holder name+skyvern credentials add --name "Portal API key" --type secret # Prompts for the secret value and an optional labelskyvern credential list # List stored credentialsskyvern credential get --id cred_xxxskyvern credential delete --id cred_xxx```-Note: `credentials` (plural) is used for the interactive `add` command; `credential` (singular) for list/get/delete. Both forms are intentional.+`--name` is required. `--type` is `password` (default), `credit_card` or `secret`.++| Type | Environment variables |+|---|---|+| `password` | `SKYVERN_CRED_USERNAME`, `SKYVERN_CRED_PASSWORD`, `SKYVERN_CRED_TOTP` |+| `credit_card` | `SKYVERN_CRED_CARD_NUMBER`, `SKYVERN_CRED_CVV`, `SKYVERN_CRED_EXP_MONTH`, `SKYVERN_CRED_EXP_YEAR`, `SKYVERN_CRED_CARD_BRAND`, `SKYVERN_CRED_HOLDER_NAME` |+| `secret` | `SKYVERN_CRED_SECRET_VALUE`, `SKYVERN_CRED_SECRET_LABEL` |++Note: `credentials` (plural) has `add`, `list`, `get` and `delete`; `credential` (singular) has only `list`, `get` and `delete`, matching the MCP tools. Both forms are intentional.### Agent blocks@@ -172,6 +210,7 @@ skyvern setup cursor # Register with Cursorskyvern setup windsurf # Register with Windsurfskyvern setup openclaw # Register with OpenClawskyvern setup opencode # Register with OpenCode (API key auth; avoids OAuth timeouts)+skyvern setup hermes # Register with Hermes (all ~/.hermes configs, including profiles)skyvern mcp switch # Interactively switch existing Skyvern MCP configsskyvern mcp switch --dry-run # Preview changes without writing filesskyvern mcp profile list # List saved switch sources@@ -189,6 +228,8 @@ skyvern docs # Open documentation in your browserskyvern quickstart # One-command setup + startskyvern init # Interactive setup without starting servicesskyvern init browser # Initialize browser configuration only+skyvern doctor # Check the installation (--fix tries to repair what fails)+skyvern doctor streaming # Check that local browser streaming works (--base-url, --browser-session-id, --timeout)```## CLI vs MCP: when to use which@@ -279,7 +320,7 @@ OpenClaw's MCP config provides the callable Skyvern tools; the workspace skill i## JSON output-All commands support `--json` for structured output, making it easy to compose with `jq` and other tools. The output is an envelope (`ok`, `action`, `data`, `error`, …), and the command's result is under `data`:+The `browser`, `workflow`, `schedule`, `config`, `credential`, `credentials`, `block`, `tasks`, `status` and `stop` commands, and `doctor streaming`, take `--json` for structured output you can pipe to `jq`. The `skill`, `setup`, `mcp`, `run`, `init`, `quickstart` and `docs` commands, and plain `doctor`, do not. The output is an envelope (`ok`, `action`, `data`, `error`, …), and the command's result is under `data`:```bash# Extract the session ID from a new session@@ -319,7 +360,7 @@ echo $SKYVERN_API_KEYYou can also pass it directly:```bash-skyvern credentials --api-key YOUR_KEY add+skyvern credentials --api-key YOUR_KEY add --name "Vendor portal"```Get an API key from [Settings](https://app.skyvern.com/settings/) in the Skyvern dashboard.diff --git a/docs/reference/configuration.mdx b/docs/reference/configuration.mdxindex e1b939094..b456f7936 100644--- a/docs/reference/configuration.mdx+++ b/docs/reference/configuration.mdx@@ -30,7 +30,7 @@ keywords:- ALLOWED_HOSTS----Every setting for a self-hosted server is an environment variable in `.env`. This page covers [LLM providers](#how-skyvern-uses-llms), [browser settings](#browser-settings), [credentials and encryption](#credentials-and-encryption), [other server settings](#other-server-settings), [storage](#storage) and [Kubernetes](#kubernetes). For setup, start at [Self-host](/developers/self-hosted/overview).+Every setting for a self-hosted server is an environment variable in `.env`. This page covers [LLM providers](#how-skyvern-uses-llms), [browser settings](#browser-settings), [credentials and encryption](#credentials-and-encryption), [other server settings](#other-server-settings), [storage](#storage), [Kubernetes](#kubernetes) and [LLM troubleshooting](#troubleshooting). For setup, start at [Self-host](/developers/self-hosted/overview).Skyvern uses LLMs to analyze screenshots and decide what actions to take. Configure at least one LLM provider before running tasks.@@ -47,7 +47,7 @@ For most deployments, configure a single provider using `LLM_KEY`. Skyvern also----## Quick Start Recommendations+## Recommended models**Best models for production (2026):**@@ -101,13 +101,12 @@ LLM_KEY=OPENAI_GPT5_5### Optional settings-```bash .env-# Use a custom API endpoint (for proxies or compatible services)-OPENAI_API_BASE=https://your-proxy.com/v1+Skyvern does not read these itself. LiteLLM, which makes the OpenAI calls, reads them from the server's environment, which includes the values in `.env`.-# Specify organization ID-OPENAI_ORGANIZATION=org-...-```+| Variable | Default | What it does |+|---|---|---|+| `OPENAI_API_BASE` | `https://api.openai.com/v1` | API endpoint for `OPENAI_*` models, for a proxy or gateway. `OPENAI_BASE_URL` wins if both are set. |+| `OPENAI_ORGANIZATION` | unset | OpenAI organization ID (`org-...`) sent with each request |---@@ -186,7 +185,7 @@ AZURE_API_VERSION=2024-08-01-preview1. Create an Azure OpenAI resource in the [Azure Portal](https://portal.azure.com)2. Open the Azure AI Foundry portal from your resource's overview page3. Go to **Shared Resources** → **Deployments**-4. Click **Deploy Model** → **Deploy Base Model** → select GPT-4o or GPT-4+4. Click **Deploy Model** → **Deploy Base Model** → select a model that accepts image input, such as `gpt-5.4` or `gpt-4.1`5. Note the **Deployment Name**. Use this for `AZURE_DEPLOYMENT`6. Copy your API key and endpoint from the Azure Portal@@ -436,7 +435,7 @@ INCEPTION_API_BASE=https://api.inceptionlabs.ai/v1----## Ollama (Local Models)+## Ollama (local models)Run open-source models locally with [Ollama](https://ollama.ai/). No API costs, but requires sufficient local compute.@@ -470,7 +469,7 @@ When running Skyvern in Docker and Ollama on the host:----## OpenAI-Compatible Endpoints+## OpenAI-compatible endpointsConnect to any service that implements the OpenAI API format, including LiteLLM, LocalAI, vLLM, and text-generation-inference.@@ -569,47 +568,6 @@ Most deployments don't need task-specific models. Start with `LLM_KEY` and `SECO----## Troubleshooting--### "To enable svg shape conversion, please set the Secondary LLM key"--Some operations require a secondary model. Set `SECONDARY_LLM_KEY` in your environment:--```bash .env-SECONDARY_LLM_KEY=OPENAI_GPT5_4-```--### "Context window exceeded"--The page content is too large for the model's context window. Options:-- Use a model with larger context support (GPT-5.5, Gemini 2.5 Pro, or Claude 4.7 Opus)-- Simplify your prompt to require less page analysis-- Start from a more specific URL with less content--### "Environment variables LLM_API_KEY are required for LLMProvider ..."--The configured `LLM_KEY` doesn't match any enabled provider, so Skyvern treats it as a generic model name and asks for `LLM_API_KEY`. Verify:-1. The provider is enabled (`ENABLE_OPENAI=true`, etc.)-2. The `LLM_KEY` value matches a supported model name exactly-3. Model names are case-sensitive: `OPENAI_GPT5_5` not `openai_gpt5_5`--### Container logs show authentication errors--Check your API key configuration:-- Ensure the key is set correctly without extra whitespace-- Verify the key hasn't expired or been revoked-- For Azure, ensure `AZURE_API_BASE` includes the full URL with `https://`--### Slow response times--LLM calls typically take 2-10 seconds. Longer times may indicate:-- Network latency to the provider-- Rate limiting (the provider may be throttling requests)-- For Ollama, insufficient local compute resources-------## Browser settingsSet these in `.env` (or the backend secret on Kubernetes) and restart the server. For the choice between headful, headless and `cdp-connect`, see [Self-host](/developers/self-hosted/overview#browser-options).@@ -1004,3 +962,45 @@ kubectl create secret tls skyvern-tls-secret --cert=path/to/tls.crt --key=path/t| Ingress not routing | `kubectl describe ingress skyvern-ingress -n skyvern` |The manifests have no authentication. Add it at the ingress, or keep the cluster off the public internet. `kubectl delete namespace skyvern` removes everything.++---++## Troubleshooting++These are LLM provider errors. For Kubernetes, see the table under [Kubernetes](#kubernetes).++### "To enable svg shape conversion, please set the Secondary LLM key"++Some operations require a secondary model. Set `SECONDARY_LLM_KEY` in your environment:++```bash .env+SECONDARY_LLM_KEY=OPENAI_GPT5_4+```++### "Context window exceeded"++The page content is too large for the model's context window. Options:+- Use a model with larger context support (GPT-5.5, Gemini 2.5 Pro, or Claude 4.7 Opus)+- Simplify your prompt to require less page analysis+- Start from a more specific URL with less content++### "Environment variables LLM_API_KEY are required for LLMProvider ..."++The configured `LLM_KEY` doesn't match any enabled provider, so Skyvern treats it as a generic model name and asks for `LLM_API_KEY`. Verify:+1. The provider is enabled (`ENABLE_OPENAI=true`, etc.)+2. The `LLM_KEY` value matches a supported model name exactly+3. Model names are case-sensitive: `OPENAI_GPT5_5` not `openai_gpt5_5`++### Container logs show authentication errors++Check your API key configuration:+- Ensure the key is set correctly without extra whitespace+- Verify the key hasn't expired or been revoked+- For Azure, ensure `AZURE_API_BASE` includes the full URL with `https://`++### Slow response times++LLM calls typically take 2-10 seconds. Longer times may indicate:+- Network latency to the provider+- Rate limiting (the provider may be throttling requests)+- For Ollama, insufficient local compute resourcesdiff --git a/docs/sdk-reference/complete-reference.mdx b/docs/sdk-reference/complete-reference.mdxindex 71643f038..f0e0b8fb6 100644--- a/docs/sdk-reference/complete-reference.mdx+++ b/docs/sdk-reference/complete-reference.mdx@@ -1,7 +1,7 @@---title: Complete Reference (For LLMs)-subtitle: Every method, parameter, and type in one page-description: Complete reference for every method, parameter, and type in the Skyvern Python SDK. Includes tasks, workflows, browser sessions, browser profiles, credentials, helper methods, browser control, and error handling in a single page.+subtitle: Every public SDK method in one page+description: Every public method of the Skyvern Python SDK on one page, with its signature and what it does. Covers tasks and runs, agents, schedules, folders, browser sessions, browser profiles, credentials, files, artifacts, helpers, browser control and error handling, with the TypeScript equivalents.slug: sdk-reference/complete-referencekeywords:- complete reference@@ -19,7 +19,7 @@ keywords:---<Note>-This page is a condensed summary. The full reference for every method, with each parameter, return type and a runnable Python and TypeScript example, is generated from the SDK code: [Tasks and runs](/sdk-reference/runs), [Agents](/sdk-reference/agents), [Schedules](/sdk-reference/schedules), [Folders](/sdk-reference/folders), [Browser sessions](/sdk-reference/browser-sessions), [Browser profiles](/sdk-reference/browser-profiles), [Credentials](/sdk-reference/credentials), [Files](/sdk-reference/files), [Artifacts](/sdk-reference/artifacts), [Helpers](/sdk-reference/helpers), and for browser automation [Browsers](/sdk-reference/browser-automation/browsers), [Page actions](/sdk-reference/browser-automation/page), [Forms](/sdk-reference/browser-automation/forms), [Frames](/sdk-reference/browser-automation/frames) and [Page agent](/sdk-reference/browser-automation/agent). Where the two disagree, the generated pages are correct.+This page lists every public SDK method with a one-line signature. Python names are shown; the TypeScript method is the camelCase name (`get_runs_v2` is `getRunsV2`) unless marked Python only. `?` marks an optional argument. The full reference for every method, with each parameter, return type and a runnable Python and TypeScript example, is generated from the SDK code: [Tasks and runs](/sdk-reference/runs), [Agents](/sdk-reference/agents), [Schedules](/sdk-reference/schedules), [Folders](/sdk-reference/folders), [Browser sessions](/sdk-reference/browser-sessions), [Browser profiles](/sdk-reference/browser-profiles), [Credentials](/sdk-reference/credentials), [Files](/sdk-reference/files), [Artifacts](/sdk-reference/artifacts), [Helpers](/sdk-reference/helpers), and for browser automation [Browsers](/sdk-reference/browser-automation/browsers), [Page actions](/sdk-reference/browser-automation/page), [Forms](/sdk-reference/browser-automation/forms), [Frames](/sdk-reference/browser-automation/frames) and [Page agent](/sdk-reference/browser-automation/agent). Where the two disagree, the generated pages are correct.</Note>## Install and initialize@@ -76,11 +76,16 @@ console.log(result.output);<CodeGroup>```python PythonSkyvern(- api_key: str, # Required- base_url: str | None = None, # Override for self-hosted deployments- environment: SkyvernEnvironment = CLOUD,# CLOUD, STAGING, or LOCAL- timeout: float | None = None, # HTTP request timeout (seconds)+ *,+ api_key: str, # Required+ environment: SkyvernEnvironment = CLOUD, # CLOUD, STAGING, or LOCAL+ base_url: str | None = None, # Override for self-hosted deployments+ headers: dict[str, str] | None = None, # Additional headers on every request+ timeout: float | None = None, # HTTP request timeout (seconds)+ follow_redirects: bool | None = True,+ httpx_client: httpx.AsyncClient | None = None, # Your own httpx client)+await client.aclose() # Close all browsers and release resources# Local mode (Python only - requires pip install "skyvern[local]")# For local browser control, also run: python -m playwright install chromium@@ -96,6 +101,7 @@ new Skyvern({maxRetries?: number, // Retry count (default: 2)headers?: Record<string, string>, // Additional headers})+await skyvern.close(); // Close all browsers and release resources```</CodeGroup>@@ -109,10 +115,11 @@ new Skyvern({```python Python# Launch / connectbrowser = await client.launch_cloud_browser(timeout=60, proxy_location=None)+# launch_cloud_browser also takes extensions, browser_profile_id and browser_type (Python only)browser = await client.use_cloud_browser(timeout=60, proxy_location=None)browser = await client.connect_to_cloud_browser_session("pbs_abc123")browser = await client.connect_to_browser_over_cdp("http://localhost:9222")-browser = await client.launch_local_browser(headless=False, port=9222) # Python only+browser = await client.launch_local_browser(headless=False, port=9222, args=None, user_data_dir=None) # Python only# Get pagespage = await browser.get_working_page() # Most recent page, or creates one@@ -206,6 +213,25 @@ await page.fill_autocomplete(selector="#city", value="San Francisco")await page.frame_switch(selector="#payment-iframe")page.frame_main() # back to main frameframes = await page.frame_list() # list all frames++# classify - pick which described state the page is in, returns the option key (Python only)+state = await page.classify(options={"logged_in": "Account dashboard", "logged_out": "Sign-in form"})++# element_fallback - let the AI reach a goal when selectors fail (Python only)+await page.element_fallback(navigation_goal="Open the most recent invoice", max_steps=5)++# dynamic_field_map - map your data to extracted form fields with AI (Python only)+mapping = await page.dynamic_field_map(form_fields=fields, data={...})++# low-level input (Python only)+await page.keypress(keys=["Enter"])+await page.move(x=100, y=200)+await page.drag(start_x=100, start_y=200, path=[(300, 200)])+await page.wait(seconds=2)++# download_file, reload_page (Python only)+path = await page.download_file(download_url="https://example.com/invoice.pdf")+await page.reload_page()``````typescript TypeScript@@ -221,7 +247,7 @@ const ok = await page.validate("User is logged in");// prompt - ask LLM about the pageconst result = await page.prompt("What is the heading?", { heading: { type: "string" } });-// find / AILocator - Python only, no TS equivalent+// locator / AILocator - Python only, no TS equivalent// click - selector, AI, or both with fallbackawait page.click("#submit-btn");@@ -240,6 +266,8 @@ await page.selectOption({ prompt: "Select United States" });// fill_form, fill_multipage_form, fill_from_mapping - Python only// extract_form_fields, validate_mapping, fill_autocomplete - Python only// frame_switch, frame_main, frame_list - Python only+// classify, element_fallback, dynamic_field_map - Python only+// keypress, move, drag, wait, download_file, reload_page - Python only```</CodeGroup>@@ -285,7 +313,7 @@ const result = await page.agent.runWorkflow("wpid_abc123", {----## Tasks+## Tasks and runs### run_task@@ -293,23 +321,23 @@ const result = await page.agent.runWorkflow("wpid_abc123", {```python Pythonresult = await client.run_task(prompt: str, # Required- url: str | None = None,engine: RunEngine = RunEngine.skyvern_v1,- wait_for_completion: bool = False,- timeout: float = 1800,- max_steps: int | None = None,- data_extraction_schema: dict | str | None = None,- browser_session_id: str | None = None,- publish_workflow: bool = False,- proxy_location: ProxyLocation | None = None,+ model: dict | None = None,+ url: str | None = None,webhook_url: str | None = None,- error_code_mapping: dict[str, str] | None = None,totp_identifier: str | None = None,totp_url: str | None = None,title: str | None = None,- model: dict | None = None,+ error_code_mapping: dict[str, str] | None = None,+ data_extraction_schema: dict | str | None = None,+ proxy_location: ProxyLocationInput = None,+ max_steps: int | None = None,+ wait_for_completion: bool = False,+ timeout: float = 1800,+ browser_session_id: str | None = None,user_agent: str | None = None,extra_http_headers: dict[str, str] | None = None,+ publish_workflow: bool = False,include_action_history_in_verification: bool | None = None,max_screenshot_scrolls: int | None = None,browser_address: str | None = None,@@ -321,72 +349,48 @@ const result = await skyvern.runTask({prompt: string, // Requiredurl?: string,engine?: RunEngine, // "skyvern-1.0" default- max_steps?: number,- data_extraction_schema?: Record<string, unknown> | string,- browser_session_id?: string,- publish_workflow?: boolean,+ title?: string,proxy_location?: ProxyLocation,- webhook_url?: string,+ data_extraction_schema?: Record<string, unknown> | string,error_code_mapping?: Record<string, string>,+ max_steps?: number,+ webhook_url?: string,totp_identifier?: string,totp_url?: string,- title?: string,+ browser_session_id?: string,model?: Record<string, unknown>,- "x-user-agent"?: string,extra_http_headers?: Record<string, string>,+ cdp_connect_headers?: Record<string, string>, // TypeScript only+ publish_workflow?: boolean,+ include_action_history_in_verification?: boolean,+ max_screenshot_scrolls?: number,browser_address?: string,run_with?: string, // TypeScript only; not accepted by Python run_task+ "x-user-agent"?: string,waitForCompletion?: boolean,- timeout?: number, // Default: 1800+ timeout?: number, // Default: 1800}): Promise<TaskRunResponse>```</CodeGroup>-**TaskRunResponse:** `run_id`, `status`, `output`, `failure_reason`, `downloaded_files`, `recording_url`, `screenshot_urls`, `app_url`, `step_count`, `script_run`, `created_at`, `finished_at`+**TaskRunResponse:** `run_id`, `status`, `output`, `failure_reason`, `downloaded_files`, `recording_url`, `screenshot_urls`, `app_url`, `browser_session_id`, `browser_profile_id`, `step_count`, `script_run`, `errors`, `created_at`, `started_at`, `finished_at`-### get_run+### Other run methods```-get_run(run_id) → GetRunResponse-```--### cancel_run--```-cancel_run(run_id)-```--### get_run_timeline--```-get_run_timeline(run_id) → list[WorkflowRunTimeline]-```--### get_run_artifacts--```-get_run_artifacts(run_id, artifact_type?) → list[Artifact]-```--### get_artifact--Removed in SDK 1.0.29. Use `get_run_artifacts` and read each artifact's `uri`, or download content with `GET /v1/artifacts/{artifact_id}/content`.--### get_runs_v2--```-get_runs_v2(page?, page_size?, status?, search_key?) → list[TaskRunListItem]-```--### retry_run_webhook--```-retry_run_webhook(run_id, request=RetryRunWebhookRequest(webhook_url=...)?) → RunWebhookReplayResponse+get_run(run_id) → GetRunResponse # Status and output of any task or agent run+get_runs_v2(page?, page_size?, status?, search_key?, run_type?) → list[TaskRunListItem] # List runs; status and run_type take one value or a list+get_run_timeline(run_id) → list[WorkflowRunTimeline] # Blocks and actions of a run, in order+cancel_run(run_id) # Cancel one run+bulk_cancel_runs(run_ids) → BulkCancelRunsResponse # Cancel several runs in one call+retry_run_webhook(run_id, request=RetryRunWebhookRequest(webhook_url=...)?) → RunWebhookReplayResponse # Resend a run's webhook, optionally to another URL```----## Workflows+## Agents++The SDK calls agents workflows: `run_workflow` runs an agent, and `workflow_id` is the agent's permanent ID (`wpid_...`).### run_workflow@@ -395,201 +399,166 @@ retry_run_webhook(run_id, request=RetryRunWebhookRequest(webhook_url=...)?) →result = await client.run_workflow(workflow_id: str, # Required. Permanent ID (wpid_...).parameters: dict | None = None,- wait_for_completion: bool = False,- timeout: float = 1800,- run_with: str | None = None, # "code" or "agent"- ai_fallback: bool | None = None,- browser_session_id: str | None = None,- browser_profile_id: str | None = None,- proxy_location: ProxyLocation | None = None,- max_steps_override: int | None = None,- webhook_url: str | None = None,- title: str | None = None,template: bool | None = None,- totp_identifier: str | None = None,+ title: str | None = None,+ proxy_location: ProxyLocationInput = None,+ webhook_url: str | None = None,totp_url: str | None = None,+ totp_identifier: str | None = None,+ browser_session_id: str | None = None,+ max_steps_override: int | None = None,user_agent: str | None = None,- extra_http_headers: dict[str, str] | None = None,+ browser_profile_id: str | None = None,max_screenshot_scrolls: int | None = None,- browser_address: str | None = None,max_elapsed_time_minutes: int | None = None,+ extra_http_headers: dict[str, str] | None = None,+ browser_address: str | None = None,+ ai_fallback: bool | None = None,+ run_with: str | None = None, # "code" or "agent"run_metadata: dict[str, str] | None = None,+ wait_for_completion: bool = False,+ timeout: float = 1800,) -> WorkflowRunResponse``````typescript TypeScriptconst result = await skyvern.runWorkflow({- agent_id: string, // Required. Permanent ID (wpid_...).+ agent_id: string, // Required. Permanent ID (wpid_...). workflow_id is accepted as an alias.parameters?: Record<string, unknown>,- browser_session_id?: string,- browser_profile_id?: string,+ template?: boolean,+ title?: string,proxy_location?: ProxyLocation,webhook_url?: string,- title?: string,- totp_identifier?: string,totp_url?: string,- "x-user-agent"?: string,+ totp_identifier?: string,+ browser_session_id?: string,+ browser_profile_id?: string,+ max_screenshot_scrolls?: number,+ max_elapsed_time_minutes?: number,extra_http_headers?: Record<string, string>,+ cdp_connect_headers?: Record<string, string>, // TypeScript onlybrowser_address?: string,ai_fallback?: boolean,run_with?: string,- template?: boolean,+ run_metadata?: Record<string, string>,+ "x-max-steps-override"?: number,+ "x-user-agent"?: string,waitForCompletion?: boolean,timeout?: number, // Default: 1800}): Promise<WorkflowRunResponse>```</CodeGroup>-**WorkflowRunResponse:** Same as TaskRunResponse plus `run_with`, `ai_fallback`, `script_run`.+**WorkflowRunResponse:** Same as TaskRunResponse plus `run_with`, `ai_fallback`, `script_id`.-### create_workflow+### Other agent methods```-create_workflow(json_definition?, yaml_definition?, folder_id?) → Workflow+retry_workflow_run(workflow_run_id, max_steps_override?, user_agent?) → WorkflowRunResponse # Retry a run with its original inputs+create_workflow(json_definition?, yaml_definition?, folder_id?) → Workflow # Create an agent from a JSON or YAML definition+get_workflow(workflow_permanent_id, version?, template?) → Workflow # Latest version, or a specific one+get_workflows(page?, page_size?, only_saved_tasks?, only_workflows?, only_templates?, search_key?, title?, folder_id?, status?, tags?, template?) → list[Workflow] # List agents; status and tags take one value or a list+get_workflow_versions(workflow_permanent_id, template?) → list[Workflow] # Every saved version of an agent+update_workflow(workflow_id, json_definition?, yaml_definition?) → Workflow # Save a new version+delete_workflow(workflow_id) # Permanent ID (wpid_...). Deletes all versions.+get_workflow_runs(page?, page_size?, status?, search_key?, error_code?) → list[WorkflowRun] # Runs across all agents+get_workflow_runs_by_id(workflow_id, page?, page_size?, status?, search_key?, error_code?, created_at_start?, created_at_end?) → list[WorkflowRun] # Runs of one agent```-### get_workflow+**Workflow fields:** `workflow_id`, `workflow_permanent_id`, `version`, `title`, `description`, `workflow_definition`, `status`, `folder_id`, `browser_profile_id`, `created_at`, `modified_at`-```-get_workflow(workflow_permanent_id, version?, template?) → Workflow-```--### get_workflows--```-get_workflows(page?, page_size?, only_saved_tasks?, only_workflows?, only_templates?, title?, search_key?, folder_id?, status?, template?) → list[Workflow]-```--### get_workflow_versions+----```-get_workflow_versions(workflow_permanent_id, template?) → list[Workflow]-```+## Schedules-### update_workflow+Schedules live on `client.schedules`. They start runs of an agent on a cron expression or an interval.```-update_workflow(workflow_id, json_definition?, yaml_definition?) → Workflow+schedules.list_all(page?, page_size?, status?, search?) → OrganizationScheduleListResponse # Every schedule in your organization+schedules.list(workflow_permanent_id) → WorkflowScheduleListResponse # Schedules of one agent+schedules.create(workflow_permanent_id, timezone, cron_expression?, interval_seconds?, first_fire_at?, run_at?, enabled?, parameters?, name?, description?) → WorkflowScheduleResponse+schedules.get(workflow_permanent_id, workflow_schedule_id) → WorkflowScheduleResponse+schedules.update(workflow_permanent_id, workflow_schedule_id, timezone, cron_expression?, interval_seconds?, first_fire_at?, run_at?, enabled?, parameters?, name?, description?) → WorkflowScheduleResponse # Replaces the schedule; timezone is required+schedules.enable(workflow_permanent_id, workflow_schedule_id) → WorkflowScheduleResponse+schedules.disable(workflow_permanent_id, workflow_schedule_id) → WorkflowScheduleResponse+schedules.cancel(workflow_permanent_id, workflow_schedule_id) → WorkflowScheduleResponse # Cancel a one-time schedule before it fires.+schedules.delete(workflow_permanent_id, workflow_schedule_id) → DeleteScheduleResponse```-### delete_workflow--```-delete_workflow(workflow_id) # Permanent ID (wpid_...). Deletes all versions.-```--### get_workflow_runs--```-get_workflow_runs(page?, page_size?, status?, search_key?, error_code?) → list[WorkflowRun]-```+----### update_workflow_folder+## Folders```-update_workflow_folder(workflow_permanent_id, folder_id?) → Workflow+create_folder(title, description?) → Folder+get_folder(folder_id) → Folder+get_folders(page?, page_size?, search?) → list[Folder]+update_folder(folder_id, title?, description?) → Folder+delete_folder(folder_id, delete_workflows?) # Set delete_workflows=True to delete the agents inside too+update_workflow_folder(workflow_permanent_id, folder_id?) → Workflow # Move an agent into a folder; omit folder_id to remove it from its folder```-**Workflow fields:** `workflow_id`, `workflow_permanent_id`, `version`, `title`, `workflow_definition`, `status`, `created_at`----## Browser Sessions-### create_browser_session-```create_browser_session(url?, timeout?, proxy_location?, proxy_session_id?, extensions?, browser_type?, browser_profile_id?, generate_browser_profile?) → BrowserSessionResponse-```--### get_browser_session--```get_browser_session(browser_session_id) → BrowserSessionResponse-```--### get_browser_sessions--```-get_browser_sessions() → list[BrowserSessionResponse]-```--### close_browser_session--```+get_browser_sessions() → list[BrowserSessionResponse] # Active sessions+update_browser_session(browser_session_id, generate_browser_profile) → BrowserSessionResponse # Turn profile generation on or off+extend_browser_session(browser_session_id, additional_minutes) → BrowserSessionResponse # Add minutes to a running sessionclose_browser_session(browser_session_id)```-**BrowserSessionResponse fields:** `browser_session_id`, `status`, `browser_address`, `app_url`, `timeout`, `started_at`, `created_at`+**BrowserSessionResponse fields:** `browser_session_id`, `status`, `browser_address`, `app_url`, `timeout`, `browser_profile_id`, `downloaded_files`, `recordings`, `started_at`, `completed_at`, `created_at`---## Browser Profiles-### create_browser_profile--```-create_browser_profile(name, description?, workflow_run_id?, browser_session_id?) → BrowserProfile-```--### list_browser_profiles--```-list_browser_profiles(include_deleted?) → list[BrowserProfile]-```--### get_browser_profile-```+create_browser_profile(name, description?, browser_session_id?, workflow_run_id?, proxy_location?, proxy_session_id?) → BrowserProfile # Save a profile from a session or a runget_browser_profile(profile_id) → BrowserProfile-```--### delete_browser_profile--```+list_browser_profiles(page?, page_size?, include_deleted?, search_key?, managed?) → list[BrowserProfile]+update_browser_profile(profile_id, name?, description?, proxy_location?, proxy_session_id?, rotate_proxy_session_id?) → BrowserProfiledelete_browser_profile(profile_id)```-**BrowserProfile fields:** `browser_profile_id`, `name`, `description`, `created_at`+**BrowserProfile fields:** `browser_profile_id`, `name`, `description`, `proxy_location`, `is_managed`, `created_at`, `modified_at`---## Credentials-### create_credential--```-create_credential(name, credential_type, credential, vault_type?) → CredentialResponse-```--### get_credential-```+create_credential(name, credential_type, credential, vault_type?, proxy_location?, proxy_session_id?, rotate_proxy_session_id?, tested_url?) → CredentialResponseget_credential(credential_id) → CredentialResponse+get_credentials(page?, page_size?, vault_type?, credential_type?, search?) → list[CredentialResponse]+update_credential(credential_id, name, credential_type, credential, vault_type?, proxy_location?, proxy_session_id?, rotate_proxy_session_id?, tested_url?) → CredentialResponse # Overwrites the stored values, keeps the same credential_id+delete_credential(credential_id)+send_totp_code(totp_identifier, content, task_id?, workflow_id?, workflow_run_id?, source?, expired_at?, type?) → TotpCode # Push a 2FA code to a waiting run```-### get_credentials--```-get_credentials(page?, page_size?, vault_type?) → list[CredentialResponse]-```+----### update_credential+## Files```-update_credential(credential_id, name, credential_type, credential, vault_type?) → CredentialResponse+upload_file(file) → UploadFileResponse # Returns s3uri and presigned_url```-### delete_credential+<Warning>+`upload_file` in the current Python and TypeScript SDKs fails with a 422 error: it sends `file` as a text field instead of the file's contents. Upload with a multipart request to `POST /v1/upload_file` instead, as shown on the [Files](/sdk-reference/files) page.+</Warning>-```-delete_credential(credential_id)-```+----### send_totp_code+## Artifacts```-send_totp_code(totp_identifier, content, task_id?, workflow_id?, workflow_run_id?, source?, expired_at?, type?) → TotpCode+get_run_artifacts(run_id, artifact_type?) → list[Artifact] # Artifacts of a run, optionally filtered by type+artifacts.get_artifact_content(artifact_id) # Content of one artifact```---@@ -607,19 +576,17 @@ Credential-specific parameters: `bitwarden_collection_id`, `bitwarden_item_id`,### download_files```-download_files(navigation_goal, url?, browser_session_id?, browser_profile_id?, proxy_location?, webhook_url?, download_suffix?, download_timeout?, max_steps_per_run?, extra_http_headers?, totp_identifier?, totp_url?, browser_address?, max_screenshot_scrolling_times?) → WorkflowRunResponse+download_files(navigation_goal, url?, browser_session_id?, browser_profile_id?, browser_address?, proxy_location?, webhook_url?, download_suffix?, download_timeout?, max_steps_per_run?, user_agent?, extra_http_headers?, cdp_connect_headers?, totp_identifier?, totp_url?, max_screenshot_scrolling_times?) → WorkflowRunResponse```Python does not support `wait_for_completion` - poll with `get_run()`. TypeScript supports `waitForCompletion`.-### upload_file+### get_version```-upload_file(file) → UploadFileResponse+get_version() → dict[str, str] # Version of the Skyvern server you are connected to```-Returns `s3uri` and `presigned_url`.----## Error Handling@@ -695,4 +662,4 @@ request_options=RequestOptions(- `engine` accepts `RunEngine` enum values: `skyvern_v1`, `skyvern_v2`, `skyvern_v3` (Python SDK and REST; the TypeScript client's type does not list `"skyvern-3.0"` yet), `openai_cua`, `anthropic_cua`, `ui_tars`, `yutori_navigator`.- `launch_local_browser` requires Python local mode (`Skyvern.local()`) and the `skyvern[local]` extra.- `page.agent` methods always wait for completion.-- Python-only features: `launch_local_browser`, `get_page_for`, `locator`/`AILocator`, `type`, `hover`, `scroll`, `upload_file` (page-level), form automation (`fill_form`, `fill_multipage_form`, `fill_from_mapping`, `extract_form_fields`, `validate_mapping`, `fill_autocomplete`), iframe management (`frame_switch`, `frame_main`, `frame_list`).+- Python-only features: `launch_local_browser`, `get_page_for`, `locator`/`AILocator`, `classify`, `element_fallback`, `type`, `hover`, `scroll`, `keypress`, `move`, `drag`, `wait`, `download_file`, `reload_page`, `upload_file` (page-level), form automation (`fill_form`, `fill_multipage_form`, `extract_form_fields`, `dynamic_field_map`, `fill_from_mapping`, `validate_mapping`, `fill_autocomplete`), and iframe management (`frame_switch`, `frame_main`, `frame_list`).diff --git a/docs/snippets/api-key-prompt.jsx b/docs/snippets/api-key-prompt.jsxindex 5e62e995d..1a8b90f27 100644--- a/docs/snippets/api-key-prompt.jsx+++ b/docs/snippets/api-key-prompt.jsx@@ -1,23 +1,24 @@-function dedent(text) {- if (typeof text !== "string") return String(text);- var lines = text.split("\n");- var indents = [];- for (var i = 1; i < lines.length; i++) {- if (lines[i].trim().length > 0) {- var match = lines[i].match(/^(\s*)/);- indents.push(match ? match[1].length : 0);+export const ApiKeyPrompt = ({ children }) => {+ // Mintlify only keeps the exported component, so helpers live inside it.+ function dedent(text) {+ if (typeof text !== "string") return String(text);+ var lines = text.split("\n");+ var indents = [];+ for (var i = 1; i < lines.length; i++) {+ if (lines[i].trim().length > 0) {+ var match = lines[i].match(/^(\s*)/);+ indents.push(match ? match[1].length : 0);+ }}+ var min = indents.length > 0 ? Math.min.apply(null, indents) : 0;+ if (min === 0) return text;+ return lines+ .map(function (line, i) {+ return i === 0 ? line : line.slice(min);+ })+ .join("\n");}- var min = indents.length > 0 ? Math.min.apply(null, indents) : 0;- if (min === 0) return text;- return lines- .map(function (line, i) {- return i === 0 ? line : line.slice(min);- })- .join("\n");-}-export const ApiKeyPrompt = ({ children }) => {const [apiKey, setApiKey] = React.useState("");const [copied, setCopied] = React.useState(false);diff --git a/docs/style.css b/docs/style.cssindex b5da1cd82..f2e83e94d 100644--- a/docs/style.css+++ b/docs/style.css@@ -78,6 +78,11 @@min-height: 380px;}+.sk-hero-left,+.sk-hero-right {+ min-width: 0;+}+.sk-hero-left {padding: 48px 40px;display: flex;@@ -191,6 +196,34 @@color: #ccc;}+/* Hero sample: a real <pre> so it copies as valid Python and scrolls instead of clipping. */+.sk-terminal-code {+ margin: 0 !important;+ background: transparent !important;+ border: none !important;+ border-radius: 0 !important;+ overflow-x: auto;+ font-family: var(--sk-mono) !important;+ font-size: 11.5px !important;+ line-height: 1.6 !important;+ color: #ccc !important;+}++.sk-terminal-code code {+ font-family: inherit !important;+ font-size: inherit !important;+ background: transparent !important;+ color: inherit !important;+ padding: 0 !important;+ border: none !important;+}++.sk-code-line {+ display: block;+ white-space: pre;+ min-height: 1.6em;+}+.sk-t-teal {color: #7b93fa !important;}@@ -592,21 +625,25 @@ article a:not([class]):hover {border-bottom-color: var(--sk-dark) !important;}-[class*="callout"],-[class*="Callout"],-[class*="note"],-[class*="Note"] {+/* Only Note and Info get the brand blue; Warning, Tip, Check and Danger keep a background+ that matches their own text color. */+[data-callout-type="note"],+[data-callout-type="info"] {border-left-color: var(--sk-blue) !important;border-radius: 0 var(--sk-radius) var(--sk-radius) 0 !important;background-color: var(--sk-navy-faint) !important;}-[class*="warning"],-[class*="Warning"] {+[data-callout-type="warning"] {border-left-color: #e5a236 !important;background-color: #fef9ee !important;}+.dark [data-callout-type="warning"],+[data-theme="dark"] [data-callout-type="warning"] {+ background-color: rgba(229, 162, 54, 0.1) !important;+}+[class*="Accordion"],[class*="accordion"],details {@@ -831,6 +868,7 @@ header[class*="header"] {.dark .sk-terminal,[data-theme="dark"] .sk-terminal {border-color: #333;+ background: #1b2038;}.dark .sk-cell-link:hover,diff --git a/pyproject.toml b/pyproject.tomlindex 3855c9df3..dd0ce3ca0 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -1,6 +1,6 @@[project]name = "skyvern"-version = "1.0.55"+version = "1.0.56"description = ""authors = [{ name = "Skyvern AI", email = "[email protected]" }]requires-python = ">=3.11,<3.15"diff --git a/skyvern-frontend/src/components/BrowserStream.test.tsx b/skyvern-frontend/src/components/BrowserStream.test.tsxindex 8b0cebdba..2373698c7 100644--- a/skyvern-frontend/src/components/BrowserStream.test.tsx+++ b/skyvern-frontend/src/components/BrowserStream.test.tsx@@ -14,6 +14,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";import { BrowserStream } from "./BrowserStream";import { BrowserSession } from "@/routes/browserSessions/BrowserSession";+import { useRecordingLauncherStore } from "@/store/useRecordingLauncherStore";const mocks = vi.hoisted(() => {type RfbListener = (event: { detail?: unknown }) => void;@@ -332,6 +333,9 @@ function renderBrowserSession() {describe("BrowserStream", () => {beforeEach(() => {+ useRecordingLauncherStore.setState(+ useRecordingLauncherStore.getInitialState(),+ );mocks.autoConnect.value = true;Object.defineProperty(globalThis, "WebSocket", {configurable: true,@@ -696,6 +700,36 @@ describe("BrowserStream", () => {).toBe(false);});+ it("takes VNC control while a demonstration records and releases it when recording ends", async () => {+ const { container } = renderBrowserStream();+ await screen.findByRole(+ "button",+ { name: /take control/i },+ { timeout: 10000 },+ );+ const stream = container.querySelector(".browser-stream");+ expect(stream?.classList.contains("user-is-controlling")).toBe(false);++ act(() =>+ useRecordingLauncherStore.setState({+ demonstrationBrowserSessionId: "pbs_test",+ demonstrationRecordingBrowserSessionId: "pbs_test",+ }),+ );+ await waitFor(() =>+ expect(stream?.classList.contains("user-is-controlling")).toBe(true),+ );++ act(() =>+ useRecordingLauncherStore.setState({+ demonstrationRecordingBrowserSessionId: null,+ }),+ );+ await waitFor(() =>+ expect(stream?.classList.contains("user-is-controlling")).toBe(false),+ );+ });+it("pastes an Edit-menu paste over VNC only while in control", async () => {const { container } = renderBrowserStream();const takeControlButton = await screen.findByRole(diff --git a/skyvern-frontend/src/components/BrowserStream.tsx b/skyvern-frontend/src/components/BrowserStream.tsxindex ffdff0ac4..5a70a05c5 100644--- a/skyvern-frontend/src/components/BrowserStream.tsx+++ b/skyvern-frontend/src/components/BrowserStream.tsx@@ -23,7 +23,11 @@ import { useLogging } from "@/hooks/useLogging";import { statusIsNotFinalized } from "@/routes/tasks/types";import { useRecordingStore } from "@/store/useRecordingStore";import { useSettingsStore } from "@/store/SettingsStore";-import { useManualSignInControl } from "@/store/useManualSignInStore";+import {+ useDemonstrationControl,+ useManualSignInControl,+} from "@/store/useManualSignInStore";+import { useRecordingLauncherStore } from "@/store/useRecordingLauncherStore";import { wssBaseUrl, newWssBaseUrl } from "@/util/env";import { installNoVncGestureCrashGuard } from "@/util/novncGestureCrashGuard";import { cn } from "@/util/utils";@@ -246,6 +250,13 @@ function BrowserStream({showControlButtons ? browserSessionId : undefined,setUserIsControlling,);+ const demonstrating = useRecordingLauncherStore(+ (state) => state.demonstrationBrowserSessionId === browserSessionId,+ );+ const recordingDemonstration = useRecordingLauncherStore(+ (state) =>+ state.demonstrationRecordingBrowserSessionId === browserSessionId,+ );const [vncDisconnectedTrigger, setVncDisconnectedTrigger] = useState(0);const [isVncConnected, setIsVncConnected] = useState<boolean>(false);// The message socket must open after VNC's handshake has set the ALB@@ -791,12 +802,14 @@ function BrowserStream({// When control can no longer be offered (buttons hidden and not inherently// interactive), a prior grab must be released or its input keeps flowing.- // Recording is exempt: it holds take-control for exfiltration.+ // Recording and demonstration modes hold control while their control buttons are hidden.useEffect(() => {- if (!interactive && !showControlButtons && !isRecording) {+ if (!interactive && !showControlButtons && !isRecording && !demonstrating) {setUserIsControlling(false);}- }, [interactive, showControlButtons, isRecording]);+ }, [interactive, showControlButtons, isRecording, demonstrating]);++ useDemonstrationControl(recordingDemonstration, true, setUserIsControlling);const theUserIsControlling =userIsControlling || (interactive && !showControlButtons);@@ -1044,7 +1057,7 @@ function BrowserStream({})}/>{/* Recording holds control for capture, so it offers no way to cede it. */}- {userIsControlling && !isRecording && (+ {userIsControlling && !isRecording && !demonstrating && (<StreamControlBaronStop={() => setUserIsControlling(false)}onPaste={pasteIntoVnc}diff --git a/skyvern-frontend/src/routes/browserSessions/BrowserSessionStream.test.tsx b/skyvern-frontend/src/routes/browserSessions/BrowserSessionStream.test.tsxindex acff480d5..e3cc44b9e 100644--- a/skyvern-frontend/src/routes/browserSessions/BrowserSessionStream.test.tsx+++ b/skyvern-frontend/src/routes/browserSessions/BrowserSessionStream.test.tsx@@ -7,6 +7,7 @@ import { act, render, screen, waitFor } from "@testing-library/react";import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";import { useSettingsStore } from "@/store/SettingsStore";+import { useRecordingLauncherStore } from "@/store/useRecordingLauncherStore";import {STREAM_MAX_RECONNECT_DELAY_MS,isTerminalStreamStatus,@@ -35,6 +36,12 @@ const telemetry = vi.hoisted(() => ({captureRecordBrowser: vi.fn(),}));+const interactiveStreamState = vi.hoisted(() => ({+ onNavigate: undefined as ((url: string) => void) | undefined,+ onHistoryNavigate: undefined as ((action: unknown) => void) | undefined,+ showControlButtons: false,+}));+const api = vi.hoisted(() => ({get: vi.fn(async () => ({ data: {} })),}));@@ -91,23 +98,32 @@ vi.mock("@/routes/streaming/InteractiveStreamView", () => ({currentUrl,showControlButtons,userIsControlling,+ onNavigate,+ onHistoryNavigate,}: {streamImgSrc: string;currentUrl?: string;showControlButtons: boolean;userIsControlling: boolean;- }) => (- <>- <div- data-frame={streamImgSrc}- data-url={currentUrl}- data-testid="stream-frame"- />- {showControlButtons && userIsControlling && (- <button type="button">stop controlling</button>- )}- </>- ),+ onNavigate?: (url: string) => void;+ onHistoryNavigate?: (action: unknown) => void;+ }) => {+ interactiveStreamState.onNavigate = onNavigate;+ interactiveStreamState.onHistoryNavigate = onHistoryNavigate;+ interactiveStreamState.showControlButtons = showControlButtons;+ return (+ <>+ <div+ data-frame={streamImgSrc}+ data-url={currentUrl}+ data-testid="stream-frame"+ />+ {showControlButtons && userIsControlling && (+ <button type="button">stop controlling</button>+ )}+ </>+ );+ },}));class FakeStreamSocket {@@ -317,6 +333,12 @@ describe("BrowserSessionStream terminal statuses", () => {beforeEach(() => {stubAnimationFrame();cdpInputState.setUserIsControlling = undefined;+ useRecordingLauncherStore.setState(+ useRecordingLauncherStore.getInitialState(),+ );+ interactiveStreamState.onNavigate = undefined;+ interactiveStreamState.onHistoryNavigate = undefined;+ interactiveStreamState.showControlButtons = false;Object.defineProperty(window, "WebSocket", {configurable: true,value: FakeStreamSocket,@@ -650,6 +672,105 @@ describe("BrowserSessionStream terminal statuses", () => {view.unmount();});+ it("holds control only while recording and retakes it after the stream is hidden", async () => {+ const navigateDemonstration = vi.fn();+ useRecordingLauncherStore.setState({+ demonstrationBrowserSessionId: "pbs_test",+ demonstrationRecordingBrowserSessionId: "pbs_test",+ navigateDemonstration,+ });+ const view = render(+ <BrowserSessionStream+ browserSessionId="pbs_test"+ showControlButtons={true}+ enableUrlInput={true}+ />,+ );+ await waitFor(() =>+ expect(FakeStreamSocket.instances.length).toBeGreaterThan(0),+ );+ const streamSocket = FakeStreamSocket.instances.find((socket) =>+ socket.url.includes("/stream/browser_sessions/pbs_test"),+ );+ act(() => {+ streamSocket?.emitStreamMessage({+ status: "running",+ screenshot: "frame",+ url: "https://current.example/",+ });+ });+ await waitFor(() =>+ expect(screen.getByTestId("stream-frame")).toBeTruthy(),+ );++ expect(interactiveStreamState.showControlButtons).toBe(false);+ expect(interactiveStreamState.onHistoryNavigate).toBeUndefined();+ act(() => interactiveStreamState.onNavigate?.("next.example"));+ expect(navigateDemonstration).toHaveBeenCalledWith("next.example");+ const inputSocket = FakeStreamSocket.instances.find((socket) =>+ socket.url.includes("/stream/cdp_input/browser_session/pbs_test"),+ );+ act(() => {+ inputSocket?.emit("message", {+ data: JSON.stringify({ kind: "ready" }),+ });+ });+ await waitFor(() =>+ expect(inputSocket?.send).toHaveBeenCalledWith(+ JSON.stringify({ kind: "take-control" }),+ ),+ );++ view.rerender(+ <BrowserSessionStream+ browserSessionId="pbs_test"+ showControlButtons={false}+ enableUrlInput={true}+ />,+ );+ await waitFor(() => expect(inputSocket?.close).toHaveBeenCalled());++ view.rerender(+ <BrowserSessionStream+ browserSessionId="pbs_test"+ showControlButtons={true}+ enableUrlInput={true}+ />,+ );+ await waitFor(() =>+ expect(+ FakeStreamSocket.instances.filter((socket) =>+ socket.url.includes("/stream/cdp_input/browser_session/pbs_test"),+ ),+ ).toHaveLength(2),+ );+ const reconnectedInputSocket = FakeStreamSocket.instances.filter((socket) =>+ socket.url.includes("/stream/cdp_input/browser_session/pbs_test"),+ )[1];+ act(() => {+ reconnectedInputSocket?.emit("message", {+ data: JSON.stringify({ kind: "ready" }),+ });+ });+ await waitFor(() =>+ expect(reconnectedInputSocket?.send).toHaveBeenCalledWith(+ JSON.stringify({ kind: "take-control" }),+ ),+ );++ act(() =>+ useRecordingLauncherStore.setState({+ demonstrationRecordingBrowserSessionId: null,+ }),+ );+ await waitFor(() =>+ expect(reconnectedInputSocket?.send).toHaveBeenCalledWith(+ JSON.stringify({ kind: "cede-control" }),+ ),+ );+ view.unmount();+ });+it("reconnects the recording channel after it disconnects during recording", async () => {vi.useFakeTimers();const view = render(diff --git a/skyvern-frontend/src/routes/browserSessions/BrowserSessionStream.tsx b/skyvern-frontend/src/routes/browserSessions/BrowserSessionStream.tsxindex e4b134e4b..0e6d9e455 100644--- a/skyvern-frontend/src/routes/browserSessions/BrowserSessionStream.tsx+++ b/skyvern-frontend/src/routes/browserSessions/BrowserSessionStream.tsx@@ -9,7 +9,10 @@ import {} from "@/routes/streaming/useRecordingMessageChannel";import { toast } from "@/components/ui/use-toast";import { InteractiveStreamView } from "@/routes/streaming/InteractiveStreamView";-import { useManualSignInControl } from "@/store/useManualSignInStore";+import {+ useDemonstrationControl,+ useManualSignInControl,+} from "@/store/useManualSignInStore";import {toastClipboardReadFailed,toastNothingToPaste,@@ -47,6 +50,7 @@ import type {StreamStateChangeHandler,} from "@/routes/streaming/streamState";import { useSettingsStore } from "@/store/SettingsStore";+import { useRecordingLauncherStore } from "@/store/useRecordingLauncherStore";import {captureRecordBrowser,getRecordBrowserContext,@@ -145,6 +149,16 @@ function BrowserSessionStream({const credentialGetter = useCredentialGetter();const logging = useLogging();const settingsStore = useSettingsStore();+ const demonstrating = useRecordingLauncherStore(+ (state) => state.demonstrationBrowserSessionId === browserSessionId,+ );+ const recordingDemonstration = useRecordingLauncherStore(+ (state) =>+ state.demonstrationRecordingBrowserSessionId === browserSessionId,+ );+ const navigateDemonstration = useRecordingLauncherStore(+ (state) => state.navigateDemonstration,+ );const socketRef = useRef<WebSocket | null>(null);const streamImgSrcRef = useRef("");@@ -306,6 +320,12 @@ function BrowserSessionStream({controllable ? browserSessionId : undefined,setUserIsControlling,);+ useDemonstrationControl(+ recordingDemonstration,+ controllable,+ setUserIsControlling,+ inputReady,+ );const keepMessageChannelAlive = !!exfiltrate || userIsControlling;keepMessageChannelAliveRef.current = keepMessageChannelAlive;@@ -761,6 +781,7 @@ function BrowserSessionStream({// eslint-disable-next-line react-hooks/exhaustive-deps}, [isReady, browserSessionId]);+ const streamNavigate = demonstrating ? navigateDemonstration : navigate;if (isReady) {return (<InteractiveStreamView@@ -771,13 +792,17 @@ function BrowserSessionStream({setUserIsControlling={setUserIsControlling}inputReady={inputReady}containerRef={containerRef}- showControlButtons={showControlButtons && !exfiltrate}+ showControlButtons={showControlButtons && !exfiltrate && !demonstrating}handlers={handlers}currentUrl={currentUrl}centered={centered}- onNavigate={enableUrlInput ? navigate : undefined}- navigateError={enableUrlInput ? navigateError : undefined}- onHistoryNavigate={enableUrlInput ? historyNavigate : undefined}+ onNavigate={enableUrlInput ? (streamNavigate ?? undefined) : undefined}+ navigateError={+ enableUrlInput && !demonstrating ? navigateError : undefined+ }+ onHistoryNavigate={+ enableUrlInput && !demonstrating ? historyNavigate : undefined+ }onFrameWidthChange={onFrameWidthChange}frameToken={streamImgToken}onFrameLoad={markLoad}diff --git a/skyvern-frontend/src/routes/runs/RunRouter.test.tsx b/skyvern-frontend/src/routes/runs/RunRouter.test.tsxindex 5d9d351a4..baa8b9d9a 100644--- a/skyvern-frontend/src/routes/runs/RunRouter.test.tsx+++ b/skyvern-frontend/src/routes/runs/RunRouter.test.tsx@@ -5,6 +5,7 @@ import { render, screen, waitFor } from "@testing-library/react";import { MemoryRouter, Route, Routes, useLocation } from "react-router-dom";import { afterEach, beforeEach, describe, expect, test, vi } from "vitest";+import { Status } from "@/api/types";import { useWorkflowPermanentId } from "@/routes/workflows/WorkflowPermanentIdContext";import { StudioPaneDefaultsProvider } from "../workflows/studio/StudioPaneDefaults";@@ -13,7 +14,11 @@ import { RunRouter } from "./RunRouter";type RunQueryResult = {data:- | { workflow_run_id: string; workflow: { workflow_permanent_id: string } }+ | {+ workflow_run_id: string;+ status: Status;+ workflow: { workflow_permanent_id: string };+ }| undefined;isLoading: boolean;isError?: boolean;@@ -21,6 +26,7 @@ type RunQueryResult = {const resolvedRun = {workflow_run_id: "wr_1",+ status: Status.Completed,workflow: { workflow_permanent_id: "wpid_123" },};@@ -31,6 +37,14 @@ const { getClientMock, realRunQuery } = vi.hoisted(() => ({realRunQuery: { enabled: false },}));+const { postHogCaptureMock } = vi.hoisted(() => ({+ postHogCaptureMock: vi.fn(),+}));++vi.mock("posthog-js/react", () => ({+ usePostHog: () => ({ capture: postHogCaptureMock }),+}));+const mocks = vi.hoisted(() => ({taskV2: vi.fn(() => ({ data: undefined, isLoading: false })),runQuery: vi.fn<@@ -38,6 +52,7 @@ const mocks = vi.hoisted(() => ({>(() => ({data: {workflow_run_id: "wr_1",+ status: "completed",workflow: { workflow_permanent_id: "wpid_123" },},isLoading: false,@@ -98,18 +113,19 @@ function renderAt(entry: string, client?: QueryClient) {new QueryClient({defaultOptions: { queries: { retry: false } },});- const tree = (- <MemoryRouter initialEntries={[entry]}>- <Routes>- <Route path="/runs/:runId/*" element={<RunRouter />} />- <Route path="/agents/*" element={<div data-testid="redirected" />} />- </Routes>- <LocationProbe />- </MemoryRouter>- );- return render(- <QueryClientProvider client={queryClient}>{tree}</QueryClientProvider>,+ const renderPage = () => (+ <QueryClientProvider client={queryClient}>+ <MemoryRouter initialEntries={[entry]}>+ <Routes>+ <Route path="/runs/:runId/*" element={<RunRouter />} />+ <Route path="/agents/*" element={<div data-testid="redirected" />} />+ </Routes>+ <LocationProbe />+ </MemoryRouter>+ </QueryClientProvider>);+ const result = render(renderPage());+ return { ...result, rerenderPage: () => result.rerender(renderPage()) };}function expectCenteredLoadingIndicator() {@@ -128,6 +144,7 @@ function expectCenteredLoadingIndicator() {describe("RunRouter", () => {beforeEach(() => {+ postHogCaptureMock.mockReset();mocks.taskV2.mockReturnValue({ data: undefined, isLoading: false });mocks.runQuery.mockReturnValue({ data: resolvedRun, isLoading: false });});@@ -143,6 +160,53 @@ describe("RunRouter", () => {expect(screen.queryByTestId("redirected")).toBeNull();});+ test("captures one safe page-view event for a run id despite status rerenders", () => {+ mocks.runQuery.mockReturnValue({+ data: { ...resolvedRun, status: Status.Failed },+ isLoading: false,+ });+ const page = renderAt("/runs/wr_1");++ expect(postHogCaptureMock.mock.calls).toEqual([+ [+ "run.page_viewed",+ { run_id: "wr_1", run_status: Status.Failed, failed: true },+ ],+ ]);++ mocks.runQuery.mockReturnValue({+ data: { ...resolvedRun, status: Status.Completed },+ isLoading: false,+ });+ page.rerenderPage();++ expect(postHogCaptureMock.mock.calls).toEqual([+ [+ "run.page_viewed",+ { run_id: "wr_1", run_status: Status.Failed, failed: true },+ ],+ ]);+ });++ test.each([+ [Status.Terminated, true],+ [Status.TimedOut, true],+ [Status.Canceled, false],+ [Status.Completed, false],+ ])("marks a %s run page view failed=%s", (status, failed) => {+ mocks.runQuery.mockReturnValue({+ data: { ...resolvedRun, status },+ isLoading: false,+ });+ renderAt("/runs/wr_1");++ expect(postHogCaptureMock).toHaveBeenCalledWith("run.page_viewed", {+ run_id: "wr_1",+ run_status: status,+ failed,+ });+ });+test.each([["overview", "timeline", "overview,browser"],["blocks", "timeline", "overview,browser"],diff --git a/skyvern-frontend/src/routes/runs/RunRouter.tsx b/skyvern-frontend/src/routes/runs/RunRouter.tsxindex eef175da6..bad44c83e 100644--- a/skyvern-frontend/src/routes/runs/RunRouter.tsx+++ b/skyvern-frontend/src/routes/runs/RunRouter.tsx@@ -10,8 +10,10 @@ import {useLocation,useParams,} from "react-router-dom";-import { useMemo } from "react";+import { useEffect, useMemo, useRef } from "react";+import { usePostHog } from "posthog-js/react";+import { Status } from "@/api/types";import { LogoMinimized } from "@/components/LogoMinimized";import { PageLayout } from "@/components/PageLayout";import { Status404 } from "@/components/Status404";@@ -38,10 +40,19 @@ const loadingIndicator = (</div>);+// Matches the analytics failure definition; canceled is not a failure.+const FAILURE_STATUSES: ReadonlySet<string> = new Set([+ Status.Failed,+ Status.Terminated,+ Status.TimedOut,+]);+function RunRouter() {+ const postHog = usePostHog();const params = useParams();const runId = params.runId;const location = useLocation();+ const capturedRunId = useRef<string | undefined>(undefined);const isTaskV2Run = Boolean(runId?.startsWith("tsk_v2"));const isWorkflowRun = Boolean(runId?.startsWith("wr_"));@@ -60,6 +71,29 @@ function RunRouter() {enabled: isWorkflowRun,});+ useEffect(() => {+ if (capturedRunId.current !== runId) {+ capturedRunId.current = undefined;+ }+ if (+ !isWorkflowRun ||+ !runId ||+ !studioRun ||+ studioRun.workflow_run_id !== runId ||+ !studioRun.workflow?.workflow_permanent_id ||+ capturedRunId.current === runId+ ) {+ return;+ }++ capturedRunId.current = runId;+ postHog.capture("run.page_viewed", {+ run_id: runId,+ run_status: studioRun.status,+ failed: FAILURE_STATUSES.has(studioRun.status),+ });+ }, [isWorkflowRun, postHog, runId, studioRun]);+const runType = isTaskV2Run? "redirect": isWorkflowRundiff --git a/skyvern-frontend/src/routes/workflows/copilot/WorkflowCopilotChat.composerControls.test.tsx b/skyvern-frontend/src/routes/workflows/copilot/WorkflowCopilotChat.composerControls.test.tsxindex d344a3cae..996056bd9 100644--- a/skyvern-frontend/src/routes/workflows/copilot/WorkflowCopilotChat.composerControls.test.tsx+++ b/skyvern-frontend/src/routes/workflows/copilot/WorkflowCopilotChat.composerControls.test.tsx@@ -611,7 +611,7 @@ describe("WorkflowCopilotChat — unflagged S4 composer", () => {);// Border + focus ring moved off the textarea onto the wrapping container.expect(ta.className).not.toContain("border-input");- expect(ta.parentElement?.className).toContain("focus-within");+ expect(ta.closest("[class*='focus-within']")).not.toBeNull();});it("starts SOP upload and task recording from the empty state", async () => {diff --git a/skyvern-frontend/src/routes/workflows/copilot/WorkflowCopilotChat.demonstration.test.tsx b/skyvern-frontend/src/routes/workflows/copilot/WorkflowCopilotChat.demonstration.test.tsxnew file mode 100644index 000000000..c90d441ea--- /dev/null+++ b/skyvern-frontend/src/routes/workflows/copilot/WorkflowCopilotChat.demonstration.test.tsx@@ -0,0 +1,897 @@+import type { ComponentProps } from "react";+import {+ act,+ cleanup,+ fireEvent,+ render,+ screen,+ waitFor,+ within,+} from "@testing-library/react";+import { QueryClient, QueryClientProvider } from "@tanstack/react-query";+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";++import { useRecordingLauncherStore } from "@/store/useRecordingLauncherStore";+import { useWorkflowYamlEditorStore } from "@/store/WorkflowYamlEditorStore";+import {+ canonicalRecoveriesByWorkflow,+ WorkflowCopilotChat,+} from "./WorkflowCopilotChat";++class MockResizeObserver {+ observe() {}+ unobserve() {}+ disconnect() {}+}++vi.stubGlobal("ResizeObserver", MockResizeObserver);++type StreamCall = {+ path: string;+ body: Record<string, unknown>;+ onMessage: (payload: Record<string, unknown>) => boolean;+};++const {+ streamCalls,+ postStreaming,+ sansApiPost,+ getClientMock,+ activeDemonstration,+ credentials,+ historyResponse,+ sansApiGet,+ demonstrationFlag,+} = vi.hoisted(() => {+ const calls: StreamCall[] = [];+ const active = { value: null as Record<string, unknown> | null };+ const savedCredentials = {+ value: [+ {+ credential_id: "credential-1",+ credential_type: "password",+ name: "Saved login",+ credential: {+ username: "[email protected]",+ totp_type: "authenticator",+ },+ },+ ],+ };+ const history = {+ data: {+ workflow_copilot_chat_id: "chat-1",+ request_turn_id: null,+ chat_history: [] as Record<string, unknown>[],+ proposed_workflow: null,+ proposed_workflow_metadata: null,+ auto_accept: false,+ pending_credential_requests: [],+ question_interactions: [],+ pending_question_cancel_token: null,+ },+ };+ const post = vi.fn().mockResolvedValue({ data: { seq: 1 } });+ const get = vi.fn((path: string) => {+ if (path === "/workflow/copilot/demonstration/active") {+ return Promise.resolve({ data: active.value });+ }+ if (path === "/credentials") {+ return Promise.resolve({ data: savedCredentials.value });+ }+ return Promise.resolve(history);+ });+ return {+ streamCalls: calls,+ postStreaming: vi.fn(+ (+ path: string,+ body: Record<string, unknown>,+ onMessage: (payload: Record<string, unknown>) => boolean,+ ) => {+ calls.push({ path, body, onMessage });+ if (path.endsWith("/chat-post")) return Promise.resolve();+ return new Promise<void>(() => {});+ },+ ),+ sansApiPost: post,+ getClientMock: vi.fn().mockResolvedValue({ get, post }),+ activeDemonstration: active,+ credentials: savedCredentials,+ historyResponse: history,+ sansApiGet: get,+ demonstrationFlag: { value: true },+ };+});++vi.mock("@/api/sse", () => ({+ getSseClient: vi.fn().mockResolvedValue({ postStreaming }),+}));+vi.mock("@/api/AxiosClient", () => ({ getClient: getClientMock }));+vi.mock("@/hooks/useCredentialGetter", () => ({+ useCredentialGetter: () => null,+}));+vi.mock("@/hooks/useFeatureFlag", () => ({+ useFeatureFlag: () => demonstrationFlag.value,+}));+vi.mock("@/components/ui/use-toast", () => ({ toast: vi.fn() }));+vi.mock("react-router-dom", async (importOriginal) => {+ const actual = await importOriginal<typeof import("react-router-dom")>();+ return {+ ...actual,+ useParams: () => ({ workflowPermanentId: "wpid_1" }),+ useSearchParams: () => [new URLSearchParams(), vi.fn()],+ useNavigate: () => vi.fn(),+ useLocation: () => ({+ pathname: "/",+ search: "",+ hash: "",+ state: null,+ key: "default",+ }),+ };+});++const saveData = {+ title: "Test workflow",+ workflow: {+ workflow_id: "wf_1",+ workflow_permanent_id: "wpid_1",+ description: "",+ totp_verification_url: null,+ is_saved_task: false,+ status: "published",+ },+ settings: {+ proxyLocation: null,+ webhookCallbackUrl: null,+ persistBrowserSession: false,+ pinSavedSessionIp: false,+ browserProfileId: null,+ browserProfileKey: null,+ model: null,+ maxScreenshotScrolls: null,+ extraHttpHeaders: null,+ runWith: "agent",+ scriptCacheKey: "",+ aiFallback: true,+ codeVersion: 2,+ runSequentially: false,+ sequentialKey: null,+ },+ parameters: [],+ blocks: [],+ workflowDefinitionVersion: 1,+};++vi.mock("@/store/WorkflowHasChangesStore", () => ({+ useWorkflowHasChangesStore: Object.assign(+ () => ({ getSaveData: () => saveData }),+ {+ getState: () => ({+ hasChanges: true,+ setHasChanges: vi.fn(),+ setSaveBlockedReason: vi.fn(),+ }),+ },+ ),+}));+vi.mock("@/routes/workflows/hooks/useWorkflowRunQuery", () => ({+ useWorkflowRunQuery: () => ({ data: undefined }),+}));++async function renderChat(+ props: ComponentProps<typeof WorkflowCopilotChat> = {},+) {+ const queryClient = new QueryClient({+ defaultOptions: { queries: { retry: false } },+ });+ render(+ <QueryClientProvider client={queryClient}>+ <WorkflowCopilotChat docked={false} {...props} />+ </QueryClientProvider>,+ );+ await waitFor(() =>+ expect(+ screen.getByRole("group", { name: "Copilot message composer" }),+ ).toBeTruthy(),+ );+}++async function startDemonstration({+ canFillCredentials = true,+}: { canFillCredentials?: boolean } = {}) {+ const previousRecordCalls = streamCalls.filter((call) =>+ call.path.endsWith("/record"),+ ).length;+ await waitFor(() =>+ expect(+ useRecordingLauncherStore.getState().startDemonstration,+ ).toBeTruthy(),+ );+ act(() => {+ useRecordingLauncherStore.getState().startDemonstration?.({+ insertion: { after_label: "open_form", before_label: "finish" },+ });+ });+ await waitFor(() =>+ expect(+ streamCalls.filter((call) => call.path.endsWith("/record")),+ ).toHaveLength(previousRecordCalls + 1),+ );+ const call = [...streamCalls]+ .reverse()+ .find((item) => item.path.endsWith("/record"));+ if (!call) throw new Error("Recorder stream did not open");+ act(() => {+ call.onMessage({+ type: "demonstration_started",+ demonstration_id: "demonstration-1",+ browser_session_id: "browser-1",+ action_count: 0,+ resumed: false,+ can_fill_credentials: canFillCredentials,+ });+ });+ return call;+}++beforeEach(() => {+ useWorkflowYamlEditorStore.setState(+ useWorkflowYamlEditorStore.getInitialState(),+ );+ useRecordingLauncherStore.setState(+ useRecordingLauncherStore.getInitialState(),+ );+ sessionStorage.clear();+ streamCalls.length = 0;+ postStreaming.mockClear();+ sansApiPost.mockClear();+ sansApiPost.mockResolvedValue({ data: { seq: 1 } });+ activeDemonstration.value = null;+ demonstrationFlag.value = true;+ sansApiGet.mockClear();+ credentials.value = [+ {+ credential_id: "credential-1",+ credential_type: "password",+ name: "Saved login",+ credential: { username: "[email protected]", totp_type: "authenticator" },+ },+ ];+ historyResponse.data.request_turn_id = null;+ historyResponse.data.chat_history = [];+ HTMLElement.prototype.scrollIntoView = vi.fn();+ HTMLElement.prototype.scrollTo = vi.fn();+ Element.prototype.hasPointerCapture = () => false;+ Element.prototype.setPointerCapture = () => {};+ Element.prototype.releasePointerCapture = () => {};+});++afterEach(() => {+ cleanup();+ canonicalRecoveriesByWorkflow.clear();+});++describe("WorkflowCopilotChat demonstration", () => {+ it("never looks for an active demonstration when the feature is off", async () => {+ demonstrationFlag.value = false;+ await renderChat({ liveBrowserSessionId: "browser-1" });+ await waitFor(() => expect(sansApiGet).toHaveBeenCalled());+ expect(+ sansApiGet.mock.calls.some(+ ([path]) => path === "/workflow/copilot/demonstration/active",+ ),+ ).toBe(false);+ });++ it("opens the recorder stream and renders recorded actions in the transcript", async () => {+ await renderChat({ liveBrowserSessionId: "browser-1" });+ const call = await startDemonstration();+ await waitFor(() =>+ expect(+ useRecordingLauncherStore.getState()+ .demonstrationRecordingBrowserSessionId,+ ).toBe("browser-1"),+ );++ expect(call.body).toEqual({+ workflow_permanent_id: "wpid_1",+ browser_session_id: "browser-1",+ insertion: { after_label: "open_form", before_label: "finish" },+ });+ const status = screen+ .getByText("Recording · 0 actions")+ .closest('[role="status"]');+ expect(status?.querySelector(".bg-red-500")).toBeTruthy();+ expect(status?.querySelector("svg")).toBeNull();+ const doneButton = screen.getByRole("button", {+ name: "Done, build the workflow",+ });+ expect((doneButton as HTMLButtonElement).disabled).toBe(false);+ const attachButton = screen.getByRole("button", { name: "Attach a file" });+ expect((attachButton as HTMLButtonElement).disabled).toBe(true);+ expect(attachButton.className).toContain("disabled:opacity-50");+ expect(attachButton.className).not.toContain(+ "disabled:pointer-events-none",+ );+ act(() => {+ call.onMessage({+ type: "demonstration_action",+ seq: 1,+ label: "Open form",+ tool_name: "click",+ });+ call.onMessage({+ type: "demonstration_action",+ seq: 2,+ label: "Submit form",+ tool_name: "click",+ });+ });++ const group = await screen.findByRole("group", {+ name: "Your demonstration",+ });+ expect(+ within(group)+ .getAllByRole("listitem")+ .map((item) => item.textContent),+ ).toEqual(["1Open form", "2Submit form"]);+ expect(within(group).getByText(/2 steps/)).toBeTruthy();+ expect((doneButton as HTMLButtonElement).disabled).toBe(false);++ postStreaming.mockImplementationOnce((path, body, onMessage) => {+ streamCalls.push({ path, body, onMessage });+ queueMicrotask(() =>+ onMessage({+ type: "turn_start",+ turn_id: "turn-demonstration",+ turn_index: 0,+ mode: "build",+ timestamp: "2026-10-14T12:00:00Z",+ }),+ );+ return new Promise(() => {});+ });+ fireEvent.click(doneButton);+ await waitFor(() =>+ expect(streamCalls.some((item) => item.path.endsWith("/chat-post"))).toBe(+ true,+ ),+ );+ expect(screen.queryByText(/^Recording/)).toBeNull();+ expect(+ useRecordingLauncherStore.getState()+ .demonstrationRecordingBrowserSessionId,+ ).toBeNull();+ expect(screen.getByRole("button", { name: "Stop" })).toBeTruthy();+ expect((screen.getByRole("textbox") as HTMLTextAreaElement).disabled).toBe(+ false,+ );+ expect(+ within(+ screen.getByRole("group", { name: "Your demonstration" }),+ ).queryByRole("listitem"),+ ).toBeNull();+ });++ it("waits for stop and preserves a failed demonstration send for retry", async () => {+ await renderChat({ liveBrowserSessionId: "browser-1" });+ const call = await startDemonstration();+ act(() => {+ call.onMessage({+ type: "demonstration_action",+ seq: 1,+ label: "Open form",+ tool_name: "click",+ });+ call.onMessage({+ type: "demonstration_action",+ seq: 2,+ label: "Submit form",+ tool_name: "click",+ });+ });++ let resolveStop: ((value: { data: { seq: number } }) => void) | undefined;+ sansApiPost.mockImplementationOnce(+ () =>+ new Promise((resolve) => {+ resolveStop = resolve;+ }),+ );+ postStreaming.mockImplementationOnce((path, body, onMessage) => {+ streamCalls.push({ path, body, onMessage });+ return Promise.reject(+ Object.assign(new Error("Recording is still active"), { status: 409 }),+ );+ });++ fireEvent.click(+ await screen.findByRole("button", {+ name: "Done, build the workflow",+ }),+ );+ await waitFor(() =>+ expect(sansApiPost).toHaveBeenCalledWith(+ "/workflow/copilot/demonstration/control",+ { demonstration_id: "demonstration-1", action: "stop" },+ ),+ );+ expect(streamCalls.some((item) => item.path.endsWith("/chat-post"))).toBe(+ false,+ );+ act(() => {+ resolveStop?.({ data: { seq: 1 } });+ });++ await waitFor(() =>+ expect(+ streamCalls.find((item) => item.path.endsWith("/chat-post"))?.body,+ ).toMatchObject({+ message:+ "I've shown you the task in the browser. Build a workflow that does it.",+ product_action: "demonstrate",+ demonstration_id: "demonstration-1",+ }),+ );+ expect(+ streamCalls.find((item) => item.path.endsWith("/chat-post"))?.body,+ ).not.toHaveProperty("demonstration_insertion");++ expect(+ await screen.findByText("Recording paused · 2 actions"),+ ).toBeTruthy();+ expect(+ useRecordingLauncherStore.getState()+ .demonstrationRecordingBrowserSessionId,+ ).toBeNull();+ expect(+ screen.getByText("I'll show you the task in the browser."),+ ).toBeTruthy();+ expect(screen.getByRole("button", { name: "Attach a file" }).title).toBe(+ "Use Build before attaching a file",+ );+ expect(+ (+ screen.getByRole("button", {+ name: "Build the workflow",+ }) as HTMLButtonElement+ ).disabled,+ ).toBe(false);++ const group = await screen.findByRole("group", {+ name: "Your demonstration",+ });+ expect(+ within(group)+ .getAllByRole("listitem")+ .map((item) => item.textContent),+ ).toEqual(["1Open form", "2Submit form"]);++ fireEvent.click(screen.getByRole("button", { name: "Build the workflow" }));+ await waitFor(() =>+ expect(+ streamCalls.filter((item) => item.path.endsWith("/chat-post")),+ ).toHaveLength(2),+ );+ await waitFor(() =>+ expect(screen.queryByText("Recording paused · 2 actions")).toBeNull(),+ );++ const sentGroup = screen.getByRole("group", {+ name: "Your demonstration",+ });+ await waitFor(() =>+ expect(within(sentGroup).queryByRole("listitem")).toBeNull(),+ );+ fireEvent.click(within(sentGroup).getByRole("button", { expanded: false }));+ expect(within(sentGroup).getAllByRole("listitem")).toHaveLength(2);+ });++ it("stops a recording with nothing captured without building", async () => {+ await renderChat({ liveBrowserSessionId: "browser-1" });+ await startDemonstration();+ sansApiPost.mockResolvedValueOnce({ data: { seq: 1, action_count: 0 } });++ fireEvent.click(+ screen.getByRole("button", { name: "Done, build the workflow" }),+ );++ expect(+ await screen.findByText("Do something in the browser first"),+ ).toBeTruthy();+ expect(+ (+ screen.getByRole("button", {+ name: "Build the workflow",+ }) as HTMLButtonElement+ ).disabled,+ ).toBe(true);+ expect(sansApiPost).toHaveBeenCalledWith(+ "/workflow/copilot/demonstration/control",+ { demonstration_id: "demonstration-1", action: "stop" },+ );+ });++ it("removes a failed start message, then discards a recording", async () => {+ await renderChat({ liveBrowserSessionId: "browser-1" });+ let rejectStart: ((reason: Error) => void) | undefined;+ postStreaming.mockImplementationOnce((path, body, onMessage) => {+ streamCalls.push({ path, body, onMessage });+ return new Promise<void>((_resolve, reject) => {+ rejectStart = reject;+ });+ });+ act(() => {+ useRecordingLauncherStore.getState().startDemonstration?.();+ });+ expect(+ await screen.findByText("I'll show you the task in the browser."),+ ).toBeTruthy();+ act(() => rejectStart?.(new Error("Already recording elsewhere")));+ await waitFor(() =>+ expect(+ screen.queryByText("I'll show you the task in the browser."),+ ).toBeNull(),+ );+ streamCalls.length = 0;++ const call = await startDemonstration();+ act(() => {+ call.onMessage({+ type: "demonstration_action",+ seq: 1,+ label: "Open form",+ tool_name: "click",+ });+ });+ expect(+ await screen.findByRole("group", { name: "Your demonstration" }),+ ).toBeTruthy();++ sansApiPost.mockRejectedValueOnce(new Error("network down"));+ fireEvent.click(await screen.findByRole("button", { name: "Discard" }));+ await waitFor(() => expect(sansApiPost).toHaveBeenCalledTimes(1));+ expect(await screen.findByText("Recording · 1 action")).toBeTruthy();+ expect(+ screen.getByRole("group", { name: "Your demonstration" }),+ ).toBeTruthy();++ sansApiPost.mockRejectedValueOnce(+ Object.assign(new Error("Demonstration has ended"), {+ isAxiosError: true,+ response: { status: 409 },+ }),+ );+ fireEvent.click(await screen.findByRole("button", { name: "Discard" }));++ await waitFor(() =>+ expect(sansApiPost).toHaveBeenCalledWith(+ "/workflow/copilot/demonstration/control",+ { demonstration_id: "demonstration-1", action: "cancel" },+ ),+ );+ await waitFor(() => {+ expect(screen.queryByText("Recording · 1 action")).toBeNull();+ expect(+ screen.queryByRole("group", { name: "Your demonstration" }),+ ).toBeNull();+ });+ });++ it("keeps a completed demonstration when a later recording is discarded", async () => {+ await renderChat({ liveBrowserSessionId: "browser-1" });+ const first = await startDemonstration();+ act(() => {+ first.onMessage({+ type: "demonstration_action",+ seq: 1,+ label: "Open first form",+ tool_name: "click",+ });+ });+ postStreaming.mockImplementationOnce((path, body, onMessage) => {+ streamCalls.push({ path, body, onMessage });+ onMessage({+ type: "turn_start",+ turn_id: "turn-first-demonstration",+ turn_index: 0,+ mode: "build",+ timestamp: "2026-10-14T12:00:00Z",+ });+ onMessage({+ type: "response",+ message: "Built the first workflow.",+ workflow_copilot_chat_id: "chat-1",+ updated_workflow: null,+ response_time: "2026-10-14T12:00:02Z",+ proposal_disposition: "no_proposal",+ turn_id: "turn-first-demonstration",+ });+ return Promise.resolve();+ });+ fireEvent.click(+ await screen.findByRole("button", {+ name: "Done, build the workflow",+ }),+ );+ await waitFor(() =>+ expect(screen.getByRole("button", { name: "Send" })).toBeTruthy(),+ );++ const second = await startDemonstration();+ act(() => {+ second.onMessage({+ type: "demonstration_action",+ seq: 1,+ label: "Open second form",+ tool_name: "click",+ });+ });+ fireEvent.click(await screen.findByRole("button", { name: "Discard" }));++ await waitFor(() =>+ expect(screen.queryByText("Open second form")).toBeNull(),+ );+ const cards = screen.getAllByRole("group", {+ name: "Your demonstration",+ });+ expect(cards).toHaveLength(1);+ fireEvent.click(within(cards[0]!).getByRole("button", { expanded: false }));+ expect(within(cards[0]!).getByText("Open first form")).toBeTruthy();+ });++ it("shows the credential card, fills from a saved login, and dismisses on Skip", async () => {+ await renderChat({ liveBrowserSessionId: "browser-1" });+ const call = await startDemonstration();+ act(() => {+ call.onMessage({+ type: "demonstration_sensitive_field",+ field: "password",+ selector: "#password",+ url: "https://example.com/login",+ });+ });++ expect(await screen.findByLabelText("Sign-in request")).toBeTruthy();+ fireEvent.click(await screen.findByText("Saved logins"));+ fireEvent.click(await screen.findByText("Saved login"));++ await waitFor(() =>+ expect(sansApiPost).toHaveBeenCalledWith(+ "/workflow/copilot/demonstration/control",+ {+ demonstration_id: "demonstration-1",+ action: "fill_credential",+ credential_id: "credential-1",+ selector: "#password",+ field: "password",+ },+ ),+ );++ act(() => {+ call.onMessage({+ type: "demonstration_action",+ seq: 2,+ label: 'Fill password from "Saved login"',+ tool_name: "fill_credential_field",+ });+ call.onMessage({+ type: "demonstration_sensitive_field",+ field: "password",+ selector: "#password-again",+ url: "https://example.com/login",+ });+ });++ fireEvent.click(+ await screen.findByRole("button", { name: "Skip for now" }),+ );+ await waitFor(() =>+ expect(screen.queryByLabelText("Sign-in request")).toBeNull(),+ );++ act(() => {+ call.onMessage({+ type: "demonstration_sensitive_field",+ field: "other",+ selector: "#private-answer",+ url: "https://example.com/form",+ });+ });+ expect(+ await screen.findByText(+ "Sensitive field skipped — its value was not recorded.",+ ),+ ).toBeTruthy();+ expect(screen.queryByLabelText("Sign-in request")).toBeNull();++ act(() => {+ call.onMessage({+ type: "demonstration_action",+ seq: 5,+ label: "Submit form",+ tool_name: "click",+ });+ });+ expect(+ screen.getByText("Sensitive field skipped — its value was not recorded."),+ ).toBeTruthy();+ const demonstration = screen.getByRole("group", {+ name: "Your demonstration",+ });+ expect(within(demonstration).getAllByRole("listitem")).toHaveLength(5);+ expect(within(demonstration).getByText(/Fill password from/)).toBeTruthy();+ expect(+ within(demonstration).getByText(/Fill sensitive field/),+ ).toBeTruthy();+ });++ it("keeps the type-it-yourself path when saved credential filling is unavailable", async () => {+ await renderChat({ liveBrowserSessionId: "browser-1" });+ const call = await startDemonstration({ canFillCredentials: false });+ act(() => {+ call.onMessage({+ type: "demonstration_sensitive_field",+ field: "password",+ selector: "#password",+ url: "https://example.com/login",+ });+ });++ const card = await screen.findByLabelText("Sign-in request");+ expect(within(card).queryByText("Saved logins")).toBeNull();+ expect(+ within(card).queryByRole("button", { name: "Connect credential" }),+ ).toBeNull();+ expect(+ within(card).getByRole("button", { name: "Skip for now" }),+ ).toBeTruthy();+ });++ it("hides Resume for a restored recording past its limit", async () => {+ activeDemonstration.value = {+ demonstration_id: "demonstration-recovered",+ status: "stopped",+ action_count: 1,+ action_labels: ["Open recovered form"],+ can_fill_credentials: false,+ can_resume: false,+ };++ await renderChat({ liveBrowserSessionId: "browser-1" });++ expect(+ await screen.findByRole("button", { name: "Build the workflow" }),+ ).toBeTruthy();+ expect(screen.queryByRole("button", { name: "Resume" })).toBeNull();+ });++ it.each(["stopped", "stopping"] as const)(+ "restores a %s recording and renders history demonstrations",+ async (status) => {+ activeDemonstration.value = {+ demonstration_id: "demonstration-recovered",+ status,+ action_count: 3,+ action_labels: [+ "Open recovered form",+ "Fill name",+ "Submit recovered form",+ ],+ can_fill_credentials: false,+ can_resume: true,+ insertion: { after_label: "first", before_label: "last" },+ };+ historyResponse.data.chat_history = [+ {+ sender: "user",+ content:+ "I've shown you the task in the browser. Build a workflow that does it.",+ created_at: "2026-10-14T12:00:00Z",+ },+ {+ sender: "ai",+ content: "Built the workflow.",+ narrative_payload: {+ demonstrationItems: ["Open form", "Submit form"],+ },+ created_at: "2026-10-14T12:01:00Z",+ },+ ];++ await renderChat({ liveBrowserSessionId: "browser-1" });++ if (status === "stopping") {+ postStreaming.mockImplementationOnce((path, body, onMessage) => {+ streamCalls.push({ path, body, onMessage });+ return Promise.reject(+ Object.assign(new Error("Recording is still active"), {+ status: 409,+ }),+ );+ });+ }++ const pausedText = await screen.findByText(+ "Recording paused · 3 actions",+ );+ const pausedStatus = pausedText.closest('[role="status"]');+ expect(pausedStatus).toBeTruthy();+ expect(pausedStatus?.querySelector("svg")).toBeTruthy();+ expect(pausedStatus?.querySelector(".bg-red-500")).toBeNull();+ expect(screen.getByRole("button", { name: "Resume" })).toBeTruthy();+ expect(+ screen.getByRole("button", { name: "Build the workflow" }),+ ).toBeTruthy();+ expect(screen.getByRole("button", { name: "Discard" })).toBeTruthy();++ const [historyCard, recoveredCard] = screen.getAllByRole("group", {+ name: "Your demonstration",+ });+ expect(recoveredCard).toBeTruthy();+ expect(+ within(recoveredCard!)+ .getAllByRole("listitem")+ .map((item) => item.textContent),+ ).toEqual([+ "1Open recovered form",+ "2Fill name",+ "3Submit recovered form",+ ]);+ expect(within(historyCard!).queryByRole("listitem")).toBeNull();+ fireEvent.click(+ within(historyCard!).getByRole("button", { expanded: false }),+ );+ expect(+ within(historyCard!)+ .getAllByRole("listitem")+ .map((item) => item.textContent),+ ).toEqual(["1Open form", "2Submit form"]);++ fireEvent.click(+ screen.getByRole("button", { name: "Build the workflow" }),+ );+ await waitFor(() =>+ expect(+ streamCalls.find((item) => item.path.endsWith("/chat-post"))?.body,+ ).toMatchObject({+ demonstration_id: "demonstration-recovered",+ }),+ );+ if (status === "stopping") {+ expect(+ await screen.findByText("Recording paused · 3 actions"),+ ).toBeTruthy();+ await waitFor(() =>+ expect(+ (+ screen.getByRole("button", {+ name: "Build the workflow",+ }) as HTMLButtonElement+ ).disabled,+ ).toBe(false),+ );+ fireEvent.click(+ screen.getByRole("button", { name: "Build the workflow" }),+ );+ await waitFor(() =>+ expect(+ streamCalls.filter((item) => item.path.endsWith("/chat-post")),+ ).toHaveLength(2),+ );+ await waitFor(() =>+ expect(screen.queryByText("Recording paused · 3 actions")).toBeNull(),+ );+ }+ expect(+ [...streamCalls]+ .reverse()+ .find((item) => item.path.endsWith("/chat-post"))?.body,+ ).not.toHaveProperty("demonstration_insertion");+ },+ );+});diff --git a/skyvern-frontend/src/routes/workflows/copilot/WorkflowCopilotChat.tsx b/skyvern-frontend/src/routes/workflows/copilot/WorkflowCopilotChat.tsxindex a9e9f9e67..85678fa32 100644--- a/skyvern-frontend/src/routes/workflows/copilot/WorkflowCopilotChat.tsx+++ b/skyvern-frontend/src/routes/workflows/copilot/WorkflowCopilotChat.tsx@@ -46,6 +46,7 @@ import {Status,} from "@/api/types";import { useCredentialGetter } from "@/hooks/useCredentialGetter";+import { useFeatureFlag } from "@/hooks/useFeatureFlag";import { CredentialsModal } from "@/routes/credentials/CredentialsModal";import { CredentialModalTypes } from "@/routes/credentials/useCredentialModalState";import { useLocation, useNavigate, useParams } from "react-router-dom";@@ -59,6 +60,7 @@ import {FileIcon,UploadIcon,PlusIcon,+ PauseIcon,ExclamationTriangleIcon,EnterFullScreenIcon,DownloadIcon,@@ -72,6 +74,7 @@ import {import { useWorkflowTitleStore } from "@/store/WorkflowTitleStore";import { useCopilotActionStore } from "@/store/useCopilotActionStore";import { useManualSignInStore } from "@/store/useManualSignInStore";+import { useRecordingLauncherStore } from "@/store/useRecordingLauncherStore";import {type CopilotAttention,useCopilotHeaderStore,@@ -127,6 +130,7 @@ import {WorkflowCopilotCredentialGenerateResult,WorkflowCopilotCredentialResponseResult,WorkflowCopilotCredentialPauseResolvedUpdate,+ WorkflowCopilotDemonstrationInsertion,WorkflowCopilotTitleUpdate,WorkflowCopilotChatSender,WorkflowCopilotChatRequest,@@ -281,6 +285,9 @@ import { TooltipProvider } from "@/components/ui/tooltip";import { useLogging } from "@/hooks/useLogging";import { getCopilotFailureLogFields } from "./copilotFailureLogFields";import { captureChatScope, type ChatScopeRefs } from "./chatReadScope";+import { WORKFLOW_COPILOT_DEMONSTRATION_FLAG } from "@/util/featureFlags";+import { safeHttpUrl } from "@/util/httpUrl";+import { useDemonstrationRecorder } from "./useDemonstrationRecorder";// Cap on retained per-turn snap-back snapshots. A typical session has a// handful of turns; this ceiling guards a runaway long-running chat.@@ -436,6 +443,9 @@ const RECOVERY_POLL_SUPERSEDE_READS = 4;const RECOVERY_IN_PROGRESS_MESSAGE ="The connection dropped, so Copilot is checking whether this turn finished.";const TEST_END_TO_END_PROMPT = "Test this workflow end to end.";+const DEMONSTRATION_START_MESSAGE = "I'll show you the task in the browser.";+const DEMONSTRATION_DONE_MESSAGE =+ "I've shown you the task in the browser. Build a workflow that does it.";const diagnoseRunReceipt = (runId: string) =>`Diagnose run ${runId} and repair the workflow.`;// Must stay equal to REFINE_RECORDING_RECEIPT in routes/workflow_copilot.py: the server@@ -503,7 +513,23 @@ const isProductAuthoredAction = (action: ArmedProductAction | null): boolean =>type ArmedProductAction =| { action: "test_end_to_end"; workflowRunId?: undefined; nonce?: undefined }| { action: "diagnose_run"; workflowRunId: string; nonce?: undefined }- | { action: "refine_recording"; workflowRunId?: undefined; nonce: string };+ | { action: "refine_recording"; workflowRunId?: undefined; nonce: string }+ | {+ action: "demonstrate";+ workflowRunId?: undefined;+ nonce?: undefined;+ demonstrationId: string;+ demonstrationItems: string[];+ };++function demonstrationItemsFromPayload(+ payload: Record<string, unknown> | null | undefined,+): string[] | undefined {+ const items = payload?.demonstrationItems;+ return Array.isArray(items) && items.every((item) => typeof item === "string")+ ? items+ : undefined;+}// Cadence for re-fetching a live test run's recorded actions. Mirrors the// backend block-status poll (5s) closely enough to surface rows soon after@@ -733,6 +759,7 @@ export interface ChatMessage {turnId?: string;};attachedFiles?: CopilotAttachedFile[];+ demonstrationItems?: string[];// Persisted row id, known after a history load or a feedback write; a live turn is rated by turnId.messageId?: string;feedback?: WorkflowCopilotMessageFeedback | null;@@ -780,6 +807,13 @@ function attachmentSizeLimit(file: File): number {: ATTACHMENT_SIZE_LIMIT_BYTES;}+function normalizeHttpUrl(value: string): string | null {+ const trimmed = value.trim();+ return safeHttpUrl(+ /^[a-z][a-z\d+.-]*:\/\//i.test(trimmed) ? trimmed : `https://${trimmed}`,+ );+}+function hasFileDragPayload(dataTransfer: DataTransfer): boolean {return Array.from(dataTransfer.types).includes("Files");}@@ -1696,6 +1730,8 @@ export function WorkflowCopilotChat({}: WorkflowCopilotChatProps = {}) {const workflowPermanentId = useWorkflowPermanentId();const logging = useLogging();+ const demonstrationEnabled =+ useFeatureFlag(WORKFLOW_COPILOT_DEMONSTRATION_FLAG) === true;useLayoutEffect(() => {if (workflowPermanentId)useWorkflowTitleStore@@ -2010,15 +2046,17 @@ export function WorkflowCopilotChat({isLastMessage: boolean;editingCredential?: CredentialApiResponse;} | null>(null);+ const pendingDemonstrationCredentialConnect = useRef(false);// Original ask turnId whose auto-continue send is in flight, so a stream// failure can roll its optimistic resolution back. Cleared in handleSend's// finally; only one continuation is ever pending (sends are serialized).const pendingTerminalContinuation = useRef<string | null>(null);// Latest handleSend for callbacks defined before it (the terminal// auto-continue fires from the modal's onCredentialCreated).- const handleSendRef = useRef<((messageOverride?: string) => void) | null>(- null,- );+ const handleSendRef = useRef<+ | ((messageOverride?: string, options?: SendOptions) => Promise<boolean>)+ | null+ >(null);// Single-flight guard for the resume POST against a double-click.const credentialResponseInFlight = useRef(false);const streamingAbortController = useRef<AbortController | null>(null);@@ -3063,6 +3101,7 @@ export function WorkflowCopilotChat({isLastMessage = false,editingCredential?: CredentialApiResponse,) => {+ pendingDemonstrationCredentialConnect.current = false;pendingCredentialConnect.current = {frame,turnId,@@ -3436,6 +3475,13 @@ export function WorkflowCopilotChat({message.attached_files && message.attached_files.length > 0? message.attached_files: undefined,+ // The list is stored on the reply's turn; the card belongs under the Done message before it.+ demonstrationItems:+ message.sender === "user"+ ? demonstrationItemsFromPayload(+ data.chat_history[index + 1]?.narrative_payload,+ )+ : undefined,narrative: (() => {const hydrated = hydrateHistoryNarrative(message.narrative_payload,@@ -4832,6 +4878,7 @@ export function WorkflowCopilotChat({undefined,true,pendingRefinementIds.get(pending.turn_id),+ chatPresentationGeneration.current,);if (adopted &&@@ -6169,6 +6216,22 @@ export function WorkflowCopilotChat({return latestChatId;};+ const uploadCopilotFile = useCallback(+ async (file: File): Promise<string> => {+ const client = await getClient(credentialGetter, "sans-api-v1");+ const formData = new FormData();+ formData.append("file", file);+ const response = await client.post<+ FormData,+ { data: { file_id: string } }+ >("/upload_file", formData, {+ headers: { "Content-Type": "multipart/form-data" },+ });+ return response.data.file_id;+ },+ [credentialGetter],+ );+const uploadAttachment = useCallback(async (file: File) => {if (file.size > attachmentSizeLimit(file)) {@@ -6217,15 +6280,7 @@ export function WorkflowCopilotChat({{ localId, filename: file.name, status: "uploading" },]);try {- const client = await getClient(credentialGetter, "sans-api-v1");- const formData = new FormData();- formData.append("file", file);- const response = await client.post<- FormData,- { data: { file_id: string } }- >("/upload_file", formData, {- headers: { "Content-Type": "multipart/form-data" },- });+ const fileId = await uploadCopilotFile(file);if (!composerMountedRef.current || pageHiddenRef.current) {// Nothing can reach this file: the composer is gone, or the page was left while the upload// was still running, so page-exit cleanup never saw an id for it.@@ -6233,7 +6288,7 @@ export function WorkflowCopilotChat({prev.filter((item) => item.localId !== localId),);const abandoned: CopilotAttachedFile = {- file_id: response.data.file_id,+ file_id: fileId,filename: file.name,size_bytes: file.size,available: true,@@ -6253,9 +6308,9 @@ export function WorkflowCopilotChat({prev.filter((item) => item.localId !== localId),);setAttachments((prev) => [- ...prev.filter((item) => item.file_id !== response.data.file_id),+ ...prev.filter((item) => item.file_id !== fileId),{- file_id: response.data.file_id,+ file_id: fileId,filename: file.name,size_bytes: file.size,available: true,@@ -6280,7 +6335,7 @@ export function WorkflowCopilotChat({});}},- [credentialGetter],+ [uploadCopilotFile],);const removeUnsentUpload = useCallback(@@ -6900,6 +6955,100 @@ export function WorkflowCopilotChat({const blockBuildTargetLabelRef = useRef<string | null>(null);// Set by a product affordance so the next send posts that typed action instead of prose.const productActionRef = useRef<ArmedProductAction | null>(null);+ const demonstrationMessageIdRef = useRef<string | null>(null);+ const demonstrationRecorder = useDemonstrationRecorder({+ workflowPermanentId,+ browserSessionId: demonstrationEnabled ? liveBrowserSessionId : null,+ onDone: async ({ demonstrationId, actions }) => {+ const messageId =+ demonstrationMessageIdRef.current ??+ `demonstration-${crypto.randomUUID()}`;+ demonstrationMessageIdRef.current = messageId;+ const demonstrationItems = actions.map((action) => action.label);+ setMessages((prev) => {+ const doneMessage: ChatMessage = {+ id: messageId,+ sender: "user",+ content: DEMONSTRATION_DONE_MESSAGE,+ demonstrationItems,+ };+ return prev.some((message) => message.id === messageId)+ ? prev.map((message) =>+ message.id === messageId+ ? { ...message, ...doneMessage }+ : message,+ )+ : [...prev, doneMessage];+ });+ const action: ArmedProductAction = {+ action: "demonstrate",+ demonstrationId,+ demonstrationItems,+ };+ productActionRef.current = action;+ let sent = false;+ try {+ sent =+ (await handleSendRef.current?.(DEMONSTRATION_DONE_MESSAGE, {+ optimisticMessageId: messageId,+ })) ?? false;+ } finally {+ if (!sent) {+ if (productActionRef.current === action) {+ productActionRef.current = null;+ }+ setMessages((prev) =>+ prev.map((message) =>+ message.id === messageId+ ? {+ ...message,+ content: DEMONSTRATION_START_MESSAGE,+ demonstrationItems: undefined,+ }+ : message,+ ),+ );+ }+ }+ return sent;+ },+ onStartFailed: () => {+ const messageId = demonstrationMessageIdRef.current;+ demonstrationMessageIdRef.current = null;+ setMessages((prev) => prev.filter((message) => message.id !== messageId));+ },+ });+ const isDemonstrating = demonstrationRecorder.phase !== "idle";+ useEffect(() => {+ if (+ !isDemonstrating ||+ !demonstrationRecorder.demonstrationId ||+ historyLoadedForRef.current !== workflowPermanentId ||+ demonstrationMessageIdRef.current+ ) {+ return;+ }+ const messageId = `demonstration-${crypto.randomUUID()}`;+ demonstrationMessageIdRef.current = messageId;+ setMessages((prev) => [+ ...prev,+ {+ id: messageId,+ sender: "user",+ content: DEMONSTRATION_START_MESSAGE,+ },+ ]);+ }, [+ demonstrationRecorder.demonstrationId,+ isDemonstrating,+ isLoadingHistory,+ workflowPermanentId,+ ]);+ const demonstrationItems = demonstrationRecorder.actions.map(+ (action) => action.label,+ );+ const { navigate: navigateRecording, start: startRecording } =+ demonstrationRecorder;const echoSenderForArmedAction = (): WorkflowCopilotChatSender =>isProductAuthoredAction(productActionRef.current) ? "product" : "user";// True only while a block-build turn is actually in flight (not a turn it queued behind).@@ -7221,6 +7370,7 @@ export function WorkflowCopilotChat({if (event.key !== "Escape" || !isOpen) {return;}+ if (isDemonstrating) return;// Dismissing an IME conversion candidate fires Escape with isComposing// set; the user is editing text, not stopping their turn.if (event.isComposing || stopBlockedByDeletionRef.current) {@@ -7245,6 +7395,7 @@ export function WorkflowCopilotChat({cancelSend,isLoading,isOpen,+ isDemonstrating,queuedPrompt,]);@@ -7339,6 +7490,47 @@ export function WorkflowCopilotChat({],);+ const navigateDemonstration = useCallback(+ (url: string) => {+ const normalizedUrl = normalizeHttpUrl(url);+ if (!normalizedUrl) {+ toast({+ variant: "destructive",+ title: "Enter an http(s) URL",+ });+ return;+ }+ void navigateRecording(normalizedUrl);+ },+ [navigateRecording],+ );+ useEffect(() => {+ useRecordingLauncherStore.setState({+ demonstrationBrowserSessionId: isDemonstrating+ ? (liveBrowserSessionId ?? null)+ : null,+ demonstrationRecordingBrowserSessionId:+ demonstrationRecorder.phase === "recording"+ ? (liveBrowserSessionId ?? null)+ : null,+ navigateDemonstration:+ demonstrationRecorder.phase === "recording"+ ? navigateDemonstration+ : null,+ });+ return () =>+ useRecordingLauncherStore.setState({+ demonstrationBrowserSessionId: null,+ demonstrationRecordingBrowserSessionId: null,+ navigateDemonstration: null,+ });+ }, [+ demonstrationRecorder.phase,+ isDemonstrating,+ liveBrowserSessionId,+ navigateDemonstration,+ ]);+const hasPendingQuestion = questionInteractions.some((item) => item.status === "pending",);@@ -7446,7 +7638,7 @@ export function WorkflowCopilotChat({Date.parse(frame.expires_at ?? "") > Date.now(),);const trayPauseFrame = openPauseFrames[0] ?? null;- const signingInSessionId =+ const credentialSignInSessionId =openPauseFrames.find((frame) =>frame.signing_in &&@@ -7454,6 +7646,7 @@ export function WorkflowCopilotChat({!manualSignIns[frame.resume_token]?.busy &&frame.sign_in_browser_session_id === (liveBrowserSessionId ?? null),)?.sign_in_browser_session_id ?? null;+ const signingInSessionId = credentialSignInSessionId;const setManualSignInSession = useManualSignInStore((state) => state.setBrowserSessionId,);@@ -7548,7 +7741,27 @@ export function WorkflowCopilotChat({: null;})();// The question always leads, so it never needs an "Up next" label.- const attentionQueue: { kind: CopilotAttention; upNext?: string }[] = [];+ const demonstrationSensitiveField =+ demonstrationRecorder.phase === "recording" &&+ (demonstrationRecorder.sensitiveField?.field === "password" ||+ demonstrationRecorder.sensitiveField?.field === "totp")+ ? demonstrationRecorder.sensitiveField+ : null;+ const demonstrationCredentialFrame: CredentialRequiredFrame | null =+ demonstrationSensitiveField+ ? {+ type: "credential_required",+ reason: "assistant_directed",+ login_page_urls: [demonstrationSensitiveField.url],+ message: demonstrationRecorder.canFillCredentials+ ? "Fill this sign-in from a saved login, or type it yourself."+ : "Type this sign-in yourself to keep its value private.",+ }+ : null;+ const attentionQueue: {+ kind: CopilotAttention | "demonstration_credential";+ upNext?: string;+ }[] = [];if (trayQuestion) {attentionQueue.push({ kind: "question" });}@@ -7561,7 +7774,23 @@ export function WorkflowCopilotChat({if (trayAccountChoice) {attentionQueue.push({ kind: "account", upNext: "Choose a Google account" });}+ if (demonstrationCredentialFrame) {+ attentionQueue.push({+ kind: "demonstration_credential",+ upNext: "Fill this sign-in",+ });+ }+ if (isDemonstrating) {+ attentionQueue.push({+ kind: "demonstration",+ upNext: "Show Copilot the task",+ });+ }const activeAttention = attentionQueue[0]?.kind ?? null;+ const headerAttention =+ activeAttention === "demonstration_credential"+ ? "credential"+ : activeAttention;const attentionUpNext = attentionQueue[1]?.upNext ?? null;// Only the open tray replaces its transcript card; queued items stay actionable inline.const dockedPauseFrame =@@ -7570,6 +7799,12 @@ export function WorkflowCopilotChat({activeAttention === "credential" ? trayTerminalAsk : null;const dockedAccountChoice =activeAttention === "account" ? trayAccountChoice : null;+ const dockedDemonstrationCredential =+ activeAttention === "demonstration_credential"+ ? demonstrationCredentialFrame+ : null;+ const hasVisibleAttentionTray =+ activeAttention !== null && activeAttention !== "demonstration";const [collapsedAttentionKey, setCollapsedAttentionKey] = useState<string | null>(null);@@ -7581,9 +7816,9 @@ export function WorkflowCopilotChat({});useEffect(() => {const store = useCopilotHeaderStore.getState();- store.setAttention(activeAttention);+ store.setAttention(headerAttention);return () => store.setAttention(null);- }, [activeAttention]);+ }, [headerAttention]);const uploadDroppedAttachments = useCallback((files: FileList) => {const droppedFiles = Array.from(files);@@ -7632,18 +7867,19 @@ export function WorkflowCopilotChat({if (!hasFileDragPayload(event.dataTransfer)) return;event.preventDefault();fileDragDepthRef.current += 1;- if (!hasPendingQuestion) setIsFileDragging(true);+ if (!hasPendingQuestion && !isDemonstrating) setIsFileDragging(true);},- [hasPendingQuestion],+ [hasPendingQuestion, isDemonstrating],);const handleComposerDragOver = useCallback((event: React.DragEvent<HTMLDivElement>) => {if (!hasFileDragPayload(event.dataTransfer)) return;event.preventDefault();- event.dataTransfer.dropEffect = hasPendingQuestion ? "none" : "copy";+ event.dataTransfer.dropEffect =+ hasPendingQuestion || isDemonstrating ? "none" : "copy";},- [hasPendingQuestion],+ [hasPendingQuestion, isDemonstrating],);const handleComposerDragLeave = useCallback(() => {@@ -7657,22 +7893,24 @@ export function WorkflowCopilotChat({event.preventDefault();fileDragDepthRef.current = 0;setIsFileDragging(false);- if (hasPendingQuestion) {+ if (hasPendingQuestion || isDemonstrating) {toast({- title: "Answer the pending question first",+ title: isDemonstrating+ ? "Finish the demonstration first"+ : "Answer the pending question first",description: "You can attach files to your next message afterward.",});return;}uploadDroppedAttachments(event.dataTransfer.files);},- [hasPendingQuestion, uploadDroppedAttachments],+ [hasPendingQuestion, isDemonstrating, uploadDroppedAttachments],);useEffect(() => {- if (!hasPendingQuestion) return;+ if (!hasPendingQuestion && !isDemonstrating) return;fileDragDepthRef.current = 0;setIsFileDragging(false);- }, [hasPendingQuestion]);+ }, [hasPendingQuestion, isDemonstrating]);useEffect(() => {if (!hasPendingQuestion || !workflowCopilotChatId) return;let disposed = false;@@ -7753,7 +7991,11 @@ export function WorkflowCopilotChat({refuseMutationDuringYamlCommit())return false;- if (authoringInProgress) return false;+ if (+ authoringInProgress ||+ (isDemonstrating && productActionRef.current?.action !== "demonstrate")+ )+ return false;// A turn started now could stage a proposal the pending Accept then clears, and the// setGateFailure below would drop a hold whose write is still unaccounted for.// Callers that consume their input first (queue, auto-send, Generate) wait for it instead.@@ -8051,6 +8293,8 @@ export function WorkflowCopilotChat({productActionRef.current?.action === "refine_recording"? productActionRef.current: null;+ const demonstrationAction =+ productActionRef.current?.action === "demonstrate";const recordingEvidence = recordingRefinementAction? useRecordingRefinementEvidenceStore.getState()@@ -8067,6 +8311,10 @@ export function WorkflowCopilotChat({id: userMessageId,sender: echoSenderForArmedAction(),content: candidate,+ demonstrationItems:+ productActionRef.current?.action === "demonstrate"+ ? productActionRef.current.demonstrationItems+ : undefined,kind: recordingRefinement ? "recording_refinement" : undefined,recordingRefinement,attachedFiles: sentAttachments.length > 0 ? sentAttachments : undefined,@@ -8996,6 +9244,10 @@ export function WorkflowCopilotChat({idempotency_key: options.idempotencyKey ?? null,target_block_label: targetBlockLabel,product_action: productAction?.action ?? null,+ demonstration_id:+ productAction?.action === "demonstrate"+ ? productAction.demonstrationId+ : null,recording_evidence:productAction?.action === "refine_recording"? useRecordingRefinementEvidenceStore@@ -9271,7 +9523,10 @@ export function WorkflowCopilotChat({error instanceof Error &&"status" in error &&typeof error.status === "number" &&- [400, 401, 403, 404, 405, 413, 415, 422, 429].includes(error.status);+ ([400, 401, 403, 404, 405, 413, 415, 422, 429].includes(+ error.status,+ ) ||+ (error.status === 409 && demonstrationAction));returnPossiblySavedFiles();console.error("Failed to send message:", error);logCopilotRequestFailure(@@ -9439,6 +9694,7 @@ export function WorkflowCopilotChat({reconcileCanonicalWorkflow,armStop,authoringInProgress,+ isDemonstrating,backfillProposalRunFacts,credentialGetter,fetchRecordedActions,@@ -9486,6 +9742,84 @@ export function WorkflowCopilotChat({void handleSend(TEST_END_TO_END_PROMPT);}, [handleSend]);+ const handleStartDemonstration = useCallback(+ ({+ insertion,+ }: { insertion?: WorkflowCopilotDemonstrationInsertion } = {}) => {+ if (!demonstrationEnabled || !liveBrowserSessionId) return;+ if (historyLoadedForRef.current !== workflowPermanentId) {+ toast({ title: "Wait for Copilot to finish loading" });+ return;+ }+ if (hasPendingQuestion) {+ toast({ title: "Answer Copilot's question first" });+ return;+ }+ if (+ inFlightRef.current ||+ queuedPromptRef.current ||+ recoveredTurnOwnerRef.current ||+ isDemonstrating+ ) {+ toast({ title: "Wait for Copilot to finish" });+ return;+ }+ void stopSpeech();+ const messageId = `demonstration-${crypto.randomUUID()}`;+ demonstrationMessageIdRef.current = messageId;+ // Shown at start so the chat reads as the user's turn; Done replaces it with the sent message.+ setMessages((prev) => [+ ...prev,+ {+ id: messageId,+ sender: "user",+ content: DEMONSTRATION_START_MESSAGE,+ },+ ]);+ startRecording({ insertion });+ },+ [+ demonstrationEnabled,+ hasPendingQuestion,+ isDemonstrating,+ liveBrowserSessionId,+ startRecording,+ stopSpeech,+ workflowPermanentId,+ ],+ );+ const setStartDemonstration = useRecordingLauncherStore(+ (state) => state.setStartDemonstration,+ );+ useEffect(() => {+ setStartDemonstration(+ demonstrationEnabled &&+ liveBrowserSessionId &&+ historyLoadedForRef.current === workflowPermanentId &&+ !hasPendingQuestion &&+ !isLoading &&+ !isLoadingHistory &&+ !queuedPrompt &&+ !isDemonstrating &&+ !authoringInProgress+ ? handleStartDemonstration+ : null,+ );+ return () => setStartDemonstration(null);+ }, [+ demonstrationEnabled,+ handleStartDemonstration,+ hasPendingQuestion,+ isLoading,+ isLoadingHistory,+ liveBrowserSessionId,+ queuedPrompt,+ isDemonstrating,+ authoringInProgress,+ setStartDemonstration,+ workflowPermanentId,+ ]);+const handleConnectedAccountChoice = useCallback((turnId: string, connectionId: string) => {// `handleSend` drops the send when the fence is up, and it does so AFTER this latch would@@ -10414,6 +10748,7 @@ export function WorkflowCopilotChat({!isStopping &&!hasPendingQuestion &&livePauseFrame === null &&+ !isDemonstrating &&turnChatIdRef.current,);// A question pause ends isLoading, so the deletion shows its own disabled Stop rather than Send.@@ -10467,6 +10802,19 @@ export function WorkflowCopilotChat({: hasComposerText? "Queue for next turn": "Stop";+ const demonstrationSendButtonLabel =+ demonstrationRecorder.phase === "recording"+ ? "Done, build the workflow"+ : "Build the workflow";+ // While recording, typing still in a focused field is only counted once Done flushes it.+ const demonstrationNothingRecorded =+ demonstrationRecorder.phase !== "recording" &&+ demonstrationRecorder.actionCount === 0;+ const demonstrationSendButtonDisabled =+ demonstrationRecorder.busy || demonstrationNothingRecorded;+ const demonstrationSendButtonReason = demonstrationNothingRecorded+ ? "Do something in the browser first"+ : null;const renderQuestionReceipt = (interaction: QuestionInteraction) =>interaction.account_group_review ? (@@ -11197,7 +11545,13 @@ export function WorkflowCopilotChat({messages,index,);- return (+ const messageDemonstrationItems =+ message.demonstrationItems ??+ (message.id === demonstrationMessageIdRef.current &&+ isDemonstrating+ ? demonstrationItems+ : undefined);+ const renderedMessage = (<MessageItemkey={message.id}message={@@ -11272,6 +11626,23 @@ export function WorkflowCopilotChat({}/>);+ return messageDemonstrationItems?.length ? (+ <Fragment key={message.id}>+ {renderedMessage}+ <div className={TURN_ROW_INSET}>+ <WorkPlanCard+ items={messageDemonstrationItems}+ label="Your demonstration"+ current={+ message.id === demonstrationMessageIdRef.current &&+ isDemonstrating+ }+ />+ </div>+ </Fragment>+ ) : (+ renderedMessage+ );})();// A turn's own view places its questions under the step that asked; any other row// lists them after itself, since that row is what asked.@@ -11747,6 +12118,28 @@ export function WorkflowCopilotChat({tray={attentionTray(`account:${dockedAccountChoice.turnId}`)}/>) : null}+ {dockedDemonstrationCredential && demonstrationSensitiveField ? (+ <CredentialCard+ key={`${demonstrationSensitiveField.field}:${demonstrationSensitiveField.selector}:${demonstrationSensitiveField.url}`}+ frame={dockedDemonstrationCredential}+ mode="inline-pause"+ offerCredentials={demonstrationRecorder.canFillCredentials}+ reloadKey={credentialsReloadKey}+ onConnect={(credentialId) => {+ if (credentialId) {+ demonstrationRecorder.fillCredential(credentialId);+ return;+ }+ pendingCredentialConnect.current = null;+ pendingDemonstrationCredentialConnect.current = true;+ setCredentialModalOpen(true);+ }}+ onSkip={demonstrationRecorder.dismissSensitiveField}+ tray={attentionTray(+ `demonstration-credential:${demonstrationSensitiveField.field}:${demonstrationSensitiveField.selector}`,+ )}+ />+ ) : null}{showQueuedStrip && queuedPrompt ? (<QueuedMessageStriptext={queuedPrompt.content}@@ -11770,11 +12163,86 @@ export function WorkflowCopilotChat({onDragLeave={handleComposerDragLeave}onDrop={handleComposerDrop}className={cn(- "relative flex items-end gap-1.5 rounded-lg border border-input bg-slate-elevation2 py-1.5 pl-3 pr-2 transition-colors focus-within:border-ring",- (showQueuedStrip || activeAttention) && "rounded-t-none",- activeAttention && "border-amber-500/50",+ "relative flex flex-col overflow-hidden rounded-lg border border-input bg-slate-elevation2 transition-colors focus-within:border-ring",+ (showQueuedStrip || hasVisibleAttentionTray) && "rounded-t-none",+ hasVisibleAttentionTray && "border-amber-500/50",+ demonstrationRecorder.phase === "recording" && "border-[#a16207]",)}>+ {isDemonstrating ? (+ <div+ role="status"+ className={cn(+ "flex min-h-9 items-center justify-between gap-3 px-3 py-1.5 text-xs",+ demonstrationRecorder.phase === "recording" &&+ "border-b border-[#a16207]/30 bg-[#a16207]/[0.14]",+ )}+ >+ <div className="flex min-w-0 items-center gap-2 font-medium">+ {demonstrationRecorder.phase === "recording" ? (+ <span+ aria-hidden+ className="h-2 w-2 shrink-0 rounded-full bg-red-500"+ />+ ) : (+ <PauseIcon+ aria-hidden+ className="h-3 w-3 shrink-0 text-muted-foreground"+ />+ )}+ <span>+ {demonstrationRecorder.phase === "recording"+ ? "Recording"+ : "Recording paused"}+ {` · ${demonstrationRecorder.actionCount} ${+ demonstrationRecorder.actionCount === 1+ ? "action"+ : "actions"+ }`}+ </span>+ </div>+ <div className="flex shrink-0 items-center gap-3">+ <button+ type="button"+ disabled={demonstrationRecorder.busy}+ onClick={async () => {+ if (!(await demonstrationRecorder.cancel())) return;+ const messageId = demonstrationMessageIdRef.current;+ demonstrationMessageIdRef.current = null;+ setMessages((prev) =>+ prev.filter((message) => message.id !== messageId),+ );+ }}+ className="font-medium text-muted-foreground transition hover:text-foreground disabled:pointer-events-none disabled:opacity-50"+ >+ Discard+ </button>+ {demonstrationRecorder.phase === "stopped" &&+ demonstrationRecorder.demonstrationId &&+ !demonstrationRecorder.limitReached ? (+ <button+ type="button"+ disabled={demonstrationRecorder.busy}+ onClick={() => {+ const demonstrationId =+ demonstrationRecorder.demonstrationId;+ if (demonstrationId) {+ demonstrationRecorder.resume(demonstrationId);+ }+ }}+ className="font-medium text-muted-foreground transition hover:text-foreground disabled:pointer-events-none disabled:opacity-50"+ >+ Resume+ </button>+ ) : null}+ </div>+ </div>+ ) : null}+ {demonstrationRecorder.hasSkippedSensitiveField ? (+ <div className="border-b border-border px-3 py-1.5">+ <RunLifecycleLine content="Sensitive field skipped — its value was not recorded." />+ </div>+ ) : null}{isFileDragging ? (<divrole="status"@@ -11785,165 +12253,208 @@ export function WorkflowCopilotChat({Drop files to attach</div>) : null}- <textarea- ref={setTextareaRef}- placeholder={composerPlaceholder({- queuedPrompt: queuedPrompt- ? queuedPrompt.origin === "typed"- ? "add"- : "replace"- : null,- isLoading,- isWaitingForLiveBrowser,- latestTurnIsAsk,- askPartChoices: !trayPart?.choices.length- ? "none"- : questionStepper.choices[trayPart.part_id] === undefined- ? "unpicked"- : "picked",- detailPrompt: questionStepper.detailPrompt,- })}- value={inputValue}- onChange={(e) => setInputValue(e.target.value)}- onFocus={() => setQuestionTrayCollapsed(false)}- onKeyDown={handleKeyPress}- aria-label={composerQuestion ? "Your response" : undefined}- aria-describedby={- composerQuestion && !questionTrayCollapsed- ? questionStepper.detailPrompt- ? `${QUESTION_PROMPT_ID} ${QUESTION_DETAIL_ID}`- : QUESTION_PROMPT_ID- : undefined- }- disabled={authoringInProgress}- rows={1}- className="min-h-10 flex-1 resize-none border-0 bg-transparent py-2 text-sm leading-6 text-foreground placeholder:truncate placeholder:text-muted-foreground focus:outline-none disabled:cursor-not-allowed disabled:opacity-50"- style={{- minHeight: "40px",- maxHeight: "150px",- overflowY: "hidden",- }}- />- <input- ref={attachmentInputRef}- type="file"- accept={ATTACHMENT_ACCEPT}- className="hidden"- onChange={(event) => {- const file = event.target.files?.[0];- // Reset first: picking the same file twice in a row fires no change event otherwise.- event.target.value = "";- if (file) void uploadAttachment(file);- }}- />- <input- ref={sopFileInputRef}- type="file"- accept=".pdf,application/pdf"- aria-label="Choose an SOP PDF"- className="hidden"- onChange={(event) => {- const file = event.target.files?.[0];- if (!file) return;- event.target.value = "";- if (refuseMutationDuringYamlCommit() || uploadSOPDisabled) return;- if (!file.name.toLowerCase().endsWith(".pdf")) {- toast({- variant: "destructive",- title: "Invalid file type",- description: "Please select a PDF file",- });- return;- }- onUploadSOP?.(file);- }}- />- {/* As tall as one textarea line box, so the controls center on the last line as the textarea grows. */}- <div className="flex h-10 shrink-0 items-center gap-1.5">- <button- type="button"- onClick={() => attachmentInputRef.current?.click()}- // An answer to a pending question is sent through the question flow, which carries- // no files, so offering the control here would stage a file nothing can send.- disabled={hasPendingQuestion || authoringInProgress}- title={- authoringInProgress- ? "Finish the current authoring action before attaching a file"- : hasPendingQuestion- ? "Answer the pending question before attaching a file"- : "Attach a file"+ <div className="flex items-end gap-1.5 py-1.5 pl-3 pr-2">+ <textarea+ ref={setTextareaRef}+ placeholder={composerPlaceholder({+ queuedPrompt: queuedPrompt+ ? queuedPrompt.origin === "typed"+ ? "add"+ : "replace"+ : null,+ isLoading,+ isWaitingForLiveBrowser,+ isDemonstrating,+ latestTurnIsAsk,+ askPartChoices: !trayPart?.choices.length+ ? "none"+ : questionStepper.choices[trayPart.part_id] === undefined+ ? "unpicked"+ : "picked",+ detailPrompt: questionStepper.detailPrompt,+ })}+ value={inputValue}+ onChange={(e) => setInputValue(e.target.value)}+ onFocus={() => setQuestionTrayCollapsed(false)}+ onKeyDown={handleKeyPress}+ aria-label={composerQuestion ? "Your response" : undefined}+ aria-describedby={+ composerQuestion && !questionTrayCollapsed+ ? questionStepper.detailPrompt+ ? `${QUESTION_PROMPT_ID} ${QUESTION_DETAIL_ID}`+ : QUESTION_PROMPT_ID+ : undefined}- aria-label="Attach a file"- className="flex h-8 w-8 shrink-0 items-center justify-center rounded-full text-muted-foreground transition hover:bg-accent hover:text-accent-foreground"- >- <PlusIcon className="h-4 w-4" />- </button>- <SpeechInputButton- isSupported={isSpeechSupported}- isListening={isSpeechListening}- isHearingSpeech={isSpeechHearing}- disabled={authoringInProgress && !isSpeechListening}- onToggle={toggleSpeech}- className="h-8 w-8 rounded-full border-0 bg-transparent"- iconClassName="h-3.5 w-3.5"+ disabled={authoringInProgress || isDemonstrating}+ rows={1}+ className="min-h-10 flex-1 resize-none border-0 bg-transparent py-2 text-sm leading-6 text-foreground placeholder:truncate placeholder:text-muted-foreground focus:outline-none disabled:cursor-not-allowed disabled:opacity-50"+ style={{+ minHeight: "40px",+ maxHeight: "150px",+ overflowY: "hidden",+ }}/>- <TooltipProvider>- <ControlTooltip- content={- acceptHoldReason ? (- <span className="block max-w-xs">{acceptHoldReason}</span>- ) : (- morphButtonLabel- )+ <input+ ref={attachmentInputRef}+ type="file"+ accept={ATTACHMENT_ACCEPT}+ className="hidden"+ onChange={(event) => {+ const file = event.target.files?.[0];+ // Reset first: picking the same file twice in a row fires no change event otherwise.+ event.target.value = "";+ if (file) void uploadAttachment(file);+ }}+ />+ <input+ ref={sopFileInputRef}+ type="file"+ accept=".pdf,application/pdf"+ aria-label="Choose an SOP PDF"+ className="hidden"+ onChange={(event) => {+ const file = event.target.files?.[0];+ if (!file) return;+ event.target.value = "";+ if (refuseMutationDuringYamlCommit() || uploadSOPDisabled)+ return;+ if (!file.name.toLowerCase().endsWith(".pdf")) {+ toast({+ variant: "destructive",+ title: "Invalid file type",+ description: "Please select a PDF file",+ });+ return;}- blocked={- stopBlockedByDeletion ||- waitingOnQueueOnly ||- acceptUnresolved ||- authoringBlocksComposerAction ||- questionAnswerMissing+ onUploadSOP?.(file);+ }}+ />+ {/* As tall as one textarea line box, so the controls center on the last line as the textarea grows. */}+ <div className="flex h-10 shrink-0 items-center gap-1.5">+ <button+ type="button"+ onClick={() => attachmentInputRef.current?.click()}+ // An answer to a pending question is sent through the question flow, which carries+ // no files, so offering the control here would stage a file nothing can send.+ disabled={+ hasPendingQuestion || authoringInProgress || isDemonstrating+ }+ title={+ isDemonstrating+ ? demonstrationRecorder.phase === "recording"+ ? "Use Done before attaching a file"+ : "Use Build before attaching a file"+ : authoringInProgress+ ? "Finish the current authoring action before attaching a file"+ : hasPendingQuestion+ ? "Answer the pending question before attaching a file"+ : "Attach a file"}+ aria-label="Attach a file"+ className="flex h-8 w-8 shrink-0 items-center justify-center rounded-full text-muted-foreground transition hover:bg-accent hover:text-accent-foreground disabled:opacity-50">- <button- type="button"- disabled={- stopBlockedByDeletion ||- waitingOnQueueOnly ||- isStopping ||- acceptUnresolved ||- authoringBlocksComposerAction ||- questionAnswerMissing+ <PlusIcon className="h-4 w-4" />+ </button>+ <SpeechInputButton+ isSupported={isSpeechSupported}+ isListening={isSpeechListening}+ isHearingSpeech={isSpeechHearing}+ disabled={+ isDemonstrating || (authoringInProgress && !isSpeechListening)+ }+ onToggle={toggleSpeech}+ className="h-8 w-8 rounded-full border-0 bg-transparent"+ iconClassName="h-3.5 w-3.5"+ />+ <TooltipProvider>+ <ControlTooltip+ content={+ isDemonstrating ? (+ demonstrationSendButtonLabel+ ) : acceptHoldReason ? (+ <span className="block max-w-xs">{acceptHoldReason}</span>+ ) : (+ morphButtonLabel+ )}- aria-busy={isStopping}- onClick={() =>- turnObservablyRunning && !hasComposerText- ? cancelSend("stop_button")- : handleSend()+ reason={+ isDemonstrating ? demonstrationSendButtonReason : undefined+ }+ blocked={+ isDemonstrating+ ? demonstrationSendButtonDisabled+ : stopBlockedByDeletion ||+ waitingOnQueueOnly ||+ acceptUnresolved ||+ authoringBlocksComposerAction ||+ questionAnswerMissing}- aria-label={morphButtonLabel}- className="relative flex h-9 w-9 shrink-0 items-center justify-center overflow-hidden rounded-lg bg-cta text-cta-foreground transition hover:bg-cta-hover focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring active:scale-[0.92] disabled:pointer-events-none disabled:cursor-not-allowed disabled:opacity-50">- {showsStopGlyph || morphButtonStarting ? (- <span- aria-hidden- data-testid="copilot-stop-ring"- className={cn(- "absolute h-[22px] w-[22px] animate-spin rounded-full border-2 border-[color-mix(in_srgb,currentColor_20%,transparent)] border-t-current motion-reduce:animate-none",- isStopping && "paused",- )}- />- ) : null}- {showsStopGlyph ? (- <span- aria-hidden- className="h-2 w-2 rounded-[1.5px] bg-current"- />- ) : (- <ArrowUpIcon className="h-4 w-4" />- )}- </button>- </ControlTooltip>- </TooltipProvider>+ <button+ type="button"+ disabled={+ isDemonstrating+ ? demonstrationSendButtonDisabled+ : stopBlockedByDeletion ||+ waitingOnQueueOnly ||+ isStopping ||+ acceptUnresolved ||+ authoringBlocksComposerAction ||+ questionAnswerMissing+ }+ aria-busy={+ isDemonstrating ? demonstrationRecorder.busy : isStopping+ }+ onClick={() => {+ if (isDemonstrating) {+ void demonstrationRecorder.done();+ return;+ }+ if (turnObservablyRunning && !hasComposerText) {+ void cancelSend("stop_button");+ return;+ }+ void handleSend();+ }}+ aria-label={+ isDemonstrating+ ? demonstrationSendButtonLabel+ : morphButtonLabel+ }+ className="relative flex h-9 w-9 shrink-0 items-center justify-center overflow-hidden rounded-lg bg-cta text-cta-foreground transition hover:bg-cta-hover focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring active:scale-[0.92] disabled:pointer-events-none disabled:cursor-not-allowed disabled:opacity-50"+ >+ {!isDemonstrating &&+ (showsStopGlyph || morphButtonStarting) ? (+ <span+ aria-hidden+ data-testid="copilot-stop-ring"+ className={cn(+ "absolute h-[22px] w-[22px] animate-spin rounded-full border-2 border-[color-mix(in_srgb,currentColor_20%,transparent)] border-t-current motion-reduce:animate-none",+ isStopping && "paused",+ )}+ />+ ) : null}+ {isDemonstrating ? (+ demonstrationRecorder.phase === "recording" ? (+ <span+ aria-hidden+ className="h-2 w-2 rounded-[1.5px] bg-current"+ />+ ) : (+ <ArrowUpIcon className="h-4 w-4" />+ )+ ) : showsStopGlyph ? (+ <span+ aria-hidden+ className="h-2 w-2 rounded-[1.5px] bg-current"+ />+ ) : (+ <ArrowUpIcon className="h-4 w-4" />+ )}+ </button>+ </ControlTooltip>+ </TooltipProvider>+ </div></div></div></div>@@ -12007,7 +12518,9 @@ export function WorkflowCopilotChat({// credential matches later asks. Terminal frames carry no URL — the// field stays empty then.defaultTestUrl={- pendingCredentialConnect.current?.frame?.login_page_urls?.[0]+ pendingDemonstrationCredentialConnect.current+ ? demonstrationSensitiveField?.url+ : pendingCredentialConnect.current?.frame?.login_page_urls?.[0]}editingCredential={pendingCredentialConnect.current?.editingCredential@@ -12020,9 +12533,21 @@ export function WorkflowCopilotChat({}onOpenChange={(open) => {setCredentialModalOpen(open);- if (!open) pendingCredentialConnect.current = null;+ if (!open) {+ pendingCredentialConnect.current = null;+ pendingDemonstrationCredentialConnect.current = false;+ }+ }}+ onCredentialCreated={(credentialId, name) => {+ if (pendingDemonstrationCredentialConnect.current) {+ pendingDemonstrationCredentialConnect.current = false;+ setCredentialModalOpen(false);+ setCredentialsReloadKey((key) => key + 1);+ demonstrationRecorder.fillCredential(credentialId);+ return;+ }+ handleCredentialCreated(credentialId, name);}}- onCredentialCreated={handleCredentialCreated}/>) : null}</div>diff --git a/skyvern-frontend/src/routes/workflows/copilot/cards/CredentialCard.tsx b/skyvern-frontend/src/routes/workflows/copilot/cards/CredentialCard.tsxindex 33b178b32..de31471ed 100644--- a/skyvern-frontend/src/routes/workflows/copilot/cards/CredentialCard.tsx+++ b/skyvern-frontend/src/routes/workflows/copilot/cards/CredentialCard.tsx@@ -146,6 +146,7 @@ export interface CredentialCardProps {signIn?: ManualSignInOffer;// Registration cards only: saves a server-generated password under the frame's username and name.onGenerate?: () => void;+ offerCredentials?: boolean;}export interface ManualSignInOffer {@@ -790,12 +791,14 @@ function CredentialAskCard({tray,signIn,onGenerate,+ offerCredentials = true,}: Readonly<CredentialCardProps>) {// Terminal mode never expires by design: its signal carries no timeout/expiry// semantics at all, so there is nothing to compare "now" against. Only a// richer, timed pause signal has real expiry data, hence gating disablement// on inline-pause exclusively.- const countdownActive = mode === "inline-pause" && !resolvedOutcome;+ const countdownActive =+ mode === "inline-pause" && !resolvedOutcome && Boolean(frame.expires_at);const { remainingMs, expired } = useCountdown(frame.expires_at ?? "",countdownActive,@@ -834,7 +837,8 @@ function CredentialAskCard({const isAsk = !resolvedOutcome;// An auto-bound receipt only needs the org list when its Change affordance is live; a read-only// scrollback receipt must not each fire its own 100-credential fetch.- const needsCredentialList = isAsk && (mode !== "auto-bound" || canChange);+ const needsCredentialList =+ offerCredentials && isAsk && (mode !== "auto-bound" || canChange);useEffect(() => {if (!needsCredentialList) return;// Generation guard: a fetch superseded by a reloadKey change (or a slow one) must not set state@@ -889,7 +893,7 @@ function CredentialAskCard({// Fetched by id rather than read from the list above: that list is one page narrowed by the// search box, so either can drop the one login the ask is about.const namedCredentialId =- isAsk && mode !== "auto-bound"+ offerCredentials && isAsk && mode !== "auto-bound"? (frame.named_credential_id ?? undefined): undefined;const [namedCredential, setNamedCredential] =@@ -1204,7 +1208,7 @@ function CredentialAskCard({>{signIn.busy ? "Saving sign-in…" : "Done"}</Button>- {connectButton}+ {offerCredentials ? connectButton : null}<div className="ml-auto"><Buttontype="button"@@ -1299,7 +1303,7 @@ function CredentialAskCard({Generate and save</Button>) : null}- {namedCredential ? (+ {offerCredentials && namedCredential ? (<Buttontype="button"size="sm"@@ -1312,13 +1316,14 @@ function CredentialAskCard({>Use '{namedCredential.name}' on {site}</Button>- ) : namedLookupPending ? (+ ) : offerCredentials && namedLookupPending ? (<Button type="button" size="sm" disabled>Use saved login</Button>) : null}- {connectButton}- {orgCredentials.status === "ready" &&+ {offerCredentials ? connectButton : null}+ {offerCredentials &&+ orgCredentials.status === "ready" &&(pickable.length > 0 || pickerEngaged) ? (<CredentialPickercredentials={pickable}@@ -1333,12 +1338,12 @@ function CredentialAskCard({onRetrySearch={retrySearch}/>) : null}- {orgCredentials.status === "loading" ? (+ {offerCredentials && orgCredentials.status === "loading" ? (<span className="text-xs text-muted-foreground">Loading saved logins…</span>) : null}- {orgCredentials.status === "error" ? (+ {offerCredentials && orgCredentials.status === "error" ? (<><span className="text-xs text-muted-foreground">Couldn't load your saved logins.@@ -1372,13 +1377,13 @@ function CredentialAskCard({className="sr-only"role={insideLiveRegion ? undefined : "status"}>- {searchFailed+ {offerCredentials && searchFailed? SEARCH_FAILED_ANNOUNCEMENT: insideLiveRegion? ""- : orgCredentials.status === "loading"+ : offerCredentials && orgCredentials.status === "loading"? "Loading saved logins…"- : orgCredentials.status === "error"+ : offerCredentials && orgCredentials.status === "error"? "Couldn't load your saved logins.": ""}</span>diff --git a/skyvern-frontend/src/routes/workflows/copilot/cards/WorkPlanCard.tsx b/skyvern-frontend/src/routes/workflows/copilot/cards/WorkPlanCard.tsxindex bfba2eb81..a7808a93a 100644--- a/skyvern-frontend/src/routes/workflows/copilot/cards/WorkPlanCard.tsx+++ b/skyvern-frontend/src/routes/workflows/copilot/cards/WorkPlanCard.tsx@@ -5,6 +5,7 @@ import { CardBody, CardHeader, CopilotCard, GutterRow } from "./cardChrome";interface WorkPlanCardProps {items: string[];+ label?: string;// The plan this one replaced. A step counts as new only when no earlier step reads exactly the// same: a reworded step is not recognizable without guessing, so it shows as new plus removed.previous?: string[] | null;@@ -18,6 +19,7 @@ function stepCount(count: number): string {export function WorkPlanCard({items,+ label,previous = null,current = true,}: WorkPlanCardProps) {@@ -51,12 +53,12 @@ export function WorkPlanCard({].filter(Boolean);return (<CopilotCard>- <div role="group" aria-label="Copilot's plan">+ <div role="group" aria-label={label ?? "Copilot's plan"}><CardHeadericon={<ListBulletIcon className="h-3.5 w-3.5 text-muted-foreground" />}- title={changes.length > 0 ? "Plan updated" : "Plan"}+ title={label ?? (changes.length > 0 ? "Plan updated" : "Plan")}meta={[stepCount(items.length), ...changes].join(" · ")}expanded={expanded}onToggle={() => setExpanded((value) => !value)}diff --git a/skyvern-frontend/src/routes/workflows/copilot/composerPlaceholder.test.ts b/skyvern-frontend/src/routes/workflows/copilot/composerPlaceholder.test.tsindex 95109ff67..24fb2fc6c 100644--- a/skyvern-frontend/src/routes/workflows/copilot/composerPlaceholder.test.ts+++ b/skyvern-frontend/src/routes/workflows/copilot/composerPlaceholder.test.ts@@ -51,5 +51,8 @@ describe("composerPlaceholder", () => {expect(composerPlaceholder({ ...base, latestTurnIsAsk: true, isLoading: true }),).toBe("Type to queue a message…");+ expect(composerPlaceholder({ ...base, isDemonstrating: true })).toBe(+ "Show Copilot the task",+ );});});diff --git a/skyvern-frontend/src/routes/workflows/copilot/composerPlaceholder.ts b/skyvern-frontend/src/routes/workflows/copilot/composerPlaceholder.tsindex 89dca1cbe..c30b65344 100644--- a/skyvern-frontend/src/routes/workflows/copilot/composerPlaceholder.ts+++ b/skyvern-frontend/src/routes/workflows/copilot/composerPlaceholder.ts@@ -4,6 +4,7 @@ export function composerPlaceholder({queuedPrompt,isLoading,isWaitingForLiveBrowser,+ isDemonstrating = false,latestTurnIsAsk,askPartChoices = "none",detailPrompt = null,@@ -12,6 +13,7 @@ export function composerPlaceholder({queuedPrompt: "add" | "replace" | null;isLoading: boolean;isWaitingForLiveBrowser: boolean;+ isDemonstrating?: boolean;latestTurnIsAsk: boolean;// The question on screen's choices: without any the composer is the answer, and typing never// clears a picked one, so text sent after a pick goes alongside it rather than replacing it.@@ -19,6 +21,7 @@ export function composerPlaceholder({// What the picked choice still needs typed, which outranks the generic invitations.detailPrompt?: string | null;}): string {+ if (isDemonstrating) return "Show Copilot the task";if (queuedPrompt === "add") return "Add to the queued message…";if (queuedPrompt === "replace") return "Type to replace the queued message…";if (isLoading) return "Type to queue a message…";diff --git a/skyvern-frontend/src/routes/workflows/copilot/useDemonstrationRecorder.ts b/skyvern-frontend/src/routes/workflows/copilot/useDemonstrationRecorder.tsnew file mode 100644index 000000000..563a0b050--- /dev/null+++ b/skyvern-frontend/src/routes/workflows/copilot/useDemonstrationRecorder.ts@@ -0,0 +1,478 @@+import { isAxiosError } from "axios";+import { useCallback, useEffect, useRef, useState } from "react";++import { getClient } from "@/api/AxiosClient";+import { getSseClient } from "@/api/sse";+import { toast } from "@/components/ui/use-toast";+import { useCredentialGetter } from "@/hooks/useCredentialGetter";++import type {+ WorkflowCopilotActiveDemonstration,+ WorkflowCopilotDemonstrationActionUpdate,+ WorkflowCopilotDemonstrationControlRequest,+ WorkflowCopilotDemonstrationControlResult,+ WorkflowCopilotDemonstrationInsertion,+ WorkflowCopilotDemonstrationSensitiveFieldUpdate,+ WorkflowCopilotDemonstrationUpdate,+} from "./workflowCopilotTypes";++type DemonstrationPhase = "idle" | "recording" | "stopped";++type DemonstrationAction = Pick<+ WorkflowCopilotDemonstrationActionUpdate,+ "seq" | "label"+>;++type DemonstrationDone = {+ demonstrationId: string;+ actions: DemonstrationAction[];+};++type UseDemonstrationRecorderOptions = {+ workflowPermanentId: string | undefined;+ browserSessionId: string | null | undefined;+ onDone: (demonstration: DemonstrationDone) => boolean | Promise<boolean>;+ onStartFailed: () => void;+};++export function useDemonstrationRecorder({+ workflowPermanentId,+ browserSessionId,+ onDone,+ onStartFailed,+}: UseDemonstrationRecorderOptions) {+ const credentialGetter = useCredentialGetter();+ const [demonstrationId, setDemonstrationId] = useState<string | null>(null);+ const [phase, setPhase] = useState<DemonstrationPhase>("idle");+ const [actions, setActions] = useState<DemonstrationAction[]>([]);+ const actionsRef = useRef<DemonstrationAction[]>([]);+ const [actionCount, setActionCount] = useState(0);+ const [sensitiveField, setSensitiveField] =+ useState<WorkflowCopilotDemonstrationSensitiveFieldUpdate | null>(null);+ const [busy, setBusy] = useState(false);+ const [canFillCredentials, setCanFillCredentials] = useState(false);+ const [hasSkippedSensitiveField, setHasSkippedSensitiveField] =+ useState(false);+ const [limitReached, setLimitReached] = useState(false);+ const busyRef = useRef(false);+ const streamControllerRef = useRef<AbortController | null>(null);+ const demonstrationIdRef = useRef<string | null>(null);+ const phaseRef = useRef<DemonstrationPhase>("idle");+ const insertionRef = useRef<WorkflowCopilotDemonstrationInsertion>();+ const onDoneRef = useRef(onDone);+ onDoneRef.current = onDone;+ const onStartFailedRef = useRef(onStartFailed);+ onStartFailedRef.current = onStartFailed;++ const updateBusy = useCallback((next: boolean) => {+ busyRef.current = next;+ setBusy(next);+ }, []);++ const updatePhase = useCallback((next: DemonstrationPhase) => {+ phaseRef.current = next;+ setPhase(next);+ }, []);++ const updateDemonstrationId = useCallback((next: string | null) => {+ demonstrationIdRef.current = next;+ setDemonstrationId(next);+ }, []);++ const clear = useCallback(() => {+ streamControllerRef.current?.abort();+ streamControllerRef.current = null;+ updateDemonstrationId(null);+ updatePhase("idle");+ actionsRef.current = [];+ setActions([]);+ setActionCount(0);+ setSensitiveField(null);+ setCanFillCredentials(false);+ setHasSkippedSensitiveField(false);+ setLimitReached(false);+ updateBusy(false);+ insertionRef.current = undefined;+ }, [updateBusy, updateDemonstrationId, updatePhase]);++ const handleStreamFrame = useCallback(+ (frame: WorkflowCopilotDemonstrationUpdate): boolean => {+ switch (frame.type) {+ case "demonstration_started":+ updateDemonstrationId(frame.demonstration_id);+ setActionCount(frame.action_count);+ setCanFillCredentials(frame.can_fill_credentials);+ updatePhase("recording");+ updateBusy(false);+ return false;+ case "demonstration_action":+ actionsRef.current = [+ ...actionsRef.current.filter((action) => action.seq !== frame.seq),+ frame,+ ].sort((left, right) => left.seq - right.seq);+ setActions(actionsRef.current);+ setActionCount((current) => Math.max(current, frame.seq));+ setSensitiveField(null);+ return false;+ case "demonstration_sensitive_field":+ {+ const seq =+ Math.max(0, ...actionsRef.current.map((action) => action.seq)) ++ 1;+ const fieldLabel =+ frame.field === "other" ? "sensitive field" : frame.field;+ actionsRef.current = [+ ...actionsRef.current,+ {+ seq,+ label: `Fill ${fieldLabel} (value not recorded)`,+ },+ ];+ setActions(actionsRef.current);+ setActionCount((current) => Math.max(current, seq));+ }+ if (frame.field === "other") {+ setHasSkippedSensitiveField(true);+ }+ setSensitiveField(frame);+ return false;+ case "demonstration_notice":+ if (frame.level === "error") {+ toast({+ title: "Could not update the demonstration",+ description: frame.message,+ variant: "destructive",+ });+ }+ return false;+ case "demonstration_stopped":+ if (frame.demonstration_id) {+ updateDemonstrationId(frame.demonstration_id);+ }+ setActionCount(frame.action_count);+ setSensitiveField(null);+ setHasSkippedSensitiveField(false);+ if (frame.reason === "time_limit") setLimitReached(true);+ if (frame.reason === "event_limit") {+ setLimitReached(true);+ toast({+ title: "This recording reached its size limit",+ description:+ "Build the workflow from what was recorded, or discard it.",+ });+ }+ updatePhase("stopped");+ return true;+ }+ },+ [updateBusy, updateDemonstrationId, updatePhase],+ );++ const openStream = useCallback(+ async (resumeId?: string) => {+ if (!workflowPermanentId || !browserSessionId) return;+ streamControllerRef.current?.abort();+ const controller = new AbortController();+ streamControllerRef.current = controller;+ updateBusy(true);+ if (resumeId) updateDemonstrationId(resumeId);+ try {+ const client = await getSseClient(credentialGetter);+ if (controller.signal.aborted) return;+ await client.postStreaming<WorkflowCopilotDemonstrationUpdate>(+ "/workflow/copilot/demonstration/record",+ {+ workflow_permanent_id: workflowPermanentId,+ browser_session_id: browserSessionId,+ ...(resumeId ? { demonstration_id: resumeId } : {}),+ ...(!resumeId && insertionRef.current+ ? { insertion: insertionRef.current }+ : {}),+ },+ handleStreamFrame,+ { signal: controller.signal },+ );+ } catch (error) {+ if (controller.signal.aborted) return;+ if (demonstrationIdRef.current) {+ updatePhase("stopped");+ } else {+ updatePhase("idle");+ if (!resumeId) onStartFailedRef.current();+ }+ toast({+ title: resumeId+ ? "Could not resume the demonstration"+ : "Could not start the demonstration",+ description:+ error instanceof Error ? error.message : "Please try again.",+ variant: "destructive",+ });+ } finally {+ if (streamControllerRef.current === controller) {+ streamControllerRef.current = null;+ updateBusy(false);+ }+ }+ },+ [+ browserSessionId,+ credentialGetter,+ handleStreamFrame,+ updateBusy,+ updateDemonstrationId,+ updatePhase,+ workflowPermanentId,+ ],+ );++ const start = useCallback(+ ({+ insertion,+ }: { insertion?: WorkflowCopilotDemonstrationInsertion } = {}) => {+ if (busyRef.current) return;+ actionsRef.current = [];+ setActions([]);+ setActionCount(0);+ setSensitiveField(null);+ setCanFillCredentials(false);+ setHasSkippedSensitiveField(false);+ setLimitReached(false);+ insertionRef.current = insertion;+ void openStream();+ },+ [openStream],+ );++ const resume = useCallback(+ (id: string) => {+ if (busyRef.current) return;+ void openStream(id);+ },+ [openStream],+ );++ const sendControl = useCallback(+ async (+ request: Omit<+ WorkflowCopilotDemonstrationControlRequest,+ "demonstration_id"+ >,+ id = demonstrationIdRef.current,+ ) => {+ if (!id) throw new Error("No demonstration is active.");+ const client = await getClient(credentialGetter, "sans-api-v1");+ const response =+ await client.post<WorkflowCopilotDemonstrationControlResult>(+ "/workflow/copilot/demonstration/control",+ { demonstration_id: id, ...request },+ );+ return response.data;+ },+ [credentialGetter],+ );++ const runControl = useCallback(+ async (+ request: Omit<+ WorkflowCopilotDemonstrationControlRequest,+ "demonstration_id"+ >,+ ) => {+ if (busyRef.current) return;+ updateBusy(true);+ try {+ await sendControl(request);+ } catch (error) {+ toast({+ title: "Could not update the demonstration",+ description:+ error instanceof Error ? error.message : "Please try again.",+ variant: "destructive",+ });+ } finally {+ updateBusy(false);+ }+ },+ [sendControl, updateBusy],+ );++ const navigate = useCallback(+ (url: string) => runControl({ action: "navigate", url }),+ [runControl],+ );++ const fillCredential = useCallback(+ (credentialId: string) => {+ if (+ !sensitiveField ||+ (sensitiveField.field !== "password" && sensitiveField.field !== "totp")+ )+ return;+ void runControl({+ action: "fill_credential",+ credential_id: credentialId,+ selector: sensitiveField.selector,+ field: sensitiveField.field,+ });+ },+ [runControl, sensitiveField],+ );++ const dismissSensitiveField = useCallback(() => {+ setSensitiveField(null);+ }, []);++ const done = useCallback(async () => {+ const id = demonstrationIdRef.current;+ if (!id || busyRef.current) return;+ updateBusy(true);+ try {+ if (phaseRef.current === "recording") {+ const stopped = await sendControl({ action: "stop" }, id);+ streamControllerRef.current?.abort();+ streamControllerRef.current = null;+ setSensitiveField(null);+ updatePhase("stopped");+ setHasSkippedSensitiveField(false);+ if (stopped?.action_count === 0) {+ setActionCount(0);+ toast({ title: "Nothing was recorded yet" });+ updateBusy(false);+ return;+ }+ }+ const recordedActions = actionsRef.current;+ // The send resolves only when the whole turn ends; the recording reads as finished from here.+ updatePhase("idle");+ const sent = await onDoneRef.current({+ demonstrationId: id,+ actions: recordedActions,+ });+ if (sent) {+ clear();+ } else {+ updatePhase("stopped");+ updateBusy(false);+ }+ } catch (error) {+ toast({+ title: "Could not finish the demonstration",+ description:+ error instanceof Error ? error.message : "Please try again.",+ variant: "destructive",+ });+ if (phaseRef.current === "idle") updatePhase("stopped");+ updateBusy(false);+ }+ }, [clear, sendControl, updateBusy, updatePhase]);++ const cancel = useCallback(async (): Promise<boolean> => {+ const id = demonstrationIdRef.current;+ if (!id || busyRef.current) return false;+ updateBusy(true);+ try {+ await sendControl({ action: "cancel" }, id);+ } catch (error) {+ const status = isAxiosError(error) ? error.response?.status : undefined;+ if (status === 404 || status === 409) {+ clear();+ return true;+ }+ toast({+ title: "Could not discard the demonstration",+ description:+ error instanceof Error ? error.message : "Please try again.",+ variant: "destructive",+ });+ updateBusy(false);+ return false;+ }+ clear();+ return true;+ }, [clear, sendControl, updateBusy]);++ useEffect(() => {+ clear();+ if (!workflowPermanentId || !browserSessionId) return;+ const controller = new AbortController();+ void (async () => {+ try {+ const client = await getClient(credentialGetter, "sans-api-v1");+ const response =+ await client.get<WorkflowCopilotActiveDemonstration | null>(+ "/workflow/copilot/demonstration/active",+ {+ params: {+ workflow_permanent_id: workflowPermanentId,+ browser_session_id: browserSessionId,+ },+ signal: controller.signal,+ },+ );+ if (+ !response.data ||+ typeof response.data.demonstration_id !== "string" ||+ typeof response.data.action_count !== "number" ||+ (response.data.status !== "recording" &&+ response.data.status !== "stopping" &&+ response.data.status !== "stopped") ||+ controller.signal.aborted+ )+ return;+ updateDemonstrationId(response.data.demonstration_id);+ setActionCount(response.data.action_count);+ actionsRef.current = response.data.action_labels.map(+ (label, index) => ({+ seq: index + 1,+ label,+ }),+ );+ setActions(actionsRef.current);+ setCanFillCredentials(response.data.can_fill_credentials);+ setLimitReached(!response.data.can_resume);+ insertionRef.current = response.data.insertion ?? undefined;+ updatePhase("stopped");+ } catch {+ return;+ }+ })();+ return () => {+ controller.abort();+ streamControllerRef.current?.abort();+ };+ }, [+ browserSessionId,+ clear,+ credentialGetter,+ updateDemonstrationId,+ updatePhase,+ workflowPermanentId,+ ]);++ useEffect(+ () => () => {+ streamControllerRef.current?.abort();+ },+ [],+ );++ return {+ demonstrationId,+ phase,+ actions,+ actionCount,+ sensitiveField,+ canFillCredentials,+ hasSkippedSensitiveField,+ limitReached,+ busy,+ start,+ resume,+ navigate,+ fillCredential,+ dismissSensitiveField,+ done,+ cancel,+ };+}diff --git a/skyvern-frontend/src/routes/workflows/copilot/workflowCopilotTypes.ts b/skyvern-frontend/src/routes/workflows/copilot/workflowCopilotTypes.tsindex 305050b51..262c5a7ae 100644--- a/skyvern-frontend/src/routes/workflows/copilot/workflowCopilotTypes.ts+++ b/skyvern-frontend/src/routes/workflows/copilot/workflowCopilotTypes.ts@@ -30,6 +30,11 @@ export type CopilotProductAction =kind: "refine_recording";nonce: string;};++export interface WorkflowCopilotDemonstrationInsertion {+ after_label: string | null;+ before_label: string | null;+}export type ProposalDisposition =| "no_proposal"| "auto_applicable"@@ -227,7 +232,9 @@ export interface WorkflowCopilotChatRequest {| "test_end_to_end"| "diagnose_run"| "refine_recording"+ | "demonstrate"| null;+ demonstration_id?: string | null;// Required by the refine_recording action; reaches the model as untrusted// evidence, never as the turn's message.recording_evidence?: RecordingEvidencePacket | null;@@ -554,6 +561,87 @@ export interface WorkflowCopilotCredentialRequiredUpdate {timestamp: string;}+export interface WorkflowCopilotDemonstrationStartedUpdate {+ type: "demonstration_started";+ demonstration_id: string;+ browser_session_id: string;+ action_count: number;+ resumed: boolean;+ can_fill_credentials: boolean;+}++export interface WorkflowCopilotDemonstrationActionUpdate {+ type: "demonstration_action";+ seq: number;+ label: string;+ tool_name:+ | "click"+ | "type_text"+ | "select_option"+ | "press_key"+ | "navigate_browser"+ | "upload_attached_file"+ | "fill_credential_field";+}++export interface WorkflowCopilotDemonstrationSensitiveFieldUpdate {+ type: "demonstration_sensitive_field";+ field: "password" | "totp" | "other";+ selector: string;+ url: string;+}++export interface WorkflowCopilotDemonstrationNoticeUpdate {+ type: "demonstration_notice";+ message: string;+ level: "info" | "error";+}++export interface WorkflowCopilotDemonstrationStoppedUpdate {+ type: "demonstration_stopped";+ demonstration_id?: string;+ reason:+ | "stopped"+ | "cancelled"+ | "disconnected"+ | "browser_gone"+ | "time_limit"+ | "event_limit"+ | "consumed";+ action_count: number;+}++export interface WorkflowCopilotDemonstrationControlRequest {+ demonstration_id: string;+ action: "navigate" | "fill_credential" | "stop" | "cancel";+ url?: string | null;+ selector?: string | null;+ credential_id?: string | null;+ field?: "password" | "totp" | null;+}++export interface WorkflowCopilotDemonstrationControlResult {+ seq: number;+ action_count?: number | null;+}++export type WorkflowCopilotDemonstrationUpdate =+ | WorkflowCopilotDemonstrationStartedUpdate+ | WorkflowCopilotDemonstrationActionUpdate+ | WorkflowCopilotDemonstrationSensitiveFieldUpdate+ | WorkflowCopilotDemonstrationNoticeUpdate+ | WorkflowCopilotDemonstrationStoppedUpdate;++export interface WorkflowCopilotActiveDemonstration {+ demonstration_id: string;+ status: "recording" | "stopping" | "stopped";+ action_count: number;+ action_labels: string[];+ can_fill_credentials: boolean;+ can_resume: boolean;+ insertion?: WorkflowCopilotDemonstrationInsertion | null;+}+export interface WorkflowCopilotCredentialResponseResult {result: "accepted" | "signed_in" | "no_sign_in_found" | "save_failed";expires_at?: string | null;diff --git a/skyvern-frontend/src/routes/workflows/editor/WorkflowAddMenu.tsx b/skyvern-frontend/src/routes/workflows/editor/WorkflowAddMenu.tsxindex 98c25e086..516402762 100644--- a/skyvern-frontend/src/routes/workflows/editor/WorkflowAddMenu.tsx+++ b/skyvern-frontend/src/routes/workflows/editor/WorkflowAddMenu.tsx@@ -14,6 +14,7 @@ type WorkflowAddMenuProps = {rotateText?: boolean;startAt?: number;isUploadingSOP?: boolean;+ canRecordTask?: boolean;// --onAdd: () => void;onRecord: () => void;@@ -29,6 +30,7 @@ function WorkflowAddMenu({rotateText = true,startAt = 90,isUploadingSOP = false,+ canRecordTask = true,// --onAdd,onRecord,@@ -58,7 +60,10 @@ function WorkflowAddMenu({menuItems.push({id: "2",icon: <SquareIcon className={buttonSize ? "h-3 w-3" : undefined} />,- enabled: settingsStore.isUsingABrowser && !recordingStore.isRecording,+ enabled:+ canRecordTask &&+ settingsStore.isUsingABrowser &&+ !recordingStore.isRecording,text: "Record Task",onClick: () => {onRecord();@@ -80,6 +85,7 @@ function WorkflowAddMenu({return menuItems;}, [buttonSize,+ canRecordTask,onAdd,onRecord,onUploadSOP,diff --git a/skyvern-frontend/src/routes/workflows/editor/Workspace.tsx b/skyvern-frontend/src/routes/workflows/editor/Workspace.tsxindex 0fb8966fc..943939357 100644--- a/skyvern-frontend/src/routes/workflows/editor/Workspace.tsx+++ b/skyvern-frontend/src/routes/workflows/editor/Workspace.tsx@@ -257,10 +257,13 @@ import {import { useStudioShellContext } from "../studio/StudioShellContext";import { StudioShellPanelPortal } from "../studio/StudioShellPanelPortal";import { useRecordingLauncherStore } from "@/store/useRecordingLauncherStore";+import { useFeatureFlag } from "@/hooks/useFeatureFlag";+import { WORKFLOW_COPILOT_DEMONSTRATION_FLAG } from "@/util/featureFlags";import type { CopilotAttachedFile } from "@/routes/workflows/copilot/workflowCopilotTypes";import { useSopToBlocksMutation } from "../hooks/useSopToBlocksMutation";import {applySopResultAtCurrentAppend,+ demonstrationInsertionFromNodes,resolveAppendInsertionPoint,resolveWorkspaceAuthoringActionAvailability,} from "./workspaceAuthoringActions";@@ -1782,6 +1785,11 @@ function Workspace({const setStartRecordingAtEnd = useRecordingLauncherStore((s) => s.setStartRecordingAtEnd,);+ const startDemonstration = useRecordingLauncherStore(+ (s) => s.startDemonstration,+ );+ const demonstrationEnabled =+ useFeatureFlag(WORKFLOW_COPILOT_DEMONSTRATION_FLAG) === true;// Stable action ref (vs the whole-store `recordingStore` object, which gets a// new reference on every store write and would re-register the launcher).const setIsRecording = useRecordingStore((s) => s.setIsRecording);@@ -1833,6 +1841,26 @@ function Workspace({);const startRecordingAtEnd = useCallback(() => {if (!authoringActionAvailability.canRecordTask) return;+ if (demonstrationEnabled) {+ if (!startDemonstration) return;+ const insertion = demonstrationInsertionFromNodes(+ getNodes() as Array<AppNode>,+ getAppendInsertionPoint(),+ );+ if (!insertion) return;+ startDemonstration({ insertion });+ if (autoRecordRequested) {+ setRecordSearchParams(+ (current) => {+ const next = new URLSearchParams(current);+ next.delete("record");+ return next;+ },+ { replace: true },+ );+ }+ return;+ }// Consume ?record=1 up front so a dismissed prompt does not re-fire auto-record.if (autoRecordRequested) {setRecordSearchParams(@@ -1875,6 +1903,9 @@ function Workspace({authoringActionAvailability.canRecordTask,autoRecordRequested,setRecordSearchParams,+ demonstrationEnabled,+ startDemonstration,+ getNodes,]);const uploadSOPAtEnd = useCallback((file: File) => {@@ -1906,7 +1937,10 @@ function Workspace({return;}setStartRecordingAtEnd(- authoringActionAvailability.canRecordTask ? startRecordingAtEnd : null,+ authoringActionAvailability.canRecordTask &&+ (!demonstrationEnabled || startDemonstration)+ ? startRecordingAtEnd+ : null,);return () => setStartRecordingAtEnd(null);}, [@@ -1914,6 +1948,8 @@ function Workspace({embedded,startRecordingAtEnd,setStartRecordingAtEnd,+ demonstrationEnabled,+ startDemonstration,]);// Listen for conditional branch changes to trigger re-layoutdiff --git a/skyvern-frontend/src/routes/workflows/editor/edges/EdgeWithAddButton.tsx b/skyvern-frontend/src/routes/workflows/editor/edges/EdgeWithAddButton.tsxindex 6807f3f9f..8934e9a6c 100644--- a/skyvern-frontend/src/routes/workflows/editor/edges/EdgeWithAddButton.tsx+++ b/skyvern-frontend/src/routes/workflows/editor/edges/EdgeWithAddButton.tsx@@ -19,6 +19,9 @@ import { useSopToBlocksMutation } from "@/routes/workflows/hooks/useSopToBlocksMimport { useDebugStore } from "@/store/useDebugStore";import { useRecordedBlocksStore } from "@/store/RecordedBlocksStore";import { useRecordingStore } from "@/store/useRecordingStore";+import { useRecordingLauncherStore } from "@/store/useRecordingLauncherStore";+import { useFeatureFlag } from "@/hooks/useFeatureFlag";+import { WORKFLOW_COPILOT_DEMONSTRATION_FLAG } from "@/util/featureFlags";import {requestRecordingStart,useIsMountedRef,@@ -40,6 +43,7 @@ import { SELECTED_RING_CLASSES } from "../selection/selectedRingClasses";import { WorkflowAddMenu } from "../WorkflowAddMenu";import { WorkflowAdderBusy } from "../WorkflowAdderBusy";import { findBranchContextForInsertion } from "../workflowInsertion";+import { demonstrationInsertionFromNodes } from "../workspaceAuthoringActions";function EdgeWithAddButton({source,@@ -66,6 +70,11 @@ function EdgeWithAddButton({});const debugStore = useDebugStore();const recordingStore = useRecordingStore();+ const startDemonstration = useRecordingLauncherStore(+ (state) => state.startDemonstration,+ );+ const demonstrationEnabled =+ useFeatureFlag(WORKFLOW_COPILOT_DEMONSTRATION_FLAG) === true;const settingsStore = useSettingsStore();const workflowStatePanel = useWorkflowPanelStore();const setRecordedBlocks = useRecordedBlocksStore(@@ -159,6 +168,21 @@ function EdgeWithAddButton({const mountedRef = useIsMountedRef();const onRecord = () => {+ if (demonstrationEnabled) {+ const insertion = demonstrationInsertionFromNodes(nodes, {+ previous: source,+ next: target,+ parent: sourceNode?.parentId,+ });+ if (!insertion) {+ toast({+ title: "Recording inside a loop or branch isn't supported yet",+ });+ return;+ }+ startDemonstration?.({ insertion });+ return;+ }if (recordingStore.isRecording) {recordingStore.setIsRecording(false);} else {@@ -268,6 +292,7 @@ function EdgeWithAddButton({onRecord={onRecord}onUploadSOP={onUploadSOP}isUploadingSOP={isUploadingSOP}+ canRecordTask={!demonstrationEnabled || startDemonstration !== null}>{adder}</WorkflowAddMenu>diff --git a/skyvern-frontend/src/routes/workflows/editor/nodes/NodeAdderNode/NodeAdderNode.tsx b/skyvern-frontend/src/routes/workflows/editor/nodes/NodeAdderNode/NodeAdderNode.tsxindex c245696cd..cc83f0fe7 100644--- a/skyvern-frontend/src/routes/workflows/editor/nodes/NodeAdderNode/NodeAdderNode.tsx+++ b/skyvern-frontend/src/routes/workflows/editor/nodes/NodeAdderNode/NodeAdderNode.tsx@@ -13,6 +13,9 @@ import {import { useWorkflowSettingsStore } from "@/store/WorkflowSettingsStore";import { useRecordedBlocksStore } from "@/store/RecordedBlocksStore";import { useRecordingStore } from "@/store/useRecordingStore";+import { useRecordingLauncherStore } from "@/store/useRecordingLauncherStore";+import { useFeatureFlag } from "@/hooks/useFeatureFlag";+import { WORKFLOW_COPILOT_DEMONSTRATION_FLAG } from "@/util/featureFlags";import {requestRecordingStart,useIsMountedRef,@@ -38,6 +41,7 @@ import { WorkflowAdderBusy } from "../../WorkflowAdderBusy";import { SELECTED_RING_CLASSES } from "../../selection/selectedRingClasses";import { useWorkflowScopeReadOnly } from "../../WorkflowScopeContext";import { findBranchContextForInsertion } from "../../workflowInsertion";+import { demonstrationInsertionFromNodes } from "../../workspaceAuthoringActions";function NodeAdderNode({ id, parentId }: NodeProps<NodeAdderNode>) {const workflowPermanentId = useWorkflowPermanentId();@@ -45,6 +49,11 @@ function NodeAdderNode({ id, parentId }: NodeProps<NodeAdderNode>) {const nodes = useNodes();const debugStore = useDebugStore();const recordingStore = useRecordingStore();+ const startDemonstration = useRecordingLauncherStore(+ (state) => state.startDemonstration,+ );+ const demonstrationEnabled =+ useFeatureFlag(WORKFLOW_COPILOT_DEMONSTRATION_FLAG) === true;const settingsStore = useSettingsStore();const workflowSettingsStore = useWorkflowSettingsStore();const setWorkflowPanelState = useWorkflowPanelStore(@@ -188,6 +197,21 @@ function NodeAdderNode({ id, parentId }: NodeProps<NodeAdderNode>) {const mountedRef = useIsMountedRef();const onRecord = () => {+ if (demonstrationEnabled) {+ const insertion = demonstrationInsertionFromNodes(nodes, {+ previous: previous ?? null,+ next: id,+ parent: parentId,+ });+ if (!insertion) {+ toast({+ title: "Recording inside a loop or branch isn't supported yet",+ });+ return;+ }+ startDemonstration?.({ insertion });+ return;+ }if (recordingStore.isRecording) {recordingStore.setIsRecording(false);} else {@@ -328,6 +352,7 @@ function NodeAdderNode({ id, parentId }: NodeProps<NodeAdderNode>) {onRecord={onRecord}onUploadSOP={onUploadSOP}isUploadingSOP={isUploadingSOP}+ canRecordTask={!demonstrationEnabled || startDemonstration !== null}onPinnedChange={setMenuPinned}>{adder}diff --git a/skyvern-frontend/src/routes/workflows/editor/workspaceAuthoringActions.test.ts b/skyvern-frontend/src/routes/workflows/editor/workspaceAuthoringActions.test.tsindex b4d2437d6..3093d9017 100644--- a/skyvern-frontend/src/routes/workflows/editor/workspaceAuthoringActions.test.ts+++ b/skyvern-frontend/src/routes/workflows/editor/workspaceAuthoringActions.test.ts@@ -2,6 +2,7 @@ import { describe, expect, it, vi } from "vitest";import {applySopResultAtCurrentAppend,+ demonstrationInsertionFromNodes,resolveWorkspaceAuthoringActionAvailability,} from "./workspaceAuthoringActions";@@ -172,3 +173,38 @@ describe("workspace authoring actions", () => {});});});++describe("demonstrationInsertionFromNodes", () => {+ it("uses adjacent workflow labels for the insertion", () => {+ const nodes = [+ { id: "first", type: "task", data: { label: "first_step" } },+ { id: "second", type: "task", data: { label: "second_step" } },+ ];++ expect(+ demonstrationInsertionFromNodes(nodes, {+ previous: "first",+ next: "second",+ parent: undefined,+ }),+ ).toEqual({+ after_label: "first_step",+ before_label: "second_step",+ });+ });++ it("refuses a parent-only insertion inside an empty container", () => {+ const nodes = [+ { id: "start", type: "start", parentId: "loop" },+ { id: "adder", type: "nodeAdder", parentId: "loop" },+ ];++ expect(+ demonstrationInsertionFromNodes(nodes, {+ previous: "start",+ next: "adder",+ parent: "loop",+ }),+ ).toBeNull();+ });+});diff --git a/skyvern-frontend/src/routes/workflows/editor/workspaceAuthoringActions.ts b/skyvern-frontend/src/routes/workflows/editor/workspaceAuthoringActions.tsindex eaf389417..88441c6db 100644--- a/skyvern-frontend/src/routes/workflows/editor/workspaceAuthoringActions.ts+++ b/skyvern-frontend/src/routes/workflows/editor/workspaceAuthoringActions.ts@@ -1,10 +1,12 @@import type { WorkflowBlock, WorkflowParameter } from "../types/workflowTypes";import type { InsertionPoint } from "@/store/RecordedBlocksStore";+import type { WorkflowCopilotDemonstrationInsertion } from "@/routes/workflows/copilot/workflowCopilotTypes";type WorkflowNodeLike = {id: string;type?: string;parentId?: string;+ data?: { label?: unknown };};type WorkflowEdgeLike = {@@ -36,6 +38,29 @@ function resolveAppendInsertionPoint(};}+function demonstrationInsertionFromNodes(+ nodes: Array<WorkflowNodeLike>,+ insertionPoint: Pick<InsertionPoint, "previous" | "next" | "parent">,+): WorkflowCopilotDemonstrationInsertion | null {+ const labelFor = (nodeId: string | null) => {+ const node = nodes.find((candidate) => candidate.id === nodeId);+ if (!node || node.type === "start" || node.type === "nodeAdder")+ return null;+ const label = node.data?.label;+ return typeof label === "string" && label.length > 0 ? label : null;+ };+ const insertion = {+ after_label: labelFor(insertionPoint.previous),+ before_label: labelFor(insertionPoint.next),+ };+ const parent =+ insertionPoint.parent ??+ nodes.find((node) => node.id === insertionPoint.previous)?.parentId;+ return parent && !insertion.after_label && !insertion.before_label+ ? null+ : insertion;+}+function applySopResultAtCurrentAppend({result,getNodes,@@ -105,6 +130,7 @@ function resolveWorkspaceAuthoringActionAvailability({export {applySopResultAtCurrentAppend,+ demonstrationInsertionFromNodes,resolveAppendInsertionPoint,resolveWorkspaceAuthoringActionAvailability,};diff --git a/skyvern-frontend/src/routes/workflows/studio/runProjections.ts b/skyvern-frontend/src/routes/workflows/studio/runProjections.tsindex 03219cc5d..90debef31 100644--- a/skyvern-frontend/src/routes/workflows/studio/runProjections.ts+++ b/skyvern-frontend/src/routes/workflows/studio/runProjections.ts@@ -178,6 +178,24 @@ export function runErrorRows(return rows;}+export type RunErrorCode = { code: string; message: string | null };++// The Overview's failure line names each code once, keeping the first message+// that carried it as the hover. Built on runErrorRows so the line and+// Outputs → Errors can't disagree on which codes a run has.+export function runErrorCodes(+ errors: Array<Record<string, unknown>>,+): RunErrorCode[] {+ const byCode = new Map<string, string | null>();+ for (const row of runErrorRows(errors)) {+ if (!row.code || byCode.has(row.code)) {+ continue;+ }+ byCode.set(row.code, row.message);+ }+ return Array.from(byCode, ([code, message]) => ({ code, message }));+}+function nonEmptyString(value: unknown): boolean {return typeof value === "string" && value.trim() !== "";}diff --git a/skyvern-frontend/src/routes/workflows/studio/runview/RunFailurePresentation.tsx b/skyvern-frontend/src/routes/workflows/studio/runview/RunFailurePresentation.tsxindex 3a798eeda..51f851890 100644--- a/skyvern-frontend/src/routes/workflows/studio/runview/RunFailurePresentation.tsx+++ b/skyvern-frontend/src/routes/workflows/studio/runview/RunFailurePresentation.tsx@@ -8,6 +8,8 @@ import {import { type WorkflowRunStatusApiResponseWithWorkflow } from "@/api/types";import { runIsRetryWaiting } from "@/routes/workflows/workflowRun/runRetryState";import { Button } from "@/components/ui/button";++import { type RunErrorCode } from "../runProjections";// Fix leads when it is offered; Retry is the primary only when it stands alone.export function FailureRecoveryActions({onFix,@@ -56,11 +58,48 @@ export function FailureTips({ tips }: { tips: Array<string> }) {);}+// Past this many the codes would own the line; the rest stay one click away in+// Outputs → Errors, which lists every code with its message.+const MAX_FAILURE_LINE_ERROR_CODES = 3;++// Verbatim, because customers match on the exact code — same text as+// Outputs → Errors, so neither surface has to be translated to the other.+export function RunFailureErrorCodes({+ codes,+}: {+ codes: Array<RunErrorCode>;+}) {+ const shown = codes.slice(0, MAX_FAILURE_LINE_ERROR_CODES);+ const hidden = codes.length - shown.length;+ return (+ <>+ {shown.map((entry) => (+ <code+ key={entry.code}+ title={entry.message ?? undefined}+ className="shrink-0 rounded border border-destructive/40 bg-destructive/10 px-1.5 py-0.5 font-mono text-[11px] leading-4 text-destructive"+ >+ {entry.code}+ </code>+ ))}+ {hidden > 0 ? (+ <span+ className="shrink-0 text-[11px] text-muted-foreground"+ title="Outputs → Errors lists every error code for this run."+ >+ +{hidden} more+ </span>+ ) : null}+ </>+ );+}+// The strip's second line. The block name is the jump; the full// reason rides the headline's title so nothing else is said at run level.export function RunFailureLine({workflowRun,blockLabel,+ errorCodes,headline,detail,onJump,@@ -69,6 +108,7 @@ export function RunFailureLine({}: {workflowRun?: WorkflowRunStatusApiResponseWithWorkflow;blockLabel: string | null;+ errorCodes: Array<RunErrorCode>;headline: string;detail: string | null;onJump?: () => void;@@ -100,6 +140,7 @@ export function RunFailureLine({{blockLabel}</button>) : null}+ <RunFailureErrorCodes codes={errorCodes} /><spanclassName="min-w-0 flex-auto truncate text-foreground"title={detail ?? undefined}diff --git a/skyvern-frontend/src/routes/workflows/studio/runview/RunView.test.tsx b/skyvern-frontend/src/routes/workflows/studio/runview/RunView.test.tsxindex 9feb943e1..c5641f4ff 100644--- a/skyvern-frontend/src/routes/workflows/studio/runview/RunView.test.tsx+++ b/skyvern-frontend/src/routes/workflows/studio/runview/RunView.test.tsx@@ -1546,6 +1546,53 @@ describe("RunView failure presentation", () => {expect(line.queryByText(/Exception: boom/)).toBeNull();});+ test("names the run's error code on the Overview, with its message on hover", () => {+ seedCompletedRun({+ status: Status.Failed,+ failure_reason:+ "login block failed. failure reason: The site is down for maintenance.",+ errors: [+ {+ error_code: "WEBSITE_DOWN",+ reasoning: "The site returned a maintenance page.",+ },+ ],+ });+ const { getByTestId } = renderRunView();+ const line = within(getByTestId("run-failure-line"));++ // No Outputs click: triage reads the code off the Overview itself.+ expect(line.getByText("WEBSITE_DOWN").getAttribute("title")).toBe(+ "The site returned a maintenance page.",+ );+ expect(line.getByText("The site is down for maintenance.")).not.toBeNull();+ });++ test("names each error code once and holds a long list to three", () => {+ seedCompletedRun({+ status: Status.Failed,+ failure_reason: "The run failed",+ errors: [+ { error_code: "WEBSITE_DOWN", reasoning: "First iteration." },+ { error_code: "WEBSITE_DOWN", reasoning: "Second iteration." },+ { error_code: "REACH_MAX_RETRIES" },+ { error_code: "NAVIGATION_FAILED" },+ { error_code: "DATA_UNAVAILABLE" },+ ],+ });+ const { getByTestId } = renderRunView();+ const line = within(getByTestId("run-failure-line"));++ expect(line.getAllByText("WEBSITE_DOWN")).toHaveLength(1);+ expect(line.getByText("WEBSITE_DOWN").getAttribute("title")).toBe(+ "First iteration.",+ );+ expect(line.getByText("REACH_MAX_RETRIES")).not.toBeNull();+ expect(line.getByText("NAVIGATION_FAILED")).not.toBeNull();+ expect(line.queryByText("DATA_UNAVAILABLE")).toBeNull();+ expect(line.getByText("+1 more")).not.toBeNull();+ });+test("shows no failure line for a user-canceled run", () => {seedCompletedRun({status: Status.Canceled,diff --git a/skyvern-frontend/src/routes/workflows/studio/runview/RunView.tsx b/skyvern-frontend/src/routes/workflows/studio/runview/RunView.tsxindex 478320c16..c76a99a34 100644--- a/skyvern-frontend/src/routes/workflows/studio/runview/RunView.tsx+++ b/skyvern-frontend/src/routes/workflows/studio/runview/RunView.tsx@@ -89,6 +89,7 @@ import {ELAPSED_NEVER_STARTED,formatElapsed,resolveLandingSelectionId,+ runErrorCodes,runHasOutputs,runOutcomeFromStatus,} from "../runProjections";@@ -796,6 +797,10 @@ export function RunView({const runErrors = useMemo<RunOutputError[]>(() => {return normalizeRunOutputErrors(workflowRun?.errors);}, [workflowRun]);+ const failureErrorCodes = useMemo(+ () => runErrorCodes(runErrors),+ [runErrors],+ );const runInputs = useMemo(() => {const definitionParameters =@@ -1026,6 +1031,7 @@ export function RunView({<RunFailureLineworkflowRun={workflowRun}blockLabel={failedBlock?.label ?? null}+ errorCodes={failureErrorCodes}headline={failureHeadline}detail={failureDetail}onJump={jumpToFailedBlock}diff --git a/skyvern-frontend/src/routes/workflows/studio/useBrowserPaneView.ts b/skyvern-frontend/src/routes/workflows/studio/useBrowserPaneView.tsindex 226587ae2..b8441cd8d 100644--- a/skyvern-frontend/src/routes/workflows/studio/useBrowserPaneView.ts+++ b/skyvern-frontend/src/routes/workflows/studio/useBrowserPaneView.ts@@ -3,6 +3,7 @@ import { useNavigate, useSearchParams } from "react-router-dom";import { useWorkflowPermanentId } from "@/routes/workflows/WorkflowPermanentIdContext";import { useRecordingStore } from "@/store/useRecordingStore";+import { useRecordingLauncherStore } from "@/store/useRecordingLauncherStore";import { useRunViewStore } from "@/store/RunViewStore";import {useStudioBrowserStore,@@ -62,34 +63,45 @@ export function useBrowserPaneView(): BrowserPaneViewState {const setViewIntent = useStudioBrowserStore((s) => s.setView);const pinNonce = useRunViewStore((s) => s.pinNonce);const isRecording = useRecordingStore((s) => s.isRecording);+ const demonstrationBrowserSessionId = useRecordingLauncherStore(+ (s) => s.demonstrationBrowserSessionId,+ );+ const userDrivingDebugBrowser =+ isRecording ||+ (demonstrationBrowserSessionId !== null &&+ demonstrationBrowserSessionId === debugBrowserSessionId);const recordingDeepLink = searchParams.get("view") === "recording";// Timeline selection outranks a pinned pill: a step click (?active= change or// a re-pin of the same step) or a run swap hands the pane back to the machine,// which lands on Screenshots for a pinned step and Live for a fresh run.const activeParam = searchParams.get("active");- const syncRef = useRef({ activeParam, pinNonce, runId, isRecording });+ const syncRef = useRef({+ activeParam,+ pinNonce,+ runId,+ userDrivingDebugBrowser,+ });useEffect(() => {const prev = syncRef.current;- const recordingStarted = isRecording && !prev.isRecording;+ const manualControlStarted =+ userDrivingDebugBrowser && !prev.userDrivingDebugBrowser;if (prev.activeParam !== activeParam ||prev.pinNonce !== pinNonce ||prev.runId !== runId ||- recordingStarted+ manualControlStarted) {- syncRef.current = { activeParam, pinNonce, runId, isRecording };if (!recordingDeepLink) {setViewIntent("auto");}- } else if (prev.isRecording !== isRecording) {- syncRef.current = { activeParam, pinNonce, runId, isRecording };}+ syncRef.current = { activeParam, pinNonce, runId, userDrivingDebugBrowser };}, [activeParam,pinNonce,runId,- isRecording,+ userDrivingDebugBrowser,recordingDeepLink,setViewIntent,]);@@ -132,7 +144,7 @@ export function useBrowserPaneView(): BrowserPaneViewState {const view = resolveBrowserPaneView({intent,- recording: isRecording,+ recording: userDrivingDebugBrowser,scrubbing: visuals.scrubbing,inspectingRun: explicit,blockRunInDebugSession,@@ -149,7 +161,7 @@ export function useBrowserPaneView(): BrowserPaneViewState {});const liveSurface = resolveLiveSurface({- recording: isRecording,+ recording: userDrivingDebugBrowser,running: visuals.running,runInDebugSession,hasRunId: runId != null,diff --git a/skyvern-frontend/src/store/useCopilotHeaderStore.ts b/skyvern-frontend/src/store/useCopilotHeaderStore.tsindex 6cef5850f..06f407500 100644--- a/skyvern-frontend/src/store/useCopilotHeaderStore.ts+++ b/skyvern-frontend/src/store/useCopilotHeaderStore.ts@@ -14,18 +14,24 @@ export type CopilotHeaderControls = {};// What the docked Copilot chat is waiting on the user for.-export type CopilotAttention = "question" | "credential" | "account";+export type CopilotAttention =+ | "question"+ | "credential"+ | "account"+ | "demonstration";export const COPILOT_ATTENTION_LABEL: Record<CopilotAttention, string> = {question: "waiting for your answer",credential: "needs to sign in",account: "needs a Google account",+ demonstration: "waiting for your demonstration",};export const COPILOT_ATTENTION_DOT: Record<CopilotAttention, string> = {question: "Copilot needs your answer",credential: "Copilot needs to sign in",account: "Copilot needs a Google account",+ demonstration: "Copilot is watching your demonstration",};/**diff --git a/skyvern-frontend/src/store/useManualSignInStore.ts b/skyvern-frontend/src/store/useManualSignInStore.tsindex 02d7b2202..e71355151 100644--- a/skyvern-frontend/src/store/useManualSignInStore.ts+++ b/skyvern-frontend/src/store/useManualSignInStore.ts@@ -30,3 +30,21 @@ export function useManualSignInControl(wasSigningIn.current = signingIn;}, [signingIn, setUserIsControlling]);}++export function useDemonstrationControl(+ recording: boolean,+ controllable: boolean,+ setUserIsControlling: (controlling: boolean) => void,+ connectionReady = true,+) {+ const holdingControl = recording && controllable;+ const wasHoldingControl = useRef(false);+ useEffect(() => {+ if (holdingControl) {+ setUserIsControlling(true);+ } else if (wasHoldingControl.current) {+ setUserIsControlling(false);+ }+ wasHoldingControl.current = holdingControl;+ }, [connectionReady, holdingControl, setUserIsControlling]);+}diff --git a/skyvern-frontend/src/store/useRecordingLauncherStore.ts b/skyvern-frontend/src/store/useRecordingLauncherStore.tsindex ace77fceb..99c4e8b08 100644--- a/skyvern-frontend/src/store/useRecordingLauncherStore.ts+++ b/skyvern-frontend/src/store/useRecordingLauncherStore.ts@@ -1,5 +1,11 @@import { create } from "zustand";+import { type WorkflowCopilotDemonstrationInsertion } from "@/routes/workflows/copilot/workflowCopilotTypes";++export type StartDemonstrationOptions = {+ insertion?: WorkflowCopilotDemonstrationInsertion;+};+/*** Bridges the studio's Browser-tab Record button to the canvas-aware Workspace.* The Workspace (which has ReactFlow nodes/edges) registers a launcher that@@ -9,6 +15,13 @@ import { create } from "zustand";type RecordingLauncherState = {startRecordingAtEnd: (() => void) | null;setStartRecordingAtEnd: (launcher: (() => void) | null) => void;+ startDemonstration: ((options?: StartDemonstrationOptions) => void) | null;+ setStartDemonstration: (+ launcher: ((options?: StartDemonstrationOptions) => void) | null,+ ) => void;+ demonstrationBrowserSessionId: string | null;+ demonstrationRecordingBrowserSessionId: string | null;+ navigateDemonstration: ((url: string) => void) | null;};export const useRecordingLauncherStore = create<RecordingLauncherState>(@@ -16,5 +29,10 @@ export const useRecordingLauncherStore = create<RecordingLauncherState>(startRecordingAtEnd: null,setStartRecordingAtEnd: (startRecordingAtEnd) =>set({ startRecordingAtEnd }),+ startDemonstration: null,+ setStartDemonstration: (startDemonstration) => set({ startDemonstration }),+ demonstrationBrowserSessionId: null,+ demonstrationRecordingBrowserSessionId: null,+ navigateDemonstration: null,}),);diff --git a/skyvern-frontend/src/util/featureFlags.ts b/skyvern-frontend/src/util/featureFlags.tsindex afc735275..ea98a8d14 100644--- a/skyvern-frontend/src/util/featureFlags.ts+++ b/skyvern-frontend/src/util/featureFlags.ts@@ -43,3 +43,6 @@ export const ONBOARDING_TRACK_SECOND_AGENT_FLAG =export const NEW_ONBOARDING_EXPERIENCE_FLAG = "new_onboarding_experience";export const BROWSER_SESSION_PREWARM_FLAG = "BROWSER_SESSION_PREWARM";++export const WORKFLOW_COPILOT_DEMONSTRATION_FLAG =+ "WORKFLOW_COPILOT_DEMONSTRATION";diff --git a/skyvern/config.py b/skyvern/config.pyindex 599309596..e3d9d622a 100644--- a/skyvern/config.py+++ b/skyvern/config.py@@ -155,6 +155,9 @@ class Settings(BaseSettings):PAGE_READY_DOM_STABLE_MS: float = 300 # Time with no DOM mutations to consider stablePAGE_READY_DOM_STABILITY_TIMEOUT_MS: float = 3000 # Max time to wait for DOM stabilityBROWSER_SCREENSHOT_TIMEOUT_MS: int = 20000+ # Minimum seconds between the full gc.collect() runs after multi-viewport screenshot stitches, per+ # process. Each run pauses every browser session on the event loop; 0 collects after every stitch.+ SCREENSHOT_STITCH_GC_MIN_INTERVAL_SECONDS: float = 60.0# Consecutive unanswered browser-protocol operations (spanning more than one kind) after which# a run stops starting new steps against that browser. Sized so a healthy run — which lands# hundreds of successful operations per step, any one of which resets the count — cannot reach@@ -285,6 +288,7 @@ class Settings(BaseSettings):# so this is a guaranteed no-op behind app.CACHE.is_shared regardless of this flag.WORKFLOW_COPILOT_CREDENTIAL_PAUSE_ENABLED: bool = TrueWORKFLOW_COPILOT_CREDENTIAL_PAUSE_TIMEOUT_SECONDS: int = 300+ WORKFLOW_COPILOT_DEMONSTRATION_ENABLED: bool = False# Replaces the pause countdown once when the user chooses to sign in themselves in the live browser.WORKFLOW_COPILOT_MANUAL_SIGN_IN_TIMEOUT_SECONDS: int = 900# Kill switch for the live codegen-progress SSE frame (drafted block labels while an authoringdiff --git a/skyvern/forge/agent.py b/skyvern/forge/agent.pyindex b3d69b49c..f2ddc2d06 100644--- a/skyvern/forge/agent.py+++ b/skyvern/forge/agent.py@@ -1407,7 +1407,13 @@ async def _read_task_v3_llm_name_override(return None# get_config synthesizes a config for unknown keys, so require an explicitly registered,# non-custom (not org-owned BYO) llm_key.- if is_custom_llm_key(variant) or not LLMConfigRegistry.is_registered(variant):+ # A non-router ANTHROPIC key takes LLMCaller's native Anthropic message path, which cannot build a Task V3+ # transcript.+ if (+ is_custom_llm_key(variant)+ or not LLMConfigRegistry.is_registered(variant)+ or (not LLMConfigRegistry.is_router_config(variant) and "ANTHROPIC" in variant)+ ):LOG.warning("TASK_V3_LLM_NAME is not a usable registered llm_key; ignoring", variant=variant)return Nonereturn variant@@ -3327,6 +3333,8 @@ class ForgeAgent:target_action_reserve=target_action_reserve,block_completion_judge=block_completion_judge,code_typed_values=recovery_code_progress.typed_values if recovery_code_progress else (),+ anthropic_cache_markers=uses_only_anthropic_models(llm_caller.llm_config)+ and not (context is not None and await self._is_anthropic_prompt_caching_disabled(context)),)finally:input_dispatch.end_login_pace()diff --git a/skyvern/forge/api_app.py b/skyvern/forge/api_app.pyindex 3a203b079..54f8b5014 100644--- a/skyvern/forge/api_app.py+++ b/skyvern/forge/api_app.py@@ -4,7 +4,6 @@# ruff: noqa: E402import asyncio-import osimport uuidfrom contextlib import asynccontextmanagerfrom datetime import datetime@@ -493,25 +492,6 @@ def create_api_app() -> FastAPI:forge_app_instance = start_forge_app()- # Initialize Laminar tracing after ForgeApp so auto-instrumentation works.- lmnr_api_key = os.environ.get("LMNR_PROJECT_API_KEY")- if lmnr_api_key:- try:- from lmnr import Laminar # noqa: PLC0415-- lmnr_base_url = os.environ.get("LMNR_BASE_URL", "http://localhost")- lmnr_grpc_port = int(os.environ.get("LMNR_GRPC_PORT", "8001"))- lmnr_http_port = int(os.environ.get("LMNR_HTTP_PORT", "8000"))- Laminar.initialize(- project_api_key=lmnr_api_key,- base_url=lmnr_base_url,- grpc_port=lmnr_grpc_port,- http_port=lmnr_http_port,- )- LOG.info("Laminar tracing initialized", base_url=lmnr_base_url, grpc_port=lmnr_grpc_port)- except Exception as e:- LOG.warning("Failed to initialize Laminar tracing", error=str(e))-assert_signing_key_usable()fastapi_app = FastAPI(lifespan=lifespan)diff --git a/skyvern/forge/request_logging.py b/skyvern/forge/request_logging.pyindex 6090aedb1..58af11ca6 100644--- a/skyvern/forge/request_logging.py+++ b/skyvern/forge/request_logging.py@@ -58,6 +58,7 @@ _SENSITIVE_ENDPOINTS = {"POST /v1/workflow/copilot/question-response","POST /v1/workflow/copilot/steer","POST /v1/workflow/copilot/credential-response",+ "POST /v1/workflow/copilot/demonstration/control","POST /v1/workflow/copilot/convert-yaml-to-blocks",# Payment-provider events carry customer billing details (name, email, address, card metadata)."POST /api/v1/stripe_webhook",diff --git a/skyvern/forge/sdk/api/llm/api_handler_factory.py b/skyvern/forge/sdk/api/llm/api_handler_factory.pyindex 5ef52754f..a863a314f 100644--- a/skyvern/forge/sdk/api/llm/api_handler_factory.py+++ b/skyvern/forge/sdk/api/llm/api_handler_factory.py@@ -19,6 +19,7 @@ from anthropic import NOT_GIVENfrom anthropic import APIConnectionError as AnthropicAPIConnectionErrorfrom anthropic import APIStatusError as AnthropicAPIStatusErrorfrom anthropic.types.beta.beta_message import BetaMessage as AnthropicMessage+from litellm.litellm_core_utils.core_helpers import map_finish_reasonfrom litellm.llms.custom_httpx.http_handler import AsyncHTTPHandlerfrom litellm.types.router import AllowedFailsPolicy@@ -56,6 +57,7 @@ from skyvern.forge.sdk.api.llm.exceptions import (LLMProviderErrorRetryableTask,)from skyvern.forge.sdk.api.llm.litellm_transport import configure_litellm_transport+from skyvern.forge.sdk.api.llm.litellm_usage import use_anthropic_reported_thinking_tokensfrom skyvern.forge.sdk.api.llm.ui_tars_response import UITarsResponsefrom skyvern.forge.sdk.api.llm.utils import (is_content_filtered_response,@@ -93,6 +95,7 @@ from skyvern.utils.url_validators import validate_fetch_url# Keep this server-only side effect out of the package __init__ so the legacy# models shim can import without litellm. Legacy LLM calls enter this module.configure_litellm_transport()+use_anthropic_reported_thinking_tokens()LOG = structlog.get_logger()_HASHED_HREF_PLACEHOLDER = re.compile(r"\{\{\s*(_[0-9a-f]{64})\s*\}\}")@@ -820,6 +823,15 @@ def _response_provider(response: object) -> str | None:return provider if isinstance(provider, str) else None+def _response_finish_reason(response: object) -> str | None:+ # The raw Anthropic SDK reports a stop_reason; map it to litellm's vocabulary so every path logs the same values.+ if isinstance(response, AnthropicMessage):+ return map_finish_reason(response.stop_reason) if response.stop_reason else None+ if isinstance(response, ModelResponse) and response.choices:+ return response.choices[0].finish_reason+ return None++def _usage_int_field(value: Any, *field_names: str) -> int | None:for field_name in field_names:field = value.get(field_name) if isinstance(value, dict) else getattr(value, field_name, None)@@ -828,6 +840,22 @@ def _usage_int_field(value: Any, *field_names: str) -> int | None:return None+def _cache_creation_tokens(usage: Any) -> int | None:+ if usage is None:+ return None+ details = getattr(usage, "prompt_tokens_details", None) or getattr(usage, "input_tokens_details", None)+ tokens = (+ _usage_int_field(details, "cache_write_tokens", "cache_creation_input_tokens") if details is not None else None+ )+ if tokens is None and details is not None:+ model_extra = getattr(details, "model_extra", None)+ if isinstance(model_extra, dict):+ tokens = _usage_int_field(model_extra, "cache_write_tokens")+ if tokens is None:+ tokens = _usage_int_field(usage, "cache_creation_input_tokens", "cache_write_input_tokens")+ return tokens++def _copilot_model_usage_event(response: object,*,@@ -850,17 +878,7 @@ def _copilot_model_usage_event(cache_read_tokens = _usage_int_field(details, "cached_tokens") if details is not None else Noneif cache_read_tokens is None and usage is not None:cache_read_tokens = _usage_int_field(usage, "cache_read_input_tokens")- cache_creation_tokens = (- _usage_int_field(details, "cache_write_tokens", "cache_creation_input_tokens")- if details is not None- else None- )- if cache_creation_tokens is None and details is not None:- model_extra = getattr(details, "model_extra", None)- if isinstance(model_extra, dict):- cache_creation_tokens = _usage_int_field(model_extra, "cache_write_tokens")- if cache_creation_tokens is None and usage is not None:- cache_creation_tokens = _usage_int_field(usage, "cache_creation_input_tokens", "cache_write_input_tokens")+ cache_creation_tokens = _cache_creation_tokens(usage)except Exception:input_tokens = Noneoutput_tokens = None@@ -4176,6 +4194,9 @@ class LLMCaller:"reasoning_tokens", call_stats.reasoning_tokens if call_stats else None),cached_tokens=_generated_log_value("cached_tokens", call_stats.cached_tokens if call_stats else None),+ cache_creation_tokens=_generated_log_value(+ "cache_creation_tokens", _cache_creation_tokens(getattr(response, "usage", None))+ ),llm_cost=_generated_log_value("llm_cost", call_stats.llm_cost if call_stats else None),image_count=_generated_log_value("image_count", image_count if image_count > 0 else None),image_tokens=_generated_log_value("image_tokens", image_tokens if image_tokens > 0 else None),@@ -4183,6 +4204,7 @@ class LLMCaller:image_tokens_source=image_source,resolved_provider=resolved_provider,service_tier=service_tier,+ finish_reason=_response_finish_reason(response),# `model` above is the router group, identical for the flex and standard legs;# without the served deployment the split between them is invisible.served_model_group=_model_log_value("served_model_group", served_model_group),@@ -4675,7 +4697,7 @@ class LLMCaller:input_tokens=usage.input_tokens,output_tokens=usage.output_tokens,cached_tokens=cached_tokens,- reasoning_tokens=0,+ reasoning_tokens=usage.output_tokens_details.thinking_tokens if usage.output_tokens_details else 0,)elif isinstance(response, (ModelResponse, CustomStreamWrapper)):input_tokens, output_tokens, reasoning_tokens, cached_tokens = LLMAPIHandlerFactory._extract_token_counts(diff --git a/skyvern/forge/sdk/api/llm/litellm_usage.py b/skyvern/forge/sdk/api/llm/litellm_usage.pynew file mode 100644index 000000000..aa927fa59--- /dev/null+++ b/skyvern/forge/sdk/api/llm/litellm_usage.py@@ -0,0 +1,25 @@+import functools+from typing import Any++from litellm.llms.anthropic.chat.transformation import AnthropicConfig+from litellm.types.utils import Usage+++def use_anthropic_reported_thinking_tokens() -> None:+ # litellm < 1.99: AnthropicConfig.calculate_usage counts the thinking text, which Claude 5.x returns empty, and+ # ignores the API's thinking_tokens. Delete once litellm >= 1.99 is pinned; a unit test fails when it is.+ original = AnthropicConfig.calculate_usage++ @functools.wraps(original)+ def calculate_usage(self: AnthropicConfig, usage_object: dict, *args: Any, **kwargs: Any) -> Usage:+ usage = original(self, usage_object, *args, **kwargs)+ details = usage_object.get("output_tokens_details")+ thinking_tokens = details.get("thinking_tokens") if isinstance(details, dict) else None+ if isinstance(thinking_tokens, int) and usage.completion_tokens_details is not None:+ reasoning_tokens = min(max(thinking_tokens, 0), usage.completion_tokens)+ usage.completion_tokens_details.reasoning_tokens = reasoning_tokens+ # litellm prices text and reasoning tokens separately; they must still sum to completion_tokens.+ usage.completion_tokens_details.text_tokens = usage.completion_tokens - reasoning_tokens+ return usage++ AnthropicConfig.calculate_usage = calculate_usage # type: ignore[method-assign]diff --git a/skyvern/forge/sdk/copilot/agent.py b/skyvern/forge/sdk/copilot/agent.pyindex 2f339dda5..46a40149d 100644--- a/skyvern/forge/sdk/copilot/agent.py+++ b/skyvern/forge/sdk/copilot/agent.py@@ -32,6 +32,7 @@ if TYPE_CHECKING:import structlogimport yamlfrom agents.items import ToolCallItem+from agents.run_config import RunConfigfrom litellm.exceptions import NotFoundError as LiteLLMNotFoundErrorfrom openai.types.responses import ResponseFunctionToolCallfrom pydantic import JsonValue, TypeAdapter, ValidationError@@ -95,8 +96,10 @@ from skyvern.forge.sdk.copilot.config import ()from skyvern.forge.sdk.copilot.context import (COPILOT_RESPONSE_TYPES,+ EDIT_BLOCK_TOOL_NAME,REPLY_TOOL_NAME,SIGNED_OUT_PAGE_SUMMARY_CHAR_CAP,+ UPDATE_AND_RUN_BLOCKS_TOOL_NAME,AgentResult,CodeAuthoringRepairContext,CopilotContext,@@ -125,6 +128,8 @@ from skyvern.forge.sdk.copilot.context import ()from skyvern.forge.sdk.copilot.credential_pause import credential_pause_reason, credential_recovery_token_digestfrom skyvern.forge.sdk.copilot.data_write_defaults import default_data_write_continue_on_failure+from skyvern.forge.sdk.copilot.demonstration_recorder import DemonstrationRejection+from skyvern.forge.sdk.copilot.demonstration_replay import replay_demonstrationfrom skyvern.forge.sdk.copilot.enforcement import (FINAL_REPLY_OBSERVATION,CopilotNonRetriableNavError,@@ -234,10 +239,6 @@ from skyvern.forge.sdk.copilot.streaming_adapter import (maybe_emit_design_end,)from skyvern.forge.sdk.copilot.tools import reply_ends_turn-from skyvern.forge.sdk.copilot.tools._shared import (- EDIT_BLOCK_TOOL_NAME,- UPDATE_AND_RUN_BLOCKS_TOOL_NAME,-)from skyvern.forge.sdk.copilot.tools.blockers import _goal_value_paths_for_code_blockfrom skyvern.forge.sdk.copilot.tools.browser_code import close_browser_code_sessionfrom skyvern.forge.sdk.copilot.tools.credential_fill import raw_secret_connected_credential@@ -2271,6 +2272,8 @@ def _build_narrative_payload(}if ctx.google_connection_notices:payload["googleConnectionNotices"] = [notice.to_payload() for notice in ctx.google_connection_notices]+ if ctx.demonstration_items:+ payload["demonstrationItems"] = list(ctx.demonstration_items)if ctx.staged_workflow_yaml is not None:review = build_review_projection(ctx.persisted_workflow_yaml or _EMPTY_REVIEW_BASELINE_YAML,@@ -3622,7 +3625,7 @@ async def _translate_to_agent_result(resp_type = action_data.get("type", "REPLY")if resp_type not in COPILOT_RESPONSE_TYPES:resp_type = "REPLY"- if resp_type == "REPLACE_WORKFLOW" and final_reply:+ if resp_type == "REPLACE_WORKFLOW" and (final_reply or budget_drain):LOG.warning("copilot final reply suppressed inline workflow replacement")user_response = _with_inline_reject_note(user_response, FINAL_REPLY_UNAPPLIED_CHANGE_NOTE)resp_type = "REPLY"@@ -4261,12 +4264,20 @@ async def _run_agent_loop_with_surface(if session is None:session = create_copilot_session(chat_id)model_token = _copilot_model_name.set(model_name)- tool_call_count_start = ctx.tool_calls_this_turntry:with model_attempt_telemetry_scope() as attempt_telemetry:try:async with MCPServerManager([mcp_server]) as manager:agent.mcp_servers = list(manager.active_servers)+ if ctx.demonstration_requested and not ctx.demonstration_ran and not final_reply:+ ctx.demonstration_ran = True+ demonstration_items = await replay_demonstration(ctx, mcp_server)+ initial_input = (+ [{"role": "user", "content": initial_input}]+ if isinstance(initial_input, str)+ else list(initial_input)+ ) + demonstration_items+ tool_call_count_start = ctx.tool_calls_this_turnif ctx.eval_mode is not None:advertised_mcp_tools = await mcp_server.list_tools()ctx.eval_tool_surface_sha256 = CopilotToolSurface(@@ -4342,6 +4353,8 @@ async def _run_agent_loop_with_surface()if empty_error is not None:raise empty_error+ except DemonstrationRejection:+ raiseexcept Exception as exc:stop_metadata = (exc.stop_metadata@@ -4970,6 +4983,8 @@ async def run_copilot_agent(private_workflow_settings=private_workflow_settings,)return result+ except DemonstrationRejection:+ raiseexcept Exception as exc:LOG.error("Copilot turn unhandled error",@@ -5030,6 +5045,8 @@ async def run_copilot_agent(if browser_session_id:await asyncio.shield(close_browser_session_quietly(organization_id, browser_session_id))raise+ except DemonstrationRejection:+ raiseexcept Exception as exc:LOG.error("Copilot turn unhandled error",@@ -5187,6 +5204,13 @@ async def _run_copilot_turn_impl(if chat_request.credential_recovery_tokenelse None),+ demonstration_requested=chat_request.product_action == "demonstrate",+ demonstration_id=chat_request.demonstration_id,+ demonstration_insertion=(+ chat_request.demonstration_insertion.model_dump()+ if chat_request.product_action != "demonstrate" and chat_request.demonstration_insertion is not None+ else None+ ),executed_block_fingerprints={label: set(fingerprints) for label, fingerprints in (prior_executed_block_fingerprints or {}).items()},@@ -5753,11 +5777,12 @@ async def _run_copilot_turn_impl(try:state = ctx.budget_expiry_stateruns_not_in_history = await recorded_open_step_runs(ctx) if state.hard_backstop_reached else []+ turn_model_name, turn_run_config, turn_llm_key, turn_is_fallback = turn_modelreply = await _run_attempt(- model_name,- run_config,- llm_key,- is_fallback=False,+ turn_model_name,+ turn_run_config,+ turn_llm_key,+ is_fallback=turn_is_fallback,final_reply=True,final_reply_observation=budget_expired_reply_observation(_budget_expiry_state_with_staged_draft(ctx), runs_not_in_history@@ -5780,6 +5805,10 @@ async def _run_copilot_turn_impl(LOG.warning("copilot_budget_expiry_reply_failed", error_type=type(reply_error).__name__)return _build_budget_expiry_exit_result(ctx, global_llm_context)+ # The model the turn is running on; a budget-expiry reply after a fallback must not go back to the+ # primary model that just failed.+ turn_model: tuple[str, RunConfig, str, bool] = (model_name, run_config, llm_key, False)+try:from skyvern.forge.sdk.copilot.session_factory import create_copilot_session@@ -5807,6 +5836,7 @@ async def _run_copilot_turn_impl())ctx.supports_vision = fallback_supports_vision+ turn_model = (fallback_model_name, fallback_run_config, fallback_resolved_key, True)result = await _run_attempt(fallback_model_name,fallback_run_config,@@ -5932,6 +5962,8 @@ async def _run_copilot_turn_impl(organization_id=organization_id,)return _build_non_retriable_nav_exit_result(ctx, global_llm_context, exc)+ except DemonstrationRejection:+ raiseexcept Exception as e:try:# Terminal-path gate-decision record; the per-tool hook records thediff --git a/skyvern/forge/sdk/copilot/browser_code_contract.py b/skyvern/forge/sdk/copilot/browser_code_contract.pyindex 584681ac1..ad9d71f58 100644--- a/skyvern/forge/sdk/copilot/browser_code_contract.py+++ b/skyvern/forge/sdk/copilot/browser_code_contract.py@@ -177,6 +177,11 @@ class BrowserCodeSession(Protocol):@propertydef page(self) -> Page: ...+ @property+ def confirmed_page(self) -> Page | None:+ """The tab the last cell is known to have finished on, or None when that is unconfirmed."""+ ...+async def rebind(self, page: Page) -> None: ...async def reset_namespace(self) -> None: ...@@ -203,6 +208,8 @@ class BrowserCodeHost:sessions_opened: int = 0interrupted: bool = Falseinterrupted_operation: BrowserCodeOperation | None = None+ # A cell that finished but whose call was cancelled before its result was returned.+ unreturned_cell: BrowserCodeCellResult | None = Nonesource_turn_token: str = field(default_factory=lambda: uuid.uuid4().hex)executed_sources: dict[str, ExecutedBrowserCodeSource] = field(default_factory=dict)expired_source_references: set[str] = field(default_factory=set)diff --git a/skyvern/forge/sdk/copilot/budget_expiry.py b/skyvern/forge/sdk/copilot/budget_expiry.pyindex 25b40ef84..fdc21f6db 100644--- a/skyvern/forge/sdk/copilot/budget_expiry.py+++ b/skyvern/forge/sdk/copilot/budget_expiry.py@@ -4,7 +4,7 @@ from __future__ import annotationsimport jsonfrom dataclasses import dataclass, field-from typing import Literal, TypedDict+from typing import Literalfrom uuid import uuid4from pydantic import BaseModel, ConfigDict@@ -42,7 +42,8 @@ class BudgetExpiryObservation(BaseModel):message: str = ("The normal turn budget has expired. headroom is the maximum number of model calls available to finish ""this turn, including this call; the hard time limit may stop it sooner. Your session history and any "- "staged draft remain available. Choose how to use these calls. New build-test runs cannot be dispatched."+ "staged draft remain available. Choose how to use these calls. This step offers no tools that change or "+ "run the workflow.")@@ -86,17 +87,6 @@ class BudgetExpiredReplyObservation(BaseModel):)-class BudgetRunDenialData(TypedDict):- budget_expired: Literal[True]- run_dispatched: Literal[False]- source: BudgetExpirySource | None---class BudgetRunDenial(TypedDict):- ok: Literal[False]- data: BudgetRunDenialData--def serialize_budget_expiry_observation(*, source: BudgetExpirySource, headroom: int, staged_draft: str | None) -> str:return BudgetExpiryObservation(source=source,@@ -118,14 +108,3 @@ def serialize_prior_budget_expiry(outcome: TurnOutcome | None) -> str | None:},separators=(",", ":"),)---def budget_run_denial(state: BudgetExpiryState) -> BudgetRunDenial:- return {- "ok": False,- "data": {- "budget_expired": True,- "run_dispatched": False,- "source": state.source,- },- }diff --git a/skyvern/forge/sdk/copilot/composition_browser_expressions.py b/skyvern/forge/sdk/copilot/composition_browser_expressions.pyindex d8d8c0e45..c79d96a26 100644--- a/skyvern/forge/sdk/copilot/composition_browser_expressions.py+++ b/skyvern/forge/sdk/copilot/composition_browser_expressions.py@@ -276,6 +276,53 @@ _JS_SELECTOR_CANDIDATES_HELPER = ("};")+_JS_SCOUT_PRE_ACTION_PACKET = (+ "function scoutPreActionPacket(el, requested, parsedRole, parsedName, maxScanElements) {"+ " let roleName = parsedRole || parsedName ? { role: parsedRole, accessible_name: parsedName } : null;"+ " if (!roleName && el) {"+ " try {"+ " const r = text(el.getAttribute('role')) || implicitRole(el);"+ " const n = accessibleName(el, r);"+ " if (r || n) roleName = { role: r, accessible_name: n };"+ " } catch (e) { roleName = null; }"+ " }"+ " const targetRole = roleName ? roleName.role : '';"+ " const targetName = roleName ? roleName.accessible_name : '';"+ " let roleNameMatchCount = -1;"+ " if (targetRole && targetName &&"+ " (maxScanElements == null || document.getElementsByTagName('*').length <= maxScanElements)) {"+ " try {"+ " let count = 0;"+ " for (const node of document.querySelectorAll('*')) {"+ " const r = text(node.getAttribute('role')) || implicitRole(node);"+ " if (r !== targetRole) continue;"+ " if (accessibleName(node, r) === targetName) count++;"+ " }"+ " roleNameMatchCount = count;"+ " } catch (e) { roleNameMatchCount = -1; }"+ " }"+ " let selectorMatchCount = -1;"+ " try { selectorMatchCount = document.querySelectorAll(requested).length; }"+ " catch (e) { selectorMatchCount = -1; }"+ " let candidates = [];"+ " try { candidates = el ? collectCandidates(el, requested) : []; } catch (e) { candidates = []; }"+ " return { role_name: roleName, role_name_match_count: roleNameMatchCount,"+ " selector_match_count: selectorMatchCount, selector_candidates: candidates };"+ "}"+)+++def scout_pre_action_packet_js() -> str:+ return (+ _JS_TEXT_HELPER+ + _JS_IS_EDITABLE_HELPER+ + _JS_IMPLICIT_ROLE_HELPER+ + _JS_NAME_FROM_CONTENT_ROLES+ + _JS_ACCESSIBLE_NAME_HELPER+ + _JS_SELECTOR_CANDIDATES_HELPER+ + _JS_SCOUT_PRE_ACTION_PACKET+ )+def selector_candidates_expression(css_selector: str) -> str:"""Return every bounded CSS identity observed for the selected source-page element.@@ -315,50 +362,13 @@ def scout_pre_action_expression(css_selector: str, role: str | None = None, nameparsed_name = json.dumps(name or "")return ("(() => {"- f" {_JS_TEXT_HELPER}"- f" {_JS_IS_EDITABLE_HELPER}"- f" {_JS_IMPLICIT_ROLE_HELPER}"- f" {_JS_NAME_FROM_CONTENT_ROLES}"- f" {_JS_ACCESSIBLE_NAME_HELPER}"- f" {_JS_SELECTOR_CANDIDATES_HELPER}"+ f" {scout_pre_action_packet_js()}"f" const requested = {sel};"f" const parsedRole = {parsed_role};"f" const parsedName = {parsed_name};"" let el = null;"" try { el = document.querySelector(requested); } catch (e) { el = null; }"- " let roleName = null;"- " if (parsedRole || parsedName) roleName = { role: parsedRole, accessible_name: parsedName };"- " else if (el) {"- " try {"- " const r = text(el.getAttribute('role')) || implicitRole(el);"- " const n = accessibleName(el, r);"- " if (r || n) roleName = { role: r, accessible_name: n };"- " } catch (e) { roleName = null; }"- " }"- " const targetRole = roleName ? roleName.role : '';"- " const targetName = roleName ? roleName.accessible_name : '';"- " let roleNameMatchCount = -1;"- " if (targetRole && targetName) {"- " try {"- " let count = 0;"- " for (const node of document.querySelectorAll('*')) {"- " const r = text(node.getAttribute('role')) || implicitRole(node);"- " if (r !== targetRole) continue;"- " if (accessibleName(node, r) === targetName) count++;"- " }"- " roleNameMatchCount = count;"- " } catch (e) { roleNameMatchCount = -1; }"- " }"- " let selectorMatchCount = -1;"- " try { selectorMatchCount = document.querySelectorAll(requested).length; } catch (e) { selectorMatchCount = -1; }"- " let candidates = [];"- " try { candidates = el ? collectCandidates(el, requested) : []; } catch (e) { candidates = []; }"- " return {"- " role_name: roleName,"- " role_name_match_count: roleNameMatchCount,"- " selector_match_count: selectorMatchCount,"- " selector_candidates: candidates,"- " };"+ " return scoutPreActionPacket(el, requested, parsedRole, parsedName);""})()")@@ -540,13 +550,12 @@ const cssMatchesFor = (selector) => {// controls would otherwise rescan its N matches for every one of them.const elementVisibilityCache = new WeakMap();const visibleMatchCountCache = new WeakMap();+const matchVisible = (el) => {+ if (!elementVisibilityCache.has(el)) elementVisibilityCache.set(el, controlVisible(el));+ return elementVisibilityCache.get(el);+};const visibleMatchCount = (matches) => {- if (!visibleMatchCountCache.has(matches)) {- visibleMatchCountCache.set(matches, matches.filter((el) => {- if (!elementVisibilityCache.has(el)) elementVisibilityCache.set(el, controlVisible(el));- return elementVisibilityCache.get(el);- }).length);- }+ if (!visibleMatchCountCache.has(matches)) visibleMatchCountCache.set(matches, matches.filter(matchVisible).length);return visibleMatchCountCache.get(matches);};const resolvesUniquely = (sel, el) => {@@ -783,6 +792,9 @@ const selectorCandidatesFor = (const matches = cssMatchesFor(selector);const candidate = { selector: selector, source: source, match_count: matches === null ? null : matches.length };if (matches !== null) candidate.visible_match_count = visibleMatchCount(matches);+ if (matches !== null && candidate.visible_match_count < matches.length) {+ candidate.first_match_visible = matchVisible(matches[0]);+ }candidates.push(candidate);};const id = attr(el, 'id');@@ -846,30 +858,35 @@ const labelAnchorCandidateFor = (carrier, labelEl, label) => {anchor = anchor.parentElement;}const base = shapeSelector(anchor);- const labelShape = shapeSelector(labelEl);const inner = anchor === carrier ? '' : shapeSelector(carrier);- const selector = base + ':has(' + labelShape + ':text-is("' + cssAttr(label) + '"))' + (inner ? ' ' + inner : '');- if (selector.length > MAX_SELECTOR_CHARS) return null;- const anchors = new Set();- for (const root of openRoots()) {- let nodes; try { nodes = Array.from(root.querySelectorAll(labelShape)); } catch (e) { return null; }- for (const node of nodes) {- if (!ownTextRuns(node).includes(label)) continue;- for (let up = composedParent(node); up; up = composedParent(up)) if (up.matches && up.matches(base)) anchors.add(up);- }- }- let matched = Array.from(anchors);- if (inner) {- matched = [];+ // A label's classes are styling that can differ between the scouted frame and a run, so they are named only+ // where the bare tag would not leave this carrier as the single match.+ for (const labelShape of new Set([(labelEl.tagName || '*').toLowerCase(), shapeSelector(labelEl)])) {+ const selector = base + ':has(' + labelShape + ':text-is("' + cssAttr(label) + '"))' + (inner ? ' ' + inner : '');+ if (selector.length > MAX_SELECTOR_CHARS) continue;+ const anchors = new Set();for (const root of openRoots()) {- let nodes; try { nodes = Array.from(root.querySelectorAll(inner)); } catch (e) { return null; }+ let nodes; try { nodes = Array.from(root.querySelectorAll(labelShape)); } catch (e) { return null; }for (const node of nodes) {- for (let up = composedParent(node); up; up = composedParent(up)) if (anchors.has(up)) { matched.push(node); break; }+ if (!ownTextRuns(node).includes(label)) continue;+ for (let up = composedParent(node); up; up = composedParent(up)) if (up.matches && up.matches(base)) anchors.add(up);+ }+ }+ let matched = Array.from(anchors);+ if (inner) {+ matched = [];+ for (const root of openRoots()) {+ let nodes; try { nodes = Array.from(root.querySelectorAll(inner)); } catch (e) { return null; }+ for (const node of nodes) {+ for (let up = composedParent(node); up; up = composedParent(up)) if (anchors.has(up)) { matched.push(node); break; }+ }}}+ if (matched.length === 1 && matched[0] === carrier) {+ return { selector: selector, source: 'text_anchor', match_count: 1, visible_match_count: visibleMatchCount([carrier]) };+ }}- if (matched.length !== 1 || matched[0] !== carrier) return null;- return { selector: selector, source: 'text_anchor', match_count: 1, visible_match_count: visibleMatchCount([carrier]) };+ return null;};// A relation's key can provide one additional factual representation. It is appended after the// generic capture so its presence does not reorder or privilege another source.@@ -1745,6 +1762,7 @@ const boundedStructuredEvidence = (payload) => {delete value.pointer_at_center;delete value.element_at_center;delete value.visible_match_count;+ delete value.first_match_visible;Object.values(value).forEach(dropAuthoringFacts);};dropAuthoringFacts(payload);diff --git a/skyvern/forge/sdk/copilot/composition_evidence.py b/skyvern/forge/sdk/copilot/composition_evidence.pyindex 9c34dc56d..180b376dc 100644--- a/skyvern/forge/sdk/copilot/composition_evidence.py+++ b/skyvern/forge/sdk/copilot/composition_evidence.py@@ -3,7 +3,7 @@from __future__ import annotationsimport re-from collections.abc import Iterable, Mapping, Sequence+from collections.abc import Callable, Iterable, Mapping, Sequencefrom typing import Any, Protocolfrom urllib.parse import urljoin, urlparse@@ -84,8 +84,9 @@ _MAX_VISIBLE_CONTROLS = 6_MAX_CLICKABLE_CONTROLS = 12_MAX_DISCLOSURE_CONTROL_ID_CHARS = 120_POINTER_AT_CENTER_STATES = frozenset({"reaches_control", "reaches_other_element", "not_measured"})-_CANDIDATE_COUNT_KEYS: frozenset[str] = frozenset({"match_count", "visible_match_count"})-_AUTHORING_FACT_KEYS: frozenset[str] = frozenset({"pointer_at_center", "element_at_center", "visible_match_count"})+CANDIDATE_COUNT_KEYS: frozenset[str] = frozenset({"match_count", "visible_match_count", "first_match_visible"})+POINTER_FACT_KEYS: frozenset[str] = frozenset({"pointer_at_center", "element_at_center"})+_AUTHORING_FACT_KEYS: frozenset[str] = POINTER_FACT_KEYS | {"visible_match_count", "first_match_visible"}_MODAL_IDENTITY_PATTERNS: frozenset[str] = frozenset({"modal", "popup", "overlay", "dialog", "drawer", "lightbox"})_MODAL_ROLE_VALUES: frozenset[str] = frozenset({"dialog", "alertdialog"})_RENDERED_STYLE_SNAPSHOT_ATTR = "data-page-evidence-rendered-style"@@ -2230,30 +2231,31 @@ def _label_anchor_candidate(carrier: Any, label_node: Any, label: str) -> Scouteelse:return Nonebase = _shape_selector(anchor)- label_shape = _shape_selector(label_node)inner = "" if anchor is carrier else _shape_selector(carrier)- selector = f'{base}:has({label_shape}:text-is("{_css_attr(label)}"))' + (f" {inner}" if inner else "")- if len(selector) > _MAX_SELECTOR_CHARS:- return Nonebase_ids = {id(node) for node in _selector_matches(carrier, base) or []}- anchor_ids = {- id(parent)- for node in _selector_matches(carrier, label_shape) or []- if label in _own_text_runs(node)- for parent in node.parents- if id(parent) in base_ids- }- if inner:- matched = [- node- for node in _selector_matches(carrier, inner) or []- if any(id(parent) in anchor_ids for parent in node.parents)- ]- else:- matched = [node for node in _selector_matches(carrier, base) or [] if id(node) in anchor_ids]- if len(matched) != 1 or matched[0] is not carrier:- return None- return {"selector": selector, "source": "text_anchor", "match_count": 1}+ # Mirror of the page side: the label's classes are named only where its bare tag is ambiguous.+ for label_shape in dict.fromkeys((str(label_node.name or "*").lower(), _shape_selector(label_node))):+ selector = f'{base}:has({label_shape}:text-is("{_css_attr(label)}"))' + (f" {inner}" if inner else "")+ if len(selector) > _MAX_SELECTOR_CHARS:+ continue+ anchor_ids = {+ id(parent)+ for node in _selector_matches(carrier, label_shape) or []+ if label in _own_text_runs(node)+ for parent in node.parents+ if id(parent) in base_ids+ }+ if inner:+ matched = [+ node+ for node in _selector_matches(carrier, inner) or []+ if any(id(parent) in anchor_ids for parent in node.parents)+ ]+ else:+ matched = [node for node in _selector_matches(carrier, base) or [] if id(node) in anchor_ids]+ if len(matched) == 1 and matched[0] is carrier:+ return {"selector": selector, "source": "text_anchor", "match_count": 1}+ return Nonedef _relation_selector_candidates(carrier: Any, key_text: str, label_node: Any) -> list[ScoutedSelectorCandidate]:@@ -3439,6 +3441,9 @@ def _structured_selector_candidates(value: Any) -> list[ScoutedSelectorCandidateand 0 <= visible_count <= match_count):candidate["visible_match_count"] = visible_count+ first_visible = item.get("first_match_visible")+ if isinstance(first_visible, bool) and match_count > 1:+ candidate["first_match_visible"] = first_visiblecandidates.append(candidate)return candidates@@ -3470,15 +3475,10 @@ def _attach_node_evidence(entry: dict[str, Any], node: dict[str, Any]) -> dict[sdef model_visible_composition_evidence(evidence: dict[str, Any], *, authoring_facts: bool = False) -> dict[str, Any]:- """Project stored page evidence into factual model input without a preferred locator.-- Capture keeps singular CSS selectors for internal measurement, binding, and execution joins.- At this model boundary every singular selector alias and its count are removed, including- key/value relations that do not carry candidate packets. Candidate selectors and all unrelated- facts retain their original order and values. `authoring_facts` also keeps a control's- `pointer_at_center` and `element_at_center` and, on a candidate whose visible matches were measured, `match_count`- when it is not 1 and `visible_match_count` when fewer of those matches are visible.- """+ """Capture keeps singular selectors for measurement, binding and execution joins, so this model boundary drops+ every singular alias and its count while candidate selectors and other facts keep their order and values.+ `authoring_facts` also keeps pointer facts and a measured candidate's `match_count` when not 1, plus+ `visible_match_count` and `first_match_visible` when some of its matches are hidden."""# `expression` embeds a singular selector, its match count and its index in one string, so# dropping only the scalar aliases would let the same facts cross as prose.@@ -3494,7 +3494,7 @@ def model_visible_composition_evidence(evidence: dict[str, Any], *, authoring_fa}def project_candidate(candidate: dict[str, Any]) -> dict[str, Any]:- projected = {key: project(value) for key, value in candidate.items() if key not in _CANDIDATE_COUNT_KEYS}+ projected = {key: project(value) for key, value in candidate.items() if key not in CANDIDATE_COUNT_KEYS}match_count = candidate.get("match_count")visible_count = candidate.get("visible_match_count")# A total without its measured visible count would read as every match being visible.@@ -3504,6 +3504,8 @@ def model_visible_composition_evidence(evidence: dict[str, Any], *, authoring_faprojected["match_count"] = match_countif visible_count < match_count:projected["visible_match_count"] = visible_count+ if isinstance(candidate.get("first_match_visible"), bool):+ projected["first_match_visible"] = candidate["first_match_visible"]return projecteddef project(value: Any) -> Any:@@ -3527,17 +3529,52 @@ def model_visible_composition_evidence(evidence: dict[str, Any], *, authoring_fareturn project(evidence)-def without_authoring_facts(evidence: Mapping[str, Any]) -> dict[str, Any]:- """Copy of stored page evidence with no pointer facts and no candidate `visible_match_count`."""+def without_authoring_facts(evidence: Mapping[str, Any], keys: frozenset[str] = _AUTHORING_FACT_KEYS) -> dict[str, Any]:+ """Copy of stored page evidence without `keys`: by default every pointer fact and candidate visibility fact."""def strip(value: Any) -> Any:if isinstance(value, dict):- return {key: strip(child) for key, child in value.items() if key not in _AUTHORING_FACT_KEYS}+ return {key: strip(child) for key, child in value.items() if key not in keys}if isinstance(value, list):return [strip(child) for child in value]return value- return {key: strip(child) for key, child in evidence.items() if key not in _AUTHORING_FACT_KEYS}+ return {key: strip(child) for key, child in evidence.items() if key not in keys}+++def _has_hidden_first_match(value: Any) -> bool:+ if isinstance(value, dict):+ return value.get("first_match_visible") is False or any(_has_hidden_first_match(v) for v in value.values())+ if isinstance(value, list):+ return any(_has_hidden_first_match(child) for child in value)+ return False+++def _hidden_first_counts_only(value: Any) -> Any:+ if isinstance(value, dict):+ keep = value.get("first_match_visible") is False+ return {+ key: _hidden_first_counts_only(child)+ for key, child in value.items()+ if key not in POINTER_FACT_KEYS and (keep or key not in CANDIDATE_COUNT_KEYS)+ }+ if isinstance(value, list):+ return [_hidden_first_counts_only(child) for child in value]+ return value+++def fit_authoring_facts(+ projected: dict[str, Any], bare: dict[str, Any], fits: Callable[[dict[str, Any]], bool]+) -> dict[str, Any]:+ """`projected` if it fits, else `bare` with the counts of each value-relation candidate whose first match is+ hidden, so a read's own hidden copy stays visible while the rest of the packet is exactly the fallback."""+ if fits(projected):+ return projected+ relations = projected.get("key_value_relations")+ if not _has_hidden_first_match(relations):+ return bare+ read_counts = {**bare, "key_value_relations": _hidden_first_counts_only(relations)}+ return bare if not fits(read_counts) and fits(bare) else read_countsdef _attach_pointer_at_center(entry: dict[str, Any], node: dict[str, Any]) -> dict[str, Any]:diff --git a/skyvern/forge/sdk/copilot/context.py b/skyvern/forge/sdk/copilot/context.pyindex 70c654f8d..3b8051e83 100644--- a/skyvern/forge/sdk/copilot/context.py+++ b/skyvern/forge/sdk/copilot/context.py@@ -73,6 +73,25 @@ class NarrativeDraft(TypedDict):REPLY_TOOL_NAME = "reply"+EDIT_BLOCK_TOOL_NAME = "edit_block"+EDIT_BLOCK_AND_RUN_TOOL_NAME = "edit_block_and_run"+UPDATE_AND_RUN_BLOCKS_TOOL_NAME = "update_and_run_blocks"+UPDATE_WORKFLOW_TOOL_NAME = "update_workflow"+RUN_BLOCKS_TOOL_NAME = "run_blocks_and_collect_debug"+BLANK_BROWSER_TEST_TOOL_NAME = "test_workflow_from_blank_browser"+BLOCK_RUNNING_TOOLS = frozenset(+ {RUN_BLOCKS_TOOL_NAME, UPDATE_AND_RUN_BLOCKS_TOOL_NAME, EDIT_BLOCK_AND_RUN_TOOL_NAME, BLANK_BROWSER_TEST_TOOL_NAME}+)+WORKFLOW_MUTATION_TOOLS = frozenset(+ {+ UPDATE_WORKFLOW_TOOL_NAME,+ UPDATE_AND_RUN_BLOCKS_TOOL_NAME,+ EDIT_BLOCK_TOOL_NAME,+ EDIT_BLOCK_AND_RUN_TOOL_NAME,+ "add_block",+ "delete_block",+ }+)USER_FACING_REASON_PARAM = "user_facing_reason"USER_FACING_REASON_SCHEMA = {"type": ["string", "null"],@@ -304,6 +323,7 @@ class TurnNarrativePayload(TypedDict):testedBlockFingerprints: NotRequired[dict[str, list[str]]]budgetExpiry: NotRequired[NarrativeBudgetExpiry]turnFacts: NotRequired[NarrativeTurnFacts]+ demonstrationItems: NotRequired[list[str]]if TYPE_CHECKING:@@ -1310,6 +1330,11 @@ class CopilotContext(AgentContext):credential_generation_spent: bool = Falsecredential_pause_connected_credential_id: str | None = Nonecredential_pause_anchor_tool_call_id: str | None = None+ demonstration_requested: bool = False+ demonstration_ran: bool = False+ demonstration_id: str | None = None+ demonstration_insertion: dict[str, str | None] | None = None+ demonstration_items: list[str] = field(default_factory=list)# Set while a ``request_credential`` ask is open, so tool calls issued alongside it in the same# model response wait for the user's answer instead of racing it.credential_pause_settled: asyncio.Event | None = Nonediff --git a/skyvern/forge/sdk/copilot/credential_pause.py b/skyvern/forge/sdk/copilot/credential_pause.pyindex f37a2917a..faca23cc4 100644--- a/skyvern/forge/sdk/copilot/credential_pause.py+++ b/skyvern/forge/sdk/copilot/credential_pause.py@@ -811,7 +811,7 @@ def _bind_connected_credential_origin(policy: RequestPolicy, credential: Credentreturn admitted_url-def _apply_connected_credential_to_policy(ctx: Any, policy: RequestPolicy, credential: Credential) -> None:+def apply_connected_credential_to_policy(ctx: CopilotContext, policy: RequestPolicy, credential: Credential) -> None:"""Record ``credential`` as the explicit answer to the product-owned pause, reopening the run gate(and un-latching ``test_after_update_done``) so the resumed run does not re-skip and re-ask; araw-secret turn keeps its run gate closed and its draft untested for a later secret-free turn."""@@ -1274,7 +1274,7 @@ async def _run_credential_pause(await answered("not_admitted")return resolutionif isinstance(policy, RequestPolicy):- _apply_connected_credential_to_policy(ctx, policy, credential)+ apply_connected_credential_to_policy(ctx, policy, credential)ctx.credential_pause_outcome = "connected"await answered("connected", credential)return replace(resolution, generated=True) if is_generated else resolutiondiff --git a/skyvern/forge/sdk/copilot/demonstration_recorder.py b/skyvern/forge/sdk/copilot/demonstration_recorder.pynew file mode 100644index 000000000..a0c726bcb--- /dev/null+++ b/skyvern/forge/sdk/copilot/demonstration_recorder.py@@ -0,0 +1,1045 @@+from __future__ import annotations++import asyncio+import contextlib+import json+import secrets+import time+from collections.abc import Awaitable, Coroutine+from dataclasses import dataclass+from datetime import UTC, datetime, timedelta+from http import HTTPStatus+from typing import Any, Literal, cast+from urllib.parse import urlsplit++import structlog+from playwright.async_api import Error as PlaywrightError+from playwright.async_api import Page++from skyvern.cli.mcp_tools._localhost import is_localhost_url+from skyvern.config import settings+from skyvern.exceptions import BlockedHost, SkyvernHTTPException+from skyvern.forge import app+from skyvern.forge.sdk.cache.base import BaseCache+from skyvern.forge.sdk.copilot.config import ALL_BLOCK_FAMILIES, AuthoringCapability+from skyvern.forge.sdk.copilot.context import CopilotContext+from skyvern.forge.sdk.copilot.credential_pause import apply_connected_credential_to_policy+from skyvern.forge.sdk.copilot.credential_resolution import safe_admitted_url+from skyvern.forge.sdk.copilot.demonstration_tap import DEMONSTRATION_MAX_EVENTS, PageTap, TapEvent+from skyvern.forge.sdk.copilot.request_policy import RequestPolicy+from skyvern.forge.sdk.copilot.runtime import (+ ScoutedInteraction,+ record_attached_browser_driver,+ resolve_persistent_browser_state,+ start_browser_driver_release,+ wait_for_driver_releases,+)+from skyvern.forge.sdk.copilot.secret_redaction import redact_raw_secrets_for_prompt+from skyvern.forge.sdk.copilot.secret_scrub import scrub_secrets_from_structure+from skyvern.forge.sdk.copilot.tools.credential_fill import fill_credential_field+from skyvern.forge.sdk.core.event_source_stream import EventSourceStream+from skyvern.forge.sdk.schemas.workflow_copilot import WorkflowCopilotDemonstrationControlRequest+from skyvern.utils.url_validators import validate_fetch_url++LOG = structlog.get_logger()+_SHIELDED_TASKS: set[asyncio.Task[Any]] = set()+++async def finish_shielded(coro: Coroutine[Any, Any, Any]) -> Any:+ # The loop keeps only weak task references, so a shielded task outliving its cancelled caller needs a strong one.+ task = asyncio.ensure_future(coro)+ _SHIELDED_TASKS.add(task)+ task.add_done_callback(_SHIELDED_TASKS.discard)+ return await asyncio.shield(task)+++DEMONSTRATION_TTL = timedelta(hours=2)+DEMONSTRATION_CANCELLED_TTL = timedelta(seconds=60)+DEMONSTRATION_MAX_CONTROLS = 200+DEMONSTRATION_MAX_VALUE_CHARS = 100_000+DEMONSTRATION_POLL_SECONDS = 0.25+DEMONSTRATION_OWNER_REFRESH_SECONDS = 5+DEMONSTRATION_OWNER_LIVE_SECONDS = 20+DEMONSTRATION_MAX_SECONDS = 30 * 60+DEMONSTRATION_REPLAY_LEASE = timedelta(minutes=5)++DemonstrationStatus = Literal["recording", "stopping", "stopped", "replaying", "consumed", "cancelled"]+DemonstrationStopReason = Literal[+ "stopped", "cancelled", "disconnected", "browser_gone", "time_limit", "event_limit", "consumed"+]+++@dataclass+class DemonstrationRecord:+ workflow_permanent_id: str+ browser_session_id: str+ status: DemonstrationStatus+ event_count: int+ owner: str | None+ owner_seen_at: datetime | None+ final_url: str | None+ stop_reason: DemonstrationStopReason | None = None+ insertion: dict[str, str | None] | None = None+ replay_started_at: datetime | None = None+ replay_token: str | None = None+ recording_started_at: datetime | None = None+ value_chars: int = 0+++@dataclass(frozen=True)+class DemonstrationClaim:+ demonstration_id: str+ owner: str+ resumed: bool+ control_cursor: int+++class DemonstrationRejection(Exception):+ def __init__(self, status_code: HTTPStatus, detail: str) -> None:+ super().__init__(detail)+ self.status_code = status_code+ self.detail = detail+++class _NoopStream:+ async def send(self, data: Any) -> bool:+ return True++ async def is_disconnected(self) -> bool:+ return False++ async def close(self) -> None:+ return None+++def demonstration_record_key(organization_id: str, demonstration_id: str) -> str:+ return f"copilot_demonstration:{organization_id}:{demonstration_id}"+++def demonstration_event_key(organization_id: str, demonstration_id: str, seq: int) -> str:+ return f"copilot_demonstration_event:{organization_id}:{demonstration_id}:{seq}"+++def demonstration_controls_key(organization_id: str, demonstration_id: str) -> str:+ return f"copilot_demonstration_controls:{organization_id}:{demonstration_id}"+++def demonstration_lock_key(organization_id: str, demonstration_id: str) -> str:+ return f"copilot_demonstration_lock:{organization_id}:{demonstration_id}"+++def demonstration_browser_key(organization_id: str, browser_session_id: str) -> str:+ return f"copilot_demonstration_browser:{organization_id}:{browser_session_id}"+++def encode_demonstration_record(record: DemonstrationRecord) -> str:+ return json.dumps(+ {+ "workflow_permanent_id": record.workflow_permanent_id,+ "browser_session_id": record.browser_session_id,+ "status": record.status,+ "event_count": record.event_count,+ "owner": record.owner,+ "owner_seen_at": record.owner_seen_at.isoformat() if record.owner_seen_at else None,+ "final_url": record.final_url,+ "stop_reason": record.stop_reason,+ "insertion": record.insertion,+ "replay_started_at": record.replay_started_at.isoformat() if record.replay_started_at else None,+ "replay_token": record.replay_token,+ "recording_started_at": record.recording_started_at.isoformat() if record.recording_started_at else None,+ "value_chars": record.value_chars,+ },+ separators=(",", ":"),+ )+++def decode_demonstration_record(raw: Any) -> DemonstrationRecord | None:+ try:+ data = json.loads(raw)+ except (TypeError, ValueError):+ return None+ if not isinstance(data, dict):+ return None+ workflow_permanent_id = data.get("workflow_permanent_id")+ browser_session_id = data.get("browser_session_id")+ record_status = data.get("status")+ event_count = data.get("event_count")+ if (+ not isinstance(workflow_permanent_id, str)+ or not isinstance(browser_session_id, str)+ or record_status not in {"recording", "stopping", "stopped", "replaying", "consumed", "cancelled"}+ or not isinstance(event_count, int)+ or isinstance(event_count, bool)+ or event_count < 0+ ):+ return None+ owner = data.get("owner")+ final_url = data.get("final_url")+ owner_seen_at = _decode_timestamp(data.get("owner_seen_at"))+ replay_started_at = _decode_timestamp(data.get("replay_started_at"))+ return DemonstrationRecord(+ workflow_permanent_id=workflow_permanent_id,+ browser_session_id=browser_session_id,+ status=record_status,+ event_count=event_count,+ owner=owner if isinstance(owner, str) else None,+ owner_seen_at=owner_seen_at,+ final_url=final_url if isinstance(final_url, str) else None,+ value_chars=data["value_chars"] if isinstance(data.get("value_chars"), int) else 0,+ stop_reason=(+ data.get("stop_reason")+ if data.get("stop_reason")+ in {"stopped", "cancelled", "disconnected", "browser_gone", "time_limit", "event_limit", "consumed"}+ else None+ ),+ insertion=(+ {+ key: value+ for key, value in data["insertion"].items()+ if key in {"after_label", "before_label"} and (isinstance(value, str) or value is None)+ }+ if isinstance(data.get("insertion"), dict)+ else None+ ),+ replay_started_at=replay_started_at,+ replay_token=data.get("replay_token") if isinstance(data.get("replay_token"), str) else None,+ recording_started_at=_decode_timestamp(data.get("recording_started_at")),+ )+++def _decode_timestamp(raw: Any) -> datetime | None:+ try:+ value = datetime.fromisoformat(raw) if isinstance(raw, str) and raw else None+ except ValueError:+ return None+ if value is None:+ return None+ return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)+++def demonstration_time_limit_reached(record: DemonstrationRecord, now: datetime | None = None) -> bool:+ started_at = record.recording_started_at+ return started_at is not None and (now or datetime.now(UTC)) - started_at >= timedelta(+ seconds=DEMONSTRATION_MAX_SECONDS+ )+++def decode_demonstration_controls(raw: Any) -> list[dict[str, Any]]:+ try:+ data = json.loads(raw) if raw else []+ except (TypeError, ValueError):+ return []+ return [entry for entry in data if isinstance(entry, dict)] if isinstance(data, list) else []+++def demonstration_owner_is_live(record: DemonstrationRecord, now: datetime | None = None) -> bool:+ if record.owner is None or record.owner_seen_at is None:+ return False+ return (now or datetime.now(UTC)) - record.owner_seen_at < timedelta(seconds=DEMONSTRATION_OWNER_LIVE_SECONDS)+++def demonstration_limit_reached(record: DemonstrationRecord) -> bool:+ return record.event_count >= DEMONSTRATION_MAX_EVENTS or record.value_chars >= DEMONSTRATION_MAX_VALUE_CHARS+++def settle_abandoned_demonstration(record: DemonstrationRecord, now: datetime | None = None) -> bool:+ if record.status in {"recording", "stopping"} and not demonstration_owner_is_live(record, now):+ # A user stop whose owner died before the final drain may be missing buffered events, so replay must say so.+ if record.status == "recording" or record.stop_reason in {None, "stopped"}:+ record.stop_reason = "disconnected"+ record.status = "cancelled" if record.stop_reason == "cancelled" else "stopped"+ record.owner = None+ record.owner_seen_at = None+ return True+ if record.status != "replaying" or record.replay_started_at is None:+ return False+ if (now or datetime.now(UTC)) - record.replay_started_at < DEMONSTRATION_REPLAY_LEASE:+ return False+ record.status = "stopped"+ record.replay_started_at = None+ record.replay_token = None+ return True+++async def load_demonstration_record(+ cache: BaseCache, organization_id: str, demonstration_id: str+) -> DemonstrationRecord | None:+ return decode_demonstration_record(await cache.get(demonstration_record_key(organization_id, demonstration_id)))+++async def expire_demonstration_payload(+ cache: BaseCache, organization_id: str, demonstration_id: str, event_count: int, ttl: timedelta+) -> None:+ keys = [demonstration_event_key(organization_id, demonstration_id, seq) for seq in range(1, event_count + 1)]+ keys.append(demonstration_controls_key(organization_id, demonstration_id))+ values = await asyncio.gather(*(cache.get(key) for key in keys))+ await asyncio.gather(+ *(cache.set(key, value, ex=ttl) for key, value in zip(keys, values, strict=True) if value is not None)+ )+++async def store_settled_demonstration(+ cache: BaseCache, organization_id: str, demonstration_id: str, record: DemonstrationRecord+) -> None:+ if not settle_abandoned_demonstration(record):+ return+ ttl = DEMONSTRATION_CANCELLED_TTL if record.status == "cancelled" else DEMONSTRATION_TTL+ await cache.set(+ demonstration_record_key(organization_id, demonstration_id), encode_demonstration_record(record), ex=ttl+ )+ if record.status == "cancelled":+ await expire_demonstration_payload(cache, organization_id, demonstration_id, record.event_count, ttl)+++async def claim_demonstration_recording(+ cache: BaseCache,+ *,+ organization_id: str,+ workflow_permanent_id: str,+ browser_session_id: str,+ demonstration_id: str | None,+ insertion: dict[str, str | None] | None = None,+) -> DemonstrationClaim:+ browser_key = demonstration_browser_key(organization_id, browser_session_id)+ replaced: tuple[str, int] | None = None+ # A Redis lock stores its token under its own name, so it cannot share the index key it guards.+ async with cache.get_lock(f"{browser_key}:lock"):+ indexed_id = await cache.get(browser_key)+ if isinstance(indexed_id, bytes):+ indexed_id = indexed_id.decode()+ if demonstration_id is not None:+ if isinstance(indexed_id, str) and indexed_id and indexed_id != demonstration_id:+ async with cache.get_lock(demonstration_lock_key(organization_id, indexed_id)):+ indexed = await load_demonstration_record(cache, organization_id, indexed_id)+ if (+ indexed is not None+ and indexed.status in {"recording", "stopping"}+ and demonstration_owner_is_live(indexed)+ ):+ raise DemonstrationRejection(+ HTTPStatus.CONFLICT, "Another demonstration is recording on this browser"+ )+ async with cache.get_lock(demonstration_lock_key(organization_id, demonstration_id)):+ record = await load_demonstration_record(cache, organization_id, demonstration_id)+ if record is None or (+ record.workflow_permanent_id != workflow_permanent_id+ or record.browser_session_id != browser_session_id+ ):+ raise DemonstrationRejection(HTTPStatus.NOT_FOUND, "Unknown demonstration")+ if record.status not in {"recording", "stopped"}:+ raise DemonstrationRejection(HTTPStatus.CONFLICT, "Demonstration cannot be resumed")+ if demonstration_limit_reached(record):+ raise DemonstrationRejection(HTTPStatus.CONFLICT, "Demonstration has reached the event limit")+ if demonstration_time_limit_reached(record):+ raise DemonstrationRejection(HTTPStatus.CONFLICT, "Demonstration has reached its time limit")+ if demonstration_owner_is_live(record):+ raise DemonstrationRejection(HTTPStatus.CONFLICT, "Demonstration is already recording")+ controls = decode_demonstration_controls(+ await cache.get(demonstration_controls_key(organization_id, demonstration_id))+ )+ owner = secrets.token_urlsafe(24)+ record.status = "recording"+ record.owner = owner+ record.owner_seen_at = datetime.now(UTC)+ record.stop_reason = None+ record.final_url = None+ await cache.set(+ demonstration_record_key(organization_id, demonstration_id),+ encode_demonstration_record(record),+ ex=DEMONSTRATION_TTL,+ )+ await cache.set(browser_key, demonstration_id, ex=DEMONSTRATION_TTL)+ return DemonstrationClaim(+ demonstration_id=demonstration_id,+ owner=owner,+ resumed=True,+ control_cursor=len(controls),+ )++ if isinstance(indexed_id, str) and indexed_id:+ async with cache.get_lock(demonstration_lock_key(organization_id, indexed_id)):+ indexed = await load_demonstration_record(cache, organization_id, indexed_id)+ if (+ indexed is not None+ and indexed.status in {"recording", "stopping"}+ and demonstration_owner_is_live(indexed)+ ):+ raise DemonstrationRejection(+ HTTPStatus.CONFLICT, "Another demonstration is recording on this browser"+ )+ if indexed is not None and indexed.status in {"recording", "stopping", "stopped"}:+ indexed.status = "cancelled"+ indexed.owner = None+ indexed.owner_seen_at = None+ await cache.set(+ demonstration_record_key(organization_id, indexed_id),+ encode_demonstration_record(indexed),+ ex=DEMONSTRATION_CANCELLED_TTL,+ )+ replaced = (indexed_id, indexed.event_count)++ demonstration_id = secrets.token_urlsafe(24)+ owner = secrets.token_urlsafe(24)+ record = DemonstrationRecord(+ workflow_permanent_id=workflow_permanent_id,+ browser_session_id=browser_session_id,+ status="recording",+ event_count=0,+ owner=owner,+ owner_seen_at=datetime.now(UTC),+ final_url=None,+ insertion=insertion,+ recording_started_at=datetime.now(UTC),+ )+ await cache.set(demonstration_controls_key(organization_id, demonstration_id), "[]", ex=DEMONSTRATION_TTL)+ await cache.set(+ demonstration_record_key(organization_id, demonstration_id),+ encode_demonstration_record(record),+ ex=DEMONSTRATION_TTL,+ )+ await cache.set(browser_key, demonstration_id, ex=DEMONSTRATION_TTL)+ if replaced is not None:+ await expire_demonstration_payload(+ cache, organization_id, replaced[0], replaced[1], DEMONSTRATION_CANCELLED_TTL+ )+ return DemonstrationClaim(+ demonstration_id=demonstration_id,+ owner=owner,+ resumed=False,+ control_cursor=0,+ )+++async def append_demonstration_control(+ cache: BaseCache,+ *,+ organization_id: str,+ request: WorkflowCopilotDemonstrationControlRequest,+) -> int:+ record_key = demonstration_record_key(organization_id, request.demonstration_id)+ controls_key = demonstration_controls_key(organization_id, request.demonstration_id)+ async with cache.get_lock(demonstration_lock_key(organization_id, request.demonstration_id)):+ record = decode_demonstration_record(await cache.get(record_key))+ if record is None:+ raise DemonstrationRejection(HTTPStatus.NOT_FOUND, "Unknown demonstration")+ if record.status not in {"recording", "stopping", "stopped"}:+ raise DemonstrationRejection(HTTPStatus.CONFLICT, "Demonstration has ended")+ if request.action == "stop":+ if record.status == "recording":+ record.status = "stopping"+ record.stop_reason = "stopped"+ await cache.set(record_key, encode_demonstration_record(record), ex=DEMONSTRATION_TTL)+ return 0+ if request.action == "cancel":+ if demonstration_owner_is_live(record):+ # The recorder commits the terminal state itself, so a stream dropped with the cancel still discards.+ record.status = "stopping"+ record.stop_reason = "cancelled"+ await cache.set(record_key, encode_demonstration_record(record), ex=DEMONSTRATION_TTL)+ return 0+ record.status = "cancelled"+ record.stop_reason = "cancelled"+ record.owner = None+ record.owner_seen_at = None+ await cache.set(record_key, encode_demonstration_record(record), ex=DEMONSTRATION_CANCELLED_TTL)+ discarded_events = record.event_count+ else:+ controls = decode_demonstration_controls(await cache.get(controls_key))+ if len(controls) >= DEMONSTRATION_MAX_CONTROLS:+ raise DemonstrationRejection(HTTPStatus.CONFLICT, "Too many browser controls for this demonstration")+ seq = len(controls) + 1+ entry = {"seq": seq, **request.model_dump(exclude_none=True, exclude={"demonstration_id"})}+ controls.append(entry)+ await cache.set(controls_key, json.dumps(controls, separators=(",", ":")), ex=DEMONSTRATION_TTL)+ return seq+ await expire_demonstration_payload(+ cache, organization_id, request.demonstration_id, discarded_events, DEMONSTRATION_CANCELLED_TTL+ )+ return 0+++async def active_demonstration(+ cache: BaseCache,+ *,+ organization_id: str,+ workflow_permanent_id: str,+ browser_session_id: str,+) -> tuple[str, DemonstrationRecord] | None:+ demonstration_id = await cache.get(demonstration_browser_key(organization_id, browser_session_id))+ if isinstance(demonstration_id, bytes):+ demonstration_id = demonstration_id.decode()+ if not isinstance(demonstration_id, str) or not demonstration_id:+ return None+ record_key = demonstration_record_key(organization_id, demonstration_id)+ async with cache.get_lock(demonstration_lock_key(organization_id, demonstration_id)):+ record = decode_demonstration_record(await cache.get(record_key))+ if record is not None:+ await store_settled_demonstration(cache, organization_id, demonstration_id, record)+ if (+ record is None+ or record.workflow_permanent_id != workflow_permanent_id+ or record.browser_session_id != browser_session_id+ or record.status not in {"recording", "stopping", "stopped"}+ ):+ return None+ return demonstration_id, record+++async def validate_demonstration_for_replay(+ cache: BaseCache,+ *,+ organization_id: str,+ workflow_permanent_id: str,+ demonstration_id: str,+) -> DemonstrationRecord:+ record_key = demonstration_record_key(organization_id, demonstration_id)+ async with cache.get_lock(demonstration_lock_key(organization_id, demonstration_id)):+ record = decode_demonstration_record(await cache.get(record_key))+ if record is None or record.workflow_permanent_id != workflow_permanent_id:+ raise DemonstrationRejection(HTTPStatus.NOT_FOUND, "Unknown demonstration")+ await store_settled_demonstration(cache, organization_id, demonstration_id, record)+ if record.status in {"recording", "stopping"}:+ raise DemonstrationRejection(HTTPStatus.CONFLICT, "Demonstration is still recording")+ if record.status != "stopped":+ raise DemonstrationRejection(HTTPStatus.CONFLICT, "Demonstration has already ended")+ return record+++async def demonstration_action_labels(+ cache: BaseCache, organization_id: str, demonstration_id: str, event_count: int+) -> list[str]:+ labels: list[str] = []+ events = await asyncio.gather(+ *(+ cache.get(demonstration_event_key(organization_id, demonstration_id, seq))+ for seq in range(1, event_count + 1)+ )+ )+ for raw in events:+ try:+ event = json.loads(raw)+ except (TypeError, ValueError):+ continue+ if isinstance(event, dict) and event.get("kind") == "type":+ packet = event.get("packet")+ role_name = packet.get("role_name") if isinstance(packet, dict) else None+ accessible_name = role_name.get("accessible_name") if isinstance(role_name, dict) else None+ target = safe_label(accessible_name) or safe_label(str(event.get("selector") or "field"))+ labels.append(f'Type into "{target}"'[:60])+ continue+ label = event.get("label") if isinstance(event, dict) else None+ if isinstance(label, str) and label:+ labels.append(label)+ return labels+++class DemonstrationRecorder:+ def __init__(+ self,+ *,+ cache: BaseCache,+ organization_id: str,+ workflow_permanent_id: str,+ browser_session_id: str,+ claim: DemonstrationClaim,+ stream: EventSourceStream,+ api_key: str | None = None,+ authoring_capability: AuthoringCapability = ALL_BLOCK_FAMILIES,+ ) -> None:+ self.cache = cache+ self.organization_id = organization_id+ self.workflow_permanent_id = workflow_permanent_id+ self.browser_session_id = browser_session_id+ self.demonstration_id = claim.demonstration_id+ self.owner = claim.owner+ self.resumed = claim.resumed+ self.control_cursor = claim.control_cursor+ self.stream = stream+ self.tap: PageTap | None = None+ self.page: Page | None = None+ self.sensitive_count = 0+ self.event_limit_reached = False+ self.credential_filled_selectors: set[str] = set()+ self.started_at = time.monotonic()+ self.recording_ctx = CopilotContext(+ organization_id=organization_id,+ workflow_id="",+ workflow_permanent_id=workflow_permanent_id,+ workflow_yaml=None,+ browser_session_id=browser_session_id,+ stream=_NoopStream(),+ request_policy=RequestPolicy(),+ api_key=api_key,+ )+ self.recording_ctx.authoring_capability = authoring_capability++ async def run(self) -> None:+ record = await load_demonstration_record(self.cache, self.organization_id, self.demonstration_id)+ action_count = record.event_count if record is not None else 0+ LOG.info("copilot_demonstration_started", resumed=self.resumed)+ reason: DemonstrationStopReason = "browser_gone"+ next_refresh = time.monotonic() + DEMONSTRATION_OWNER_REFRESH_SECONDS+ try:+ browser_state = await resolve_persistent_browser_state(+ session_id=self.browser_session_id,+ organization_id=self.organization_id,+ )+ if browser_state is None or browser_state.browser_context is None:+ return+ record_attached_browser_driver(self.recording_ctx, self.browser_session_id, browser_state)+ browser_context = browser_state.browser_context+ pages = list(browser_context.pages)+ self.page = await browser_state.get_working_page(prune_excess_pages=False)+ if self.page is None and pages:+ self.page = pages[-1]+ self.tap = PageTap(browser_context, max_events=DEMONSTRATION_MAX_EVENTS)+ await self.tap.start(pages)+ with contextlib.suppress(Exception):+ await self.tap.set_intercept_file_chooser_dialog(True)+ await self.stream.send(+ {+ "type": "demonstration_started",+ "demonstration_id": self.demonstration_id,+ "browser_session_id": self.browser_session_id,+ "action_count": action_count,+ "resumed": self.resumed,+ "can_fill_credentials": self.recording_ctx.authoring_capability.code_blocks,+ }+ )+ while True:+ record = await load_demonstration_record(self.cache, self.organization_id, self.demonstration_id)+ if record is None or record.owner != self.owner:+ reason = (+ record.stop_reason if record is not None and record.stop_reason is not None else "cancelled"+ )+ break+ if record.status in {"stopping", "stopped"}:+ reason = await self._run_controls() or record.stop_reason or "stopped"+ break+ if record.status == "consumed":+ reason = "consumed"+ break+ if record.status == "cancelled":+ reason = "cancelled"+ break+ if await self.stream.is_disconnected():+ reason = record.stop_reason or "disconnected"+ break+ working_pages = [page for page in browser_context.pages if not page.is_closed()]+ if self.page is None or self.page.is_closed():+ self.page = working_pages[-1] if working_pages else None+ if self.page is None:+ reason = "browser_gone"+ break+ if demonstration_time_limit_reached(record):+ reason = "time_limit"+ break+ stop_reason = await self._drain_tap()+ if stop_reason is not None:+ reason = stop_reason+ break+ stop_reason = await self._run_controls()+ if stop_reason is not None:+ reason = stop_reason+ break+ if time.monotonic() >= next_refresh:+ await self._refresh_owner()+ next_refresh = time.monotonic() + DEMONSTRATION_OWNER_REFRESH_SECONDS+ await asyncio.sleep(DEMONSTRATION_POLL_SECONDS)+ finally:+ await finish_shielded(self._stop(reason))++ async def _append_event(self, event: dict[str, Any]) -> int | None:+ record_key = demonstration_record_key(self.organization_id, self.demonstration_id)+ async with self.cache.get_lock(demonstration_lock_key(self.organization_id, self.demonstration_id)):+ record = decode_demonstration_record(await self.cache.get(record_key))+ if record is None or record.owner != self.owner or record.status not in {"recording", "stopping"}:+ return None+ if demonstration_limit_reached(record):+ self.event_limit_reached = True+ if record.status == "recording":+ record.status = "stopping"+ record.stop_reason = "event_limit"+ await self.cache.set(record_key, encode_demonstration_record(record), ex=DEMONSTRATION_TTL)+ return None+ seq = record.event_count + 1+ stored = {"seq": seq, **event}+ await self.cache.set(+ demonstration_event_key(self.organization_id, self.demonstration_id, seq),+ json.dumps(stored, separators=(",", ":")),+ ex=DEMONSTRATION_TTL,+ )+ record.event_count = seq+ value = event.get("value")+ record.value_chars += len(value) if isinstance(value, str) else 0+ if demonstration_limit_reached(record):+ self.event_limit_reached = True+ if record.status == "recording":+ record.status = "stopping"+ record.stop_reason = "event_limit"+ await self.cache.set(record_key, encode_demonstration_record(record), ex=DEMONSTRATION_TTL)+ return seq++ async def _drain_tap(self) -> DemonstrationStopReason | None:+ if self.tap is None:+ return None+ for event in self.tap.drain_events():+ await self._record_tap_event(event)+ if self.event_limit_reached:+ return "event_limit"+ return "event_limit" if self.tap.event_limit_reached else None++ async def _record_tap_event(self, event: TapEvent) -> None:+ self.page = event.page+ payload = event.payload+ frame: dict[str, Any] = {"is_main": event.is_main_frame, "name": event.frame_name}+ if not event.is_main_frame:+ path = await self.tap.frame_path(event) if self.tap is not None else None+ if path is None or len(path) > 1:+ LOG.info("copilot_demonstration_nested_frame_unsupported")+ await self._record_gap_event(event, frame, "nested_frame")+ return+ frame["path"] = path+ kind = payload.get("kind")+ selector = payload.get("selector")+ if kind == "secret_focus":+ field = payload.get("field")+ await self.stream.send(+ {+ "type": "demonstration_sensitive_field",+ "field": field if field in {"password", "totp", "other"} else "other",+ "selector": selector if isinstance(selector, str) else "",+ "url": redact_raw_secrets_for_prompt(+ safe_admitted_url(str(payload.get("url") or event.page.url or ""))+ ),+ }+ )+ return+ event_kind = "secret" if kind in {"secret", "secret_select"} else kind+ if event_kind not in {"click", "type", "select", "key", "navigate", "secret"}:+ gap_reason = (+ "file_chooser" if kind in {"file_click", "upload"} else f"unsupported_event:{kind or 'unknown'}"+ )+ await self._record_gap_event(event, frame, gap_reason)+ return+ source_url = str(payload.get("url") or "") if event.is_main_frame else str(event.page.url or "")+ if event_kind == "navigate":+ if not event.is_main_frame:+ await self._record_gap_event(event, frame, "frame_navigation")+ return+ source_url = str(payload.get("source_url") or "")+ stored: dict[str, Any] = {+ "ts": payload.get("ts"),+ "kind": event_kind,+ "source_url": source_url,+ "frame": frame,+ }+ for key in ("selector", "packet", "value", "key", "url", "field", "field_label", "label"):+ value = payload.get(key)+ if value is not None and not (event_kind == "secret" and key == "value"):+ stored[key] = value+ if event_kind == "secret":+ if isinstance(selector, str) and selector in self.credential_filled_selectors:+ return+ self.sensitive_count += 1+ if kind == "secret_select":+ stored["control"] = "select"+ stored["label"] = "Select a sensitive option"+ seq = await self._append_event(stored)+ if seq is not None and kind == "secret_select":+ await self.stream.send(+ {"type": "demonstration_action", "seq": seq, "label": stored["label"], "tool_name": "select_option"}+ )+ return+ label = _event_label(stored)+ stored["label"] = label+ seq = await self._append_event(stored)+ if seq is not None:+ await self.stream.send(+ {+ "type": "demonstration_action",+ "seq": seq,+ "label": label,+ "tool_name": {+ "click": "click",+ "type": "type_text",+ "select": "select_option",+ "key": "press_key",+ "navigate": "navigate_browser",+ }[event_kind],+ }+ )++ async def _record_gap_event(self, event: TapEvent, frame: dict[str, Any], reason: str) -> None:+ stored = {+ "ts": event.payload.get("ts"),+ "kind": "gap",+ "reason": reason,+ "source_url": (str(event.payload.get("url") or "") if event.is_main_frame else str(event.page.url or "")),+ "frame": frame,+ "label": "Step not recorded",+ }+ seq = await self._append_event(stored)+ if seq is not None:+ await self.stream.send(+ {"type": "demonstration_action", "seq": seq, "label": stored["label"], "tool_name": "gap"}+ )++ async def _run_controls(self) -> DemonstrationStopReason | None:+ controls = decode_demonstration_controls(+ await self.cache.get(demonstration_controls_key(self.organization_id, self.demonstration_id))+ )+ for entry in controls[self.control_cursor :]:+ self.control_cursor += 1+ action = entry.get("action")+ if action == "cancel":+ return "cancelled"+ if action == "navigate":+ await self._run_control_with_owner_heartbeat(self._navigate(entry))+ elif action == "fill_credential":+ await self._run_control_with_owner_heartbeat(self._fill_credential(entry))+ return None++ async def _run_control_with_owner_heartbeat(self, control: Awaitable[None]) -> None:+ owner_heartbeat = asyncio.create_task(self._owner_heartbeat())+ try:+ await control+ finally:+ owner_heartbeat.cancel()+ with contextlib.suppress(asyncio.CancelledError):+ await owner_heartbeat++ async def _owner_heartbeat(self) -> None:+ while True:+ await self._refresh_owner()+ await asyncio.sleep(DEMONSTRATION_OWNER_REFRESH_SECONDS)++ async def _navigate(self, entry: dict[str, Any]) -> None:+ url = entry.get("url")+ if not isinstance(url, str) or self.page is None:+ return+ try:+ validated_url = await asyncio.to_thread(validate_fetch_url, url)+ except BlockedHost:+ # Same rule as skyvern_navigate for a cloud session: localhost only in local development.+ if not (settings.ENV == "local" and is_localhost_url(url)):+ await self._notice("That URL cannot be opened.", level="error")+ return+ validated_url = url+ except SkyvernHTTPException:+ await self._notice("That URL cannot be opened.", level="error")+ return+ source_url = str(self.page.url or "")+ if self.tap is not None:+ self.tap.begin_control_navigation()+ try:+ await self.page.goto(validated_url)+ except PlaywrightError:+ await self._notice("The page could not be opened.", level="error")+ return+ finally:+ if self.tap is not None:+ self.tap.end_control_navigation()+ event = {+ "ts": datetime.now(UTC).timestamp() * 1000,+ "kind": "navigate",+ "url": url,+ "source_url": source_url,+ "frame": {"is_main": True, "name": ""},+ }+ event["label"] = _event_label(event)+ seq = await self._append_event(event)+ if seq is not None:+ await self.stream.send(+ {+ "type": "demonstration_action",+ "seq": seq,+ "label": event["label"],+ "tool_name": "navigate_browser",+ }+ )++ async def _fill_credential(self, entry: dict[str, Any]) -> None:+ credential_id = entry.get("credential_id")+ selector = entry.get("selector")+ field = entry.get("field")+ if (+ not isinstance(credential_id, str)+ or not isinstance(selector, str)+ or field not in {"password", "totp"}+ or self.page is None+ ):+ return+ credentials = await app.DATABASE.credentials.get_credentials_by_ids(+ [credential_id], organization_id=self.organization_id+ )+ if not credentials:+ await self._notice("The saved credential is no longer available.", level="error")+ return+ credential = credentials[0]+ policy = self.recording_ctx.request_policy+ if not isinstance(policy, RequestPolicy):+ return+ source_url = str(self.page.url or "")+ policy.credential_ask_login_page_urls = [credential.tested_url or source_url]+ apply_connected_credential_to_policy(self.recording_ctx, policy, credential)+ before = len(self.recording_ctx.scout_trajectory)+ self.credential_filled_selectors.add(selector)+ try:+ result = await fill_credential_field(self.recording_ctx, selector, credential_id, field)+ finally:+ if self.tap is not None:+ with contextlib.suppress(Exception):+ await self.tap.flush()+ await self._drain_tap()+ self.credential_filled_selectors.discard(selector)+ scrubbed = scrub_secrets_from_structure(self.recording_ctx, result)+ if not isinstance(scrubbed, dict) or scrubbed.get("ok") is not True:+ await self._notice("The saved credential could not be filled.", level="error")+ return+ scout_fact: ScoutedInteraction | None = None+ if len(self.recording_ctx.scout_trajectory) > before:+ scout_fact = cast(ScoutedInteraction, dict(self.recording_ctx.scout_trajectory[-1]))+ scout_fact["actor"] = "user"+ event: dict[str, Any] = {+ "ts": datetime.now(UTC).timestamp() * 1000,+ "kind": "credential_fill",+ "selector": selector,+ "source_url": source_url,+ "frame": {"is_main": True, "name": ""},+ "credential_id": credential_id,+ "credential_name": credential.name,+ "field": field,+ "result": scrubbed,+ "label": f'Fill {field} from "{safe_label(credential.name)}"'[:60],+ }+ if scout_fact is not None:+ event["scout_fact"] = scout_fact+ seq = await self._append_event(event)+ if seq is not None:+ await self.stream.send(+ {+ "type": "demonstration_action",+ "seq": seq,+ "label": event["label"],+ "tool_name": "fill_credential_field",+ }+ )++ async def _notice(self, message: str, *, level: Literal["info", "error"] = "info") -> None:+ await self.stream.send({"type": "demonstration_notice", "message": message, "level": level})++ async def _refresh_owner(self) -> None:+ record_key = demonstration_record_key(self.organization_id, self.demonstration_id)+ async with self.cache.get_lock(demonstration_lock_key(self.organization_id, self.demonstration_id)):+ record = decode_demonstration_record(await self.cache.get(record_key))+ if record is None or record.owner != self.owner or record.status not in {"recording", "stopping"}:+ return+ record.owner_seen_at = datetime.now(UTC)+ await self.cache.set(record_key, encode_demonstration_record(record), ex=DEMONSTRATION_TTL)++ async def _stop(self, reason: DemonstrationStopReason) -> None:+ await self._run_control_with_owner_heartbeat(self._flush_and_stop(reason))++ async def _refresh_browser_index(self) -> None:+ browser_key = demonstration_browser_key(self.organization_id, self.browser_session_id)+ async with self.cache.get_lock(f"{browser_key}:lock"):+ indexed_id = await self.cache.get(browser_key)+ if isinstance(indexed_id, bytes):+ indexed_id = indexed_id.decode()+ if indexed_id == self.demonstration_id:+ await self.cache.set(browser_key, self.demonstration_id, ex=DEMONSTRATION_TTL)++ async def _flush_and_stop(self, reason: DemonstrationStopReason) -> None:+ tap = self.tap+ if tap is not None:+ with contextlib.suppress(Exception):+ await tap.flush()+ reason = await self._drain_tap() or reason+ with contextlib.suppress(Exception):+ await tap.stop()+ reason = await self._drain_tap() or reason+ with contextlib.suppress(Exception):+ await tap.set_intercept_file_chooser_dialog(False)+ final_url = str(self.page.url or "") if self.page is not None and not self.page.is_closed() else None+ record_key = demonstration_record_key(self.organization_id, self.demonstration_id)+ event_ttl: timedelta | None = None+ async with self.cache.get_lock(demonstration_lock_key(self.organization_id, self.demonstration_id)):+ record = decode_demonstration_record(await self.cache.get(record_key))+ if record is not None and record.owner == self.owner and record.status in {"recording", "stopping"}:+ reason = reason if reason == "cancelled" else record.stop_reason or reason+ record.status = "cancelled" if reason == "cancelled" else "stopped"+ record.stop_reason = reason+ record.owner = None+ record.owner_seen_at = None+ record.final_url = final_url+ event_ttl = DEMONSTRATION_CANCELLED_TTL if record.status == "cancelled" else DEMONSTRATION_TTL+ await self.cache.set(record_key, encode_demonstration_record(record), ex=event_ttl)+ action_count = record.event_count+ else:+ action_count = record.event_count if record is not None else 0+ if event_ttl is not None:+ await expire_demonstration_payload(+ self.cache, self.organization_id, self.demonstration_id, action_count, event_ttl+ )+ if event_ttl == DEMONSTRATION_TTL:+ await self._refresh_browser_index()+ await wait_for_driver_releases(+ [+ start_browser_driver_release(self.organization_id, attached)+ for attached in self.recording_ctx.attached_browser_drivers.values()+ ]+ )+ await self.stream.send(+ {+ "type": "demonstration_stopped",+ "demonstration_id": self.demonstration_id,+ "reason": reason,+ "action_count": action_count,+ }+ )+ LOG.info(+ "copilot_demonstration_stopped",+ reason=reason,+ action_count=action_count,+ sensitive_count=self.sensitive_count,+ duration_s=round(time.monotonic() - self.started_at, 3),+ )+++def safe_label(value: str | None) -> str:+ if not isinstance(value, str):+ return ""+ return " ".join(redact_raw_secrets_for_prompt(value).replace('"', "").split())[:200]+++def _event_label(event: dict[str, Any]) -> str:+ kind = event.get("kind")+ packet = event.get("packet")+ role_name = packet.get("role_name") if isinstance(packet, dict) else None+ accessible_name = role_name.get("accessible_name") if isinstance(role_name, dict) else None+ if kind == "click":+ return f'Click "{safe_label(accessible_name) or "element"}"'[:60]+ if kind == "type":+ value = event.get("value")+ text = redact_raw_secrets_for_prompt(value) if isinstance(value, str) else ""+ return f'Type "{text}"'[:60]+ if kind == "select":+ label = event.get("label") or event.get("value") or "option"+ return f'Select "{safe_label(str(label))}"'[:60]+ if kind == "key":+ return f"Press {event.get('key') or 'key'}"[:60]+ if kind == "navigate":+ url = event.get("url")+ parsed = None+ try:+ parsed = urlsplit(url) if isinstance(url, str) else None+ host = parsed.hostname if parsed is not None else None+ port = f":{parsed.port}" if parsed is not None and parsed.port is not None else ""+ except ValueError:+ host = None+ port = ""+ destination = f"{host}{port}{parsed.path}" if parsed is not None and host is not None else "page"+ destination = redact_raw_secrets_for_prompt(destination)+ return f"Go to {destination}"[:60]+ return "Action"diff --git a/skyvern/forge/sdk/copilot/demonstration_replay.py b/skyvern/forge/sdk/copilot/demonstration_replay.pynew file mode 100644index 000000000..f9cd7e5d2--- /dev/null+++ b/skyvern/forge/sdk/copilot/demonstration_replay.py@@ -0,0 +1,537 @@+from __future__ import annotations++import asyncio+import json+import secrets+import time+from datetime import UTC, datetime+from http import HTTPStatus+from typing import Any, cast++import structlog+from mcp.types import CallToolResult, TextContent++from skyvern.forge import app+from skyvern.forge.sdk.api.llm.api_handler import LLMAPIHandler+from skyvern.forge.sdk.cache.base import BaseCache+from skyvern.forge.sdk.copilot.context import CopilotContext+from skyvern.forge.sdk.copilot.credential_pause import apply_connected_credential_to_policy+from skyvern.forge.sdk.copilot.credential_resolution import safe_admitted_url+from skyvern.forge.sdk.copilot.demonstration_recorder import (+ DEMONSTRATION_CANCELLED_TTL,+ DEMONSTRATION_TTL,+ DemonstrationRecord,+ DemonstrationRejection,+ decode_demonstration_record,+ demonstration_event_key,+ demonstration_lock_key,+ demonstration_record_key,+ encode_demonstration_record,+ expire_demonstration_payload,+ finish_shielded,+ safe_label,+ store_settled_demonstration,+)+from skyvern.forge.sdk.copilot.enforcement import NUDGE_SENTINEL+from skyvern.forge.sdk.copilot.llm_config import resolve_raw_secret_safety_handler+from skyvern.forge.sdk.copilot.mcp_adapter import SkyvernOverlayMCPServer+from skyvern.forge.sdk.copilot.output_utils import sanitize_tool_result_for_llm, summarize_tool_result+from skyvern.forge.sdk.copilot.request_policy import RequestPolicy, screen_raw_secret_safety+from skyvern.forge.sdk.copilot.runtime import DemonstratedAction, ScoutedInteraction, staged_demonstrated_action+from skyvern.forge.sdk.copilot.secret_redaction import redact_raw_secrets_for_prompt+from skyvern.forge.sdk.copilot.secret_scrub import scrub_secrets_from_structure+from skyvern.forge.sdk.copilot.streaming_adapter import parse_tool_output+from skyvern.forge.sdk.copilot.tools.scouting import record_scouted_interaction_fact+from skyvern.forge.sdk.experimentation.copilot_demonstration import shared_copilot_demonstration_cache++LOG = structlog.get_logger()++DEMONSTRATION_DONE_INSTRUCTION = (+ "The browser actions above are the user's demonstration of this task, not your own. A value shown as not "+ "recorded was withheld by the secret screen; a replay gap states when the screen could not check it."+ "{insertion}"+)+_TYPED_VALUE_SCREEN_CONCURRENCY = 4+++async def replay_demonstration(ctx: CopilotContext, mcp_server: SkyvernOverlayMCPServer) -> list[dict[str, Any]]:+ cache = shared_copilot_demonstration_cache()+ demonstration_id = ctx.demonstration_id+ if cache is None or not demonstration_id:+ return []+ record = await _claim_replay(cache, ctx, demonstration_id)+ replay_token = record.replay_token+ if replay_token is None:+ raise DemonstrationRejection(HTTPStatus.CONFLICT, "Demonstration replay lease is unavailable")+ try:+ events = await _load_events(cache, ctx, demonstration_id, record.event_count)+ history = await _replay_events(ctx, mcp_server, events, record)+ except BaseException:+ await finish_shielded(_finish_replay(cache, ctx, demonstration_id, replay_token, consumed=False))+ raise+ await _finish_replay(cache, ctx, demonstration_id, replay_token, consumed=True)+ return history+++async def _replay_events(+ ctx: CopilotContext,+ mcp_server: SkyvernOverlayMCPServer,+ events: list[dict[str, Any]],+ record: DemonstrationRecord,+) -> list[dict[str, Any]]:+ final_url = record.final_url or ""+ raw_secret_safety_handler = await resolve_raw_secret_safety_handler(+ ctx.workflow_permanent_id,+ ctx.organization_id,+ )+ screen_started_at = time.monotonic()+ typed_screens = await _screen_typed_values(+ events,+ raw_secret_safety_handler,+ organization_id=ctx.organization_id,+ )+ screen_latency_ms = (time.monotonic() - screen_started_at) * 1000+ detected_count = sum(screen.status == "detected" for screen in typed_screens.values())+ unavailable_count = sum(screen.status == "blocked" for screen in typed_screens.values())+ withheld_count = sum(+ not _screen_passed(screen, events[index].get("value")) for index, screen in typed_screens.items()+ )+ failure_kinds = sorted({screen.failure_kind for screen in typed_screens.values() if screen.failure_kind != "none"})++ history: list[dict[str, Any]] = []+ if unavailable_count:+ history.append(+ {+ "role": "user",+ "content": NUDGE_SENTINEL+ + f"The secret check was unavailable, so {unavailable_count} typed values were withheld.",+ }+ )+ call_counter = 0+ current_frame_path: tuple[tuple[str, str], ...] | None = None+ sensitive_count = 0+ credential_fill_count = 0++ def next_call_id() -> str:+ nonlocal call_counter+ call_counter += 1+ return f"call_demo_{ctx.turn_id}_{call_counter}"++ async def record_call(+ alias: str,+ real_args: dict[str, Any],+ model_args: dict[str, Any],+ output: CallToolResult | dict[str, Any],+ ) -> None:+ call_id = next_call_id()+ parsed = parse_tool_output(output.content if isinstance(output, CallToolResult) else output)+ if isinstance(output, CallToolResult) and output.isError and parsed.get("ok") is not False:+ parsed = {**parsed, "ok": False}+ safe_output = scrub_secrets_from_structure(ctx, sanitize_tool_result_for_llm(alias, parsed))+ output_text = _call_tool_result_text(output) or json.dumps(safe_output, separators=(",", ":"))+ history.extend(+ [+ {+ "type": "function_call",+ "call_id": call_id,+ "name": alias,+ "arguments": json.dumps(model_args, separators=(",", ":")),+ },+ {"type": "function_call_output", "call_id": call_id, "output": output_text},+ ]+ )+ ctx.tool_calls_this_turn += 1+ ctx.tool_activity.append({"tool": alias, "summary": summarize_tool_result(alias, parsed)})++ async def replay_recorded_call(+ alias: str,+ real_args: dict[str, Any],+ model_args: dict[str, Any],+ action: DemonstratedAction,+ ) -> None:+ with staged_demonstrated_action(action):+ output = await mcp_server.call_tool(alias, real_args)+ await record_call(alias, real_args, model_args, output)++ async def _replay_frame_switch(alias: str, args: dict[str, Any], source_url: str) -> None:+ action = DemonstratedAction(+ selector=str(args.get("selector") or ""),+ source_url=source_url,+ packet=None,+ synthesized_result={+ "ok": True,+ "data": args,+ "browser_context": {"url": source_url, "title": ""},+ },+ )+ await replay_recorded_call(alias, args, args, action)++ for index, event in enumerate(events):+ kind = event.get("kind")+ if kind == "gap":+ reason = event.get("reason")+ if not isinstance(reason, str) or not reason:+ reason = "unknown"+ history.append(_gap_fact(event, reason))+ continue+ frame = event.get("frame")+ if isinstance(frame, dict):+ frame_path = _frame_path(frame)+ if frame_path is None:+ history.append(_gap_fact(event, "nested_frame"))+ continue+ if frame_path != current_frame_path:+ source_url = _safe_model_visible_url(_event_source_url(event))+ if current_frame_path is not None:+ await _replay_frame_switch("skyvern_frame_main", {}, source_url)+ for name, selector in frame_path:+ args = {"name": name} if name else {"selector": selector}+ await _replay_frame_switch("skyvern_frame_switch", args, source_url)+ current_frame_path = frame_path++ if kind == "secret":+ sensitive_count += 1+ selector = str(event.get("selector") or "")+ field = event.get("field")+ is_select = event.get("control") == "select"+ args = {"selector": selector, "value" if is_select else "text": "<value not recorded>"}+ data: dict[str, Any] = {"value_recorded": False}+ # Only a field the page marked by type or autocomplete is named; anything else may be over-withheld.+ if field in {"password", "totp"}:+ data["sensitive_field"] = field+ field_label = safe_label(event.get("field_label"))+ if field_label:+ data["field_label"] = field_label+ output = {"ok": True, "data": data}+ await record_call("select_option" if is_select else "type_text", args, args, output)+ continue+ if kind == "credential_fill":+ credential_fill_count += 1+ credential_id = event.get("credential_id")+ credential_selector = event.get("selector")+ field = event.get("field")+ source_url = _safe_model_visible_url(_event_source_url(event))+ credentials = []+ if isinstance(credential_id, str):+ credentials = await app.DATABASE.credentials.get_credentials_by_ids(+ [credential_id], organization_id=ctx.organization_id+ )+ policy = ctx.request_policy+ if credentials and isinstance(policy, RequestPolicy):+ policy.credential_ask_login_page_urls = [source_url]+ apply_connected_credential_to_policy(ctx, policy, credentials[0])+ scout_fact = event.get("scout_fact")+ if credentials and isinstance(scout_fact, dict) and isinstance(scout_fact.get("tool_name"), str):+ fact = cast(ScoutedInteraction, dict(scout_fact))+ fact["actor"] = "user"+ record_scouted_interaction_fact(ctx, fact)+ args = {+ "selector": credential_selector if isinstance(credential_selector, str) else "",+ "credential_id": credential_id if isinstance(credential_id, str) else "",+ "field": field if field in {"password", "totp"} else "password",+ }+ output = (+ event.get("result") or {"ok": False}+ if credentials+ else {"ok": False, "error": "The demonstrated saved credential no longer exists."}+ )+ await record_call("fill_credential_field", args, args, output)+ continue++ translated = _translate_action(+ ctx,+ event,+ events[index + 1] if index + 1 < len(events) else None,+ final_url,+ typed_screens.get(index),+ )+ if translated is None:+ history.append(_gap_fact(event, f"unsupported_event:{kind or 'unknown'}"))+ continue+ alias, real_args, model_args, action = translated+ await replay_recorded_call(alias, real_args, model_args, action)++ action_count = len(events)+ LOG.info(+ "copilot_demonstration_replayed",+ action_count=action_count,+ sensitive_count=sensitive_count,+ credential_fill_count=credential_fill_count,+ screen_detected_count=detected_count,+ screen_withheld_count=withheld_count,+ screen_unavailable_count=unavailable_count,+ screen_failure_kinds=failure_kinds,+ screen_latency_ms=round(screen_latency_ms, 3),+ )+ ctx.demonstration_items = [+ _demonstration_label(event, typed_screens.get(index)) for index, event in enumerate(events)+ ]+ insertion_data = record.insertion or {}+ after_label = safe_label(insertion_data.get("after_label"))+ before_label = safe_label(insertion_data.get("before_label"))+ bounds = [+ text+ for text in (f"after {after_label}" if after_label else "", f"before {before_label}" if before_label else "")+ if text+ ]+ insertion = f" The requested insertion bounds are {' and '.join(bounds)}." if bounds else ""+ instruction = DEMONSTRATION_DONE_INSTRUCTION.format(insertion=insertion)+ stop_reason = record.stop_reason+ if stop_reason is not None and stop_reason != "stopped":+ history.append(+ {+ "role": "user",+ "content": NUDGE_SENTINEL + f"The demonstration ended because {stop_reason.replace('_', ' ')}.",+ }+ )+ return history + [{"role": "user", "content": NUDGE_SENTINEL + instruction}]+++def _screen_passed(screen: Any, value: Any) -> bool:+ return screen is not None and screen.status == "clean" and screen.canonical_user_message == value+++async def _screen_typed_values(+ events: list[dict[str, Any]],+ raw_secret_safety_handler: LLMAPIHandler | None,+ *,+ organization_id: str,+) -> dict[int, Any]:+ semaphore = asyncio.Semaphore(_TYPED_VALUE_SCREEN_CONCURRENCY)++ async def screen(value: str) -> tuple[str, Any]:+ async with semaphore:+ return value, await screen_raw_secret_safety(+ value,+ raw_secret_safety_handler,+ organization_id=organization_id,+ )++ typed_values = {+ index: value+ for index, event in enumerate(events)+ if event.get("kind") == "type" and isinstance((value := event.get("value")), str)+ }+ results = dict(await asyncio.gather(*(screen(value) for value in set(typed_values.values()))))+ return {index: results[value] for index, value in typed_values.items()}+++def _frame_path(frame: dict[str, Any]) -> tuple[tuple[str, str], ...] | None:+ if frame.get("is_main") is True:+ return ()+ raw_path = frame.get("path")+ if isinstance(raw_path, list):+ path: list[tuple[str, str]] = []+ for entry in raw_path:+ if not isinstance(entry, dict):+ return None+ name = str(entry.get("name") or "")+ selector = str(entry.get("owner_selector") or "")+ if not name and not selector:+ return None+ path.append((name, selector))+ return tuple(path) if len(path) == 1 else None+ name = str(frame.get("name") or "")+ selector = str(frame.get("owner_selector") or "")+ return ((name, selector),) if name or selector else None+++async def _claim_replay(cache: BaseCache, ctx: CopilotContext, demonstration_id: str) -> DemonstrationRecord:+ record_key = demonstration_record_key(ctx.organization_id, demonstration_id)+ async with cache.get_lock(demonstration_lock_key(ctx.organization_id, demonstration_id)):+ record = decode_demonstration_record(await cache.get(record_key))+ if record is None or record.workflow_permanent_id != ctx.workflow_permanent_id:+ LOG.info("copilot_demonstration_replay_skipped", reason="unavailable")+ raise DemonstrationRejection(HTTPStatus.NOT_FOUND, "Unknown demonstration")+ await store_settled_demonstration(cache, ctx.organization_id, demonstration_id, record)+ if record.status in {"recording", "stopping"}:+ raise DemonstrationRejection(HTTPStatus.CONFLICT, "Demonstration is still recording")+ if record.status != "stopped":+ raise DemonstrationRejection(HTTPStatus.CONFLICT, "Demonstration has already ended")+ record.status = "replaying"+ record.replay_started_at = datetime.now(UTC)+ record.replay_token = secrets.token_urlsafe(24)+ await cache.set(record_key, encode_demonstration_record(record), ex=DEMONSTRATION_TTL)+ return record+++async def _load_events(+ cache: BaseCache, ctx: CopilotContext, demonstration_id: str, event_count: int+) -> list[dict[str, Any]]:+ raw_events = await asyncio.gather(+ *(+ cache.get(demonstration_event_key(ctx.organization_id, demonstration_id, seq))+ for seq in range(1, event_count + 1)+ )+ )+ events: list[dict[str, Any]] = []+ for seq, raw in enumerate(raw_events, 1):+ try:+ event = json.loads(raw)+ except (TypeError, ValueError):+ events.append({"seq": seq, "kind": "gap", "reason": "recorded_event_unavailable"})+ continue+ if isinstance(event, dict):+ events.append(event)+ else:+ events.append({"seq": seq, "kind": "gap", "reason": "recorded_event_unavailable"})+ return events+++async def _finish_replay(+ cache: BaseCache,+ ctx: CopilotContext,+ demonstration_id: str,+ replay_token: str,+ *,+ consumed: bool,+) -> None:+ record_key = demonstration_record_key(ctx.organization_id, demonstration_id)+ async with cache.get_lock(demonstration_lock_key(ctx.organization_id, demonstration_id)):+ record = decode_demonstration_record(await cache.get(record_key))+ if record is None or record.status != "replaying" or record.replay_token != replay_token:+ return+ record.status = "consumed" if consumed else "stopped"+ record.replay_started_at = None+ record.replay_token = None+ await cache.set(+ record_key,+ encode_demonstration_record(record),+ ex=DEMONSTRATION_CANCELLED_TTL if consumed else DEMONSTRATION_TTL,+ )+ event_count = record.event_count+ if consumed:+ await expire_demonstration_payload(+ cache, ctx.organization_id, demonstration_id, event_count, DEMONSTRATION_CANCELLED_TTL+ )+++def _translate_action(+ ctx: CopilotContext,+ event: dict[str, Any],+ next_event: dict[str, Any] | None,+ final_url: str,+ typed_screen: Any,+) -> tuple[str, dict[str, Any], dict[str, Any], DemonstratedAction] | None:+ kind = event.get("kind")+ selector = str(event.get("selector") or "")+ packet = event.get("packet")+ packet = packet if isinstance(packet, dict) else None+ source_url = _safe_model_visible_url(_event_source_url(event))+ result_url = _safe_model_visible_url(_event_source_url(next_event)) if next_event is not None else ""+ if not result_url:+ result_url = _safe_model_visible_url(final_url)+ readback: str | None = None+ if kind == "click" and selector:+ alias = "click"+ real_args = model_args = {"selector": selector}+ data: dict[str, Any] = {"selector": selector}+ elif kind == "type" and selector:+ value = event.get("value")+ if not isinstance(value, str):+ return None+ visible_value = value if _screen_passed(typed_screen, value) else "<value not recorded>"+ alias = "type_text"+ real_args = model_args = {"selector": selector, "text": visible_value}+ data = {"selector": selector, "text_length": len(value)}+ readback = visible_value if visible_value == value else None+ elif kind == "select" and selector:+ value = event.get("value")+ if not isinstance(value, str):+ return None+ alias = "select_option"+ real_args = {"selector": selector, "value": value}+ model_args = {"selector": selector, "value": redact_raw_secrets_for_prompt(value)}+ data = dict(model_args)+ elif kind == "key":+ key = event.get("key")+ if not isinstance(key, str) or not key:+ return None+ alias = "press_key"+ real_args = {"key": key}+ if selector:+ real_args["selector"] = selector+ model_args = dict(real_args)+ data = {"key": key, "selector": selector}+ elif kind == "navigate":+ url = event.get("url")+ if not isinstance(url, str) or not url:+ return None+ alias = "navigate_browser"+ real_args = {"url": url}+ visible_url = _safe_model_visible_url(url)+ model_args = {"url": visible_url}+ data = {"url": visible_url}+ result_url = visible_url+ if isinstance(ctx.request_policy, RequestPolicy):+ ctx.request_policy.project_demonstration_navigation(url)+ else:+ return None+ action = DemonstratedAction(+ selector=selector,+ source_url=source_url,+ packet=packet,+ synthesized_result={+ "ok": True,+ "data": data,+ "browser_context": {"url": result_url, "title": "", "url_changed": source_url != result_url},+ },+ readback=readback,+ )+ return alias, real_args, model_args, action+++def _event_source_url(event: dict[str, Any] | None) -> str:+ if event is None:+ return ""+ source_url = event.get("source_url")+ return source_url if isinstance(source_url, str) else ""+++def _model_visible_url(url: str) -> str:+ return safe_admitted_url(url) or url+++def _safe_model_visible_url(url: str) -> str:+ return redact_raw_secrets_for_prompt(_model_visible_url(url))+++def _gap_fact(event: dict[str, Any], reason: str) -> dict[str, str]:+ seq = event.get("seq")+ step = f"Demonstration step {seq}" if isinstance(seq, int) else "A demonstration step"+ return {+ "role": "user",+ "content": NUDGE_SENTINEL + f"{step} was not recorded because {reason.replace('_', ' ')}.",+ }+++def _demonstration_label(event: dict[str, Any], typed_screen: Any = None) -> str:+ kind = event.get("kind")+ if kind == "select" and isinstance(event.get("value"), str):+ return f'Select "{safe_label(event["value"])}"'[:60]+ if kind == "secret" and event.get("control") == "select":+ return "Select a sensitive option"+ if kind in {"type", "secret"}:+ value = event.get("value")+ if kind == "type" and isinstance(value, str) and _screen_passed(typed_screen, value):+ return f'Type "{safe_label(value)}"'[:60]+ packet = event.get("packet")+ role_name = packet.get("role_name") if isinstance(packet, dict) else None+ accessible_name = role_name.get("accessible_name") if isinstance(role_name, dict) else None+ target = safe_label(accessible_name) or safe_label(str(event.get("selector") or "field"))+ return f'Type into "{target}"'[:60]+ label = event.get("label")+ if isinstance(label, str) and label:+ return safe_label(label)+ if kind == "gap":+ return "Step not recorded"+ return "Action"+++def _call_tool_result_text(output: CallToolResult | dict[str, Any]) -> str | None:+ if not isinstance(output, CallToolResult):+ return None+ for item in output.content:+ if isinstance(item, TextContent):+ return item.text+ return Nonediff --git a/skyvern/forge/sdk/copilot/demonstration_tap.js b/skyvern/forge/sdk/copilot/demonstration_tap.jsnew file mode 100644index 000000000..33ddc8812--- /dev/null+++ b/skyvern/forge/sdk/copilot/demonstration_tap.js@@ -0,0 +1,1196 @@+(() => {+ if (+ globalThis.__skyvernDemoTap?.document === document &&+ !globalThis.__skyvernDemoTap.stopped+ )+ return;++ // Inputs that were type=password when the tap saw them; a show-password toggle keeps them secret.+ const knownPasswords = new WeakSet();+ const cssEscape =+ globalThis.CSS && globalThis.CSS.escape+ ? globalThis.CSS.escape.bind(globalThis.CSS)+ : (value) => String(value).replace(/[^a-zA-Z0-9_-]/g, (c) => `\\${c}`);+ const isUniqueSelector = (selector, root = document) => {+ try {+ return root.querySelectorAll(selector).length === 1;+ } catch (err) {+ return false;+ }+ };+ const looksGeneratedId = (id) =>+ /^\d|[_-]\d{3,}|^[0-9a-fA-F-]{16,}$|^:|:$/.test(id);+ const TEST_ATTRIBUTES = [+ "data-testid",+ "data-test-id",+ "data-test",+ "data-qa",+ "data-cy",+ ];+ const selectorWithinRoot = (element, root) => {+ if (!element || !element.tagName || !element.getAttribute) {+ return null;+ }+ const id = element.id;+ if (id && !looksGeneratedId(id)) {+ const selector = `#${cssEscape(id)}`;+ if (isUniqueSelector(selector, root)) return selector;+ }+ for (const attribute of TEST_ATTRIBUTES) {+ const value = element.getAttribute(attribute);+ if (value) {+ const selector = `[${attribute}="${cssEscape(value)}"]`;+ if (isUniqueSelector(selector, root)) return selector;+ }+ }+ const tag = element.tagName.toLowerCase();+ if (["input", "select", "textarea", "button"].includes(tag)) {+ const name = element.getAttribute("name");+ if (name) {+ const selector = `${tag}[name="${cssEscape(name)}"]`;+ if (isUniqueSelector(selector, root)) return selector;+ }+ }+ if (tag === "html" || tag === "body") return tag;+ const segments = [];+ let current = element;+ let anchor = "";+ let depth = 0;+ while (current && current !== root && current.tagName && depth < 8) {+ const currentTag = current.tagName.toLowerCase();+ if (currentTag === "body" || currentTag === "html") break;+ const currentId = current.id;+ if (+ depth > 0 &&+ currentId &&+ !looksGeneratedId(currentId) &&+ isUniqueSelector(`#${cssEscape(currentId)}`, root)+ ) {+ anchor = `#${cssEscape(currentId)}`;+ break;+ }+ let nth = 1;+ let sibling = current.previousElementSibling;+ while (sibling) {+ if (sibling.tagName === current.tagName) nth += 1;+ sibling = sibling.previousElementSibling;+ }+ segments.unshift(`${currentTag}:nth-of-type(${nth})`);+ current = current.parentElement;+ depth += 1;+ }+ let selector = `${anchor ? `${anchor} > ` : ""}${segments.join(" > ")}`;+ if (selector && isUniqueSelector(selector, root)) return selector;+ while (current && current !== root && current.tagName) {+ const currentTag = current.tagName.toLowerCase();+ if (currentTag === "html") break;+ if (currentTag === "body") {+ segments.unshift("body");+ break;+ }+ let nth = 1;+ let sibling = current.previousElementSibling;+ while (sibling) {+ if (sibling.tagName === current.tagName) nth += 1;+ sibling = sibling.previousElementSibling;+ }+ segments.unshift(`${currentTag}:nth-of-type(${nth})`);+ current = current.parentElement;+ }+ selector = segments.join(" > ");+ return selector && isUniqueSelector(selector, root) ? selector : null;+ };+ const getDurableSelector = (element) => {+ if (!element || !element.getRootNode) return null;+ const root = element.getRootNode();+ const selector = selectorWithinRoot(element, root);+ if (!selector) return null;+ if (root && root.nodeType === 11 && root.host) {+ const hostSelector = getDurableSelector(root.host);+ return hostSelector ? `${hostSelector} ${selector}` : null;+ }+ return selector;+ };++ const SECRET_AUTOCOMPLETE_TOKENS = new Set([+ "current-password",+ "new-password",+ "one-time-code",+ "cc-name",+ "cc-number",+ "cc-csc",+ "cc-exp",+ "cc-exp-month",+ "cc-exp-year",+ ]);+ const CREDIT_CARD_HINT_PHRASES = [+ "card number",+ "card no",+ "credit card",+ "credit or debit card",+ "debit card",+ "security code",+ "cardholder",+ "cvv",+ "cvv2",+ "cvc",+ ];+ const TOTP_HINT_PHRASES = [+ "otp",+ "totp",+ "2fa",+ "two factor",+ "one time code",+ "verification code",+ "authenticator code",+ "pin",+ "recovery code",+ "login code",+ "access code",+ ];+ const PASSWORD_HINT_PHRASES = ["password", "passcode", "passphrase"];+ const SECRET_HINT_PHRASES = [+ "api key",+ "apikey",+ "api token",+ "access token",+ "refresh token",+ "auth token",+ "token",+ "access key",+ "secret key",+ "client secret",+ "webhook secret",+ "private key",+ "bearer token",+ "secret value",+ "secret",+ "routing number",+ "account number",+ "bank account",+ "acct",+ "social security number",+ "social security",+ "ssn",+ "passport",+ "tax id",+ "security answer",+ "seed phrase",+ "recovery phrase",+ "iban",+ ];++ const normalizeHaystack = (...parts) => {+ const joined = parts+ .filter(Boolean)+ .join(" ")+ .replace(/([a-z0-9])([A-Z])/g, "$1 $2")+ .replace(/([A-Z]+)([A-Z][a-z])/g, "$1 $2")+ .toLowerCase();+ const normalized = joined+ .replace(/#/g, " number ")+ .replace(/[^a-z0-9]+/g, " ")+ .trim();+ return normalized ? ` ${normalized} ` : "";+ };++ const haystackHasPhrase = (haystack, phrase) => {+ if (!haystack) {+ return false;+ }+ const needle = ` ${String(phrase)+ .toLowerCase()+ .replace(/[^a-z0-9]+/g, " ")+ .trim()} `;+ return haystack.includes(needle);+ };++ const HIDDEN_INPUT_SELECTOR = 'input[type="hidden" i]';+ const FIELD_SELECTOR = 'input:not([type="hidden" i]),select,textarea';+ const LABEL_LIKE_SELECTOR =+ "label,span,p,div,strong,b,td,th,dt,legend,h1,h2,h3,h4,h5,h6";+ const ICON_SELECTOR =+ '[aria-hidden="true"],[class~="icon"],[class*="material-icons"],[class*="material-symbols"]';+ const MAX_LABEL_TEXT_LENGTH = 40;++ const isFieldGroup = (node) => {+ const children = Array.from(node.children).filter(+ (child) => !child.matches(HIDDEN_INPUT_SELECTOR),+ );+ return (+ children.length > 1 &&+ children.every((child) => child.matches(FIELD_SELECTOR))+ );+ };+ const holdsOneField = (node) =>+ node.querySelectorAll(FIELD_SELECTOR).length === 1;++ // Nearest short label-like text before the field, skipping buttons and icons; another field's row ends the walk.+ // A wrapper holding only fields (a card number split into boxes) takes the label above it.+ const HEADING_LIKE_SELECTOR = "h1,h2,h3,h4,h5,h6,legend,dt";+ const LABEL_BOUNDARY_SELECTOR = "form,section,fieldset,dialog";+ const MAX_HEADING_TEXT_LENGTH = 80;++ const hasText = (node) =>+ Boolean(node && String(node.textContent || "").trim());+ const labelTexts = (element) =>+ element.labels+ ? Array.from(element.labels, (label) => label.textContent)+ : [];+ const labelledByNodes = (element) => {+ const root = element.getRootNode && element.getRootNode();+ return String(+ (element.getAttribute && element.getAttribute("aria-labelledby")) || "",+ )+ .split(/\s+/)+ .filter(Boolean)+ .map(+ (id) =>+ (root && root.getElementById && root.getElementById(id)) ||+ (element.ownerDocument && element.ownerDocument.getElementById(id)),+ )+ .filter(Boolean);+ };+ const hasExplicitLabel = (element) =>+ Boolean(+ (element.labels && Array.from(element.labels).some(hasText)) ||+ (element.getAttribute &&+ String(element.getAttribute("aria-label") || "").trim()) ||+ labelledByNodes(element).some(hasText),+ );+ // The field's own label, sent to the model with a withheld value so the model can judge what the field was.+ const ownLabelText = (element) =>+ [+ ...labelTexts(element),+ element.getAttribute && element.getAttribute("aria-label"),+ ...labelledByNodes(element).map((node) => node.textContent),+ element.getAttribute && element.getAttribute("placeholder"),+ element.getAttribute && element.getAttribute("title"),+ ]+ .map((text) =>+ String(text || "")+ .replace(/\s+/g, " ")+ .trim(),+ )+ .find(Boolean)+ ?.slice(0, MAX_HEADING_TEXT_LENGTH) || "";++ const precedingLabelText = (element) => {+ // An email address is never a password, code or card number, so text near it decides nothing.+ if (String(element.type || "").toLowerCase() === "email") return "";+ const parentGroup =+ element.parentElement && isFieldGroup(element.parentElement);+ // A box in a split code or card group keeps the group's label even when it carries "Digit 2".+ if (hasExplicitLabel(element) && !parentGroup) return "";+ let current = element;+ for (let depth = 0; current && depth < 3; depth += 1) {+ const singleField = current === element || holdsOneField(current);+ if (+ current !== element &&+ current.matches(LABEL_BOUNDARY_SELECTOR) &&+ !singleField+ ) {+ return "";+ }+ const parent = current.parentElement;+ // A heading labels only a field, field group or single-field container below it; any other heading starts+ // a section the field is not part of.+ const headingAllowed =+ singleField || isFieldGroup(current) || current.matches("dd");+ if (!parent || !isFieldGroup(parent)) {+ for (+ let sibling = current.previousElementSibling;+ sibling;+ sibling = sibling.previousElementSibling+ ) {+ if (+ sibling.matches(FIELD_SELECTOR) ||+ sibling.querySelector(FIELD_SELECTOR)+ ) {+ return "";+ }+ const text = String(sibling.textContent || "").trim();+ if (sibling.matches(HEADING_LIKE_SELECTOR)) {+ return headingAllowed && text.length <= MAX_HEADING_TEXT_LENGTH+ ? text+ : "";+ }+ if (+ text.length <= MAX_LABEL_TEXT_LENGTH &&+ /[a-z]/i.test(text) &&+ sibling.matches(LABEL_LIKE_SELECTOR) &&+ !sibling.matches(ICON_SELECTOR)+ ) {+ return text;+ }+ }+ if (+ !parent ||+ Array.from(parent.querySelectorAll(FIELD_SELECTOR)).some(+ (field) => !current.contains(field),+ )+ ) {+ return "";+ }+ }+ current = parent;+ }+ return "";+ };++ const secretFieldHaystack = (element) =>+ normalizeHaystack(+ element.id,+ element.name,+ element.getAttribute && element.getAttribute("name"),+ element.getAttribute && element.getAttribute("aria-label"),+ element.getAttribute && element.getAttribute("placeholder"),+ element.getAttribute && element.getAttribute("title"),+ ...labelTexts(element),+ ...labelledByNodes(element).map((node) => node.textContent),+ precedingLabelText(element),+ );++ const inputTypeOf = (element) => String(element.type || "").toLowerCase();+ const autocompleteTokens = (element) =>+ String((element.getAttribute && element.getAttribute("autocomplete")) || "")+ .toLowerCase()+ .split(/\s+/);+ // A show-password toggle reveals into a text-like type; a remembered password moved to one of these is a+ // different field now.+ const NEVER_SECRET_INPUT_TYPES = new Set(["email", "search", "url"]);+ const isPasswordField = (element) => {+ if (!element) return false;+ if (inputTypeOf(element) === "password") return true;+ if (+ knownPasswords.has(element) &&+ !NEVER_SECRET_INPUT_TYPES.has(inputTypeOf(element))+ ) {+ return true;+ }+ return (+ autocompleteTokens(element).some((token) =>+ ["current-password", "new-password"].includes(token),+ ) ||+ PASSWORD_HINT_PHRASES.some((phrase) =>+ haystackHasPhrase(secretFieldHaystack(element), phrase),+ )+ );+ };++ const isHidden = (element) =>+ !element.getClientRects || element.getClientRects().length === 0;++ // A show-password toggle that reveals a separate text input: its value matches a password in the form, or it+ // sits alone beside a password input the person already filled and that is now hidden.+ const sharesCopiedPassword = (element) => {+ if (+ !element ||+ String(element.tagName || "").toLowerCase() !== "input" ||+ inputTypeOf(element) !== "text"+ ) {+ return false;+ }+ const value = String(element.value || "");+ const form = element.form || element.ownerDocument;+ const copiedFromPassword =+ value &&+ Array.from(form.querySelectorAll("input")).some(+ (candidate) =>+ candidate !== element &&+ isPasswordField(candidate) &&+ String(candidate.value || "") === value,+ );+ if (copiedFromPassword) return true;+ const parentInputs = element.parentElement+ ? Array.from(element.parentElement.querySelectorAll("input"))+ : [];+ return (+ parentInputs.length === 2 &&+ parentInputs.some(+ (candidate) =>+ candidate !== element &&+ isPasswordField(candidate) &&+ isHidden(candidate) &&+ secretElements.has(candidate) &&+ Boolean(candidate.value),+ )+ );+ };++ const NON_PASSWORD_SECRET_PHRASES = [+ ...TOTP_HINT_PHRASES,+ ...CREDIT_CARD_HINT_PHRASES,+ ...SECRET_HINT_PHRASES,+ ];++ // Whether to keep the value: leans toward withholding, so text near the field counts.+ const isSecretField = (element) => {+ if (!element) return false;+ if (isPasswordField(element) || sharesCopiedPassword(element)) return true;+ if (+ autocompleteTokens(element).some((token) =>+ SECRET_AUTOCOMPLETE_TOKENS.has(token),+ )+ ) {+ return true;+ }+ const haystack = secretFieldHaystack(element);+ return [...NON_PASSWORD_SECRET_PHRASES, ...PASSWORD_HINT_PHRASES].some(+ (phrase) => haystackHasPhrase(haystack, phrase),+ );+ };++ // What kind of field it is is a fact for the model and the credential card, so only the type and autocomplete+ // name one. Page text may veto "password" for a masked code box but never names a kind.+ const secretFieldKind = (element) => {+ if (!isSecretField(element)) return null;+ const tokens = autocompleteTokens(element);+ if (tokens.includes("one-time-code")) return "totp";+ if (tokens.some((token) => token.startsWith("cc-"))) return "other";+ if (inputTypeOf(element) !== "password") return "other";+ const haystack = secretFieldHaystack(element);+ return NON_PASSWORD_SECRET_PHRASES.some((phrase) =>+ haystackHasPhrase(haystack, phrase),+ )+ ? "other"+ : "password";+ };++ const ACTIONABLE =+ "a,button,input,select,textarea,label,summary,option,[role=button],[role=link]," ++ "[role=menuitem],[role=option],[role=tab],[role=checkbox],[role=radio],[role=switch],[onclick]";+ const TEXT_INPUT_TYPES = new Set([+ "",+ "text",+ "search",+ "email",+ "tel",+ "url",+ "password",+ "number",+ "date",+ "datetime-local",+ "month",+ "time",+ "week",+ ]);+ const UNSUPPORTED_VALUE_INPUT_TYPES = new Set(["range", "color"]);+ const CLICK_DRAG_THRESHOLD = 5;+ const KEY_NAMES = new Set([+ "Enter",+ "Tab",+ "Escape",+ "ArrowUp",+ "ArrowDown",+ "ArrowLeft",+ "ArrowRight",+ "PageUp",+ "PageDown",+ "Home",+ "End",+ ]);+ const listeners = [];+ const secretElements = new WeakSet();+ const recordedSecretElements = new Set();+ const secretFieldSelectors = new Set();+ const secretFieldPositions = new Set();+ const announcedSecretFocus = new WeakSet();+ const queue = [];+ const stream = `${Date.now()}-${Math.random()}`;+ let seq = 0;+ let waiter = null;+ let holdTimer = null;+ let holding = false;+ let clickStart = null;+ let clickPointer = null;+ let pointerUp = false;+ let typing = null;+ let selectStart = null;+ let stopped = false;+ let installedDocument = document;+ let wakeQueued = false;++ const eventElement = (event) => {+ const first = event.composedPath && event.composedPath()[0];+ return first && first.nodeType === 1 ? first : event.target;+ };+ const clickElement = (element) =>+ element && element.closest+ ? element.closest(ACTIONABLE) || element+ : element;+ const fieldValue = (element) =>+ element.isContentEditable+ ? String(element.textContent || "")+ : String(element.value || "");+ const textField = (element) => {+ if (!element || element.nodeType !== 1) return null;+ if (element.isContentEditable)+ return element.closest("[contenteditable]") || element;+ const tag = (element.tagName || "").toLowerCase();+ if (tag === "textarea") return element;+ if (tag !== "input") return null;+ if (element.readOnly || element.disabled) return null;+ return TEXT_INPUT_TYPES.has(String(element.type || "").toLowerCase())+ ? element+ : null;+ };+ const unsupportedValueField = (element) => {+ if (!element || !element.tagName) return null;+ return element.tagName.toLowerCase() === "input" &&+ UNSUPPORTED_VALUE_INPUT_TYPES.has(+ String(element.type || "").toLowerCase(),+ )+ ? element+ : null;+ };+ const structuralPosition = (element) => {+ if (!element || !element.tagName) return "";+ const segments = [];+ let current = element;+ const root = element.getRootNode && element.getRootNode();+ while (current && current.tagName && current !== root) {+ let index = 1;+ let sibling = current.previousElementSibling;+ while (sibling) {+ if (sibling.tagName === current.tagName) index += 1;+ sibling = sibling.previousElementSibling;+ }+ segments.unshift(+ `${current.tagName.toLowerCase()}:nth-of-type(${index})`,+ );+ current = current.parentElement;+ }+ return segments.join(" > ");+ };+ const nativeSelect = (element) => {+ if (!element || !element.tagName) return null;+ const tag = element.tagName.toLowerCase();+ if (tag === "select") return element;+ return tag === "option" && element.closest+ ? element.closest("select")+ : null;+ };+ const labelTextField = (element) =>+ element && element.matches && element.matches("label")+ ? textField(element.control)+ : null;+ const packetFor = (element, selector) => {+ const packet = scoutPreActionPacket(element, selector, "", "", 5000);+ if (textField(element)) {+ packet.role_name = null;+ packet.role_name_match_count = -1;+ }+ if (element.getRootNode && element.getRootNode().nodeType === 11) {+ packet.selector_match_count = 1;+ }+ return packet;+ };+ const snapshot = (element) => {+ const selector = getDurableSelector(element);+ return selector+ ? { element, selector, packet: packetFor(element, selector) }+ : null;+ };+ const finishWaiter = (events) => {+ if (!waiter) return;+ const current = waiter;+ waiter = null;+ current(events);+ };+ const drained = () => ({ stream, events: queue.slice() });+ const empty = () => ({ stream, events: [] });+ const wakeNow = () => {+ if (waiter && !holding && queue.length) finishWaiter(drained());+ };+ const wake = () => {+ if (wakeQueued) return;+ wakeQueued = true;+ queueMicrotask(() => {+ wakeQueued = false;+ wakeNow();+ });+ };+ const beginHold = () => {+ holding = true;+ clearTimeout(holdTimer);+ holdTimer = setTimeout(() => {+ holding = false;+ holdTimer = null;+ if (clickStart && pointerUp) {+ const start = clickStart;+ clickStart = null;+ clickPointer = null;+ pointerUp = false;+ emit("click", start);+ } else {+ wake();+ }+ }, 600);+ };+ const endHold = () => {+ holding = false;+ clearTimeout(holdTimer);+ holdTimer = null;+ };+ const cancelClick = () => {+ clickStart = null;+ clickPointer = null;+ pointerUp = false;+ endHold();+ wake();+ };+ const emit = (kind, start, extra = {}) => {+ if (!start) return;+ if (queue.length >= 1000) queue.shift();+ const payload = {+ kind,+ ts: Date.now(),+ url: location.href,+ seq: ++seq,+ ...extra,+ };+ if (start.selector) {+ payload.selector = start.selector;+ payload.packet = start.packet;+ }+ if (kind.startsWith("secret") && start.element) {+ recordedSecretElements.add(start.element);+ }+ queue.push(payload);+ wake();+ };+ const add = (target, type, listener) => {+ target.addEventListener(type, listener, true);+ listeners.push([target, type, listener]);+ };+ const startTyping = (element, initialValue = fieldValue(element)) => {+ const start = snapshot(element);+ if (!start) return;+ const field = secretFieldKind(element);+ const position = structuralPosition(element);+ const previouslySecret = secretFieldSelectors.has(start.selector);+ const secretByPosition = secretFieldPositions.has(position);+ if (field || previouslySecret || secretByPosition) {+ secretElements.add(element);+ secretFieldSelectors.add(start.selector);+ if (position) secretFieldPositions.add(position);+ }+ typing = {+ ...start,+ initialValue,+ secret: Boolean(field || previouslySecret || secretByPosition),+ field: field || "other",+ };+ if (+ (field || previouslySecret || secretByPosition) &&+ !announcedSecretFocus.has(element)+ ) {+ announcedSecretFocus.add(element);+ emit("secret_focus", typing, { field: field || "other" });+ }+ };+ const commitTyping = () => {+ if (!typing) return;+ const span = typing;+ typing = null;+ const value = fieldValue(span.element);+ if (span.initialValue !== null && value === span.initialValue) return;+ const field = secretFieldKind(span.element);+ if (field) {+ secretElements.add(span.element);+ secretFieldSelectors.add(span.selector);+ const position = structuralPosition(span.element);+ if (position) secretFieldPositions.add(position);+ }+ if (span.secret || secretElements.has(span.element) || field) {+ emit("secret", span, {+ typed_length: value.length,+ field: field || span.field,+ field_label: ownLabelText(span.element),+ });+ } else {+ emit("type", span, { value, typed_length: value.length });+ }+ };+ const fileInputFor = (element) => {+ if (!element) return null;+ if (+ element.tagName &&+ element.tagName.toLowerCase() === "input" &&+ String(element.type || "").toLowerCase() === "file"+ ) {+ return element;+ }+ if (!element.matches || !element.matches("label")) return null;+ if (+ element.control &&+ String(element.control.type || "").toLowerCase() === "file"+ )+ return element.control;+ const nested = element.querySelector('input[type="file"]');+ if (nested) return nested;+ const id = element.getAttribute("for");+ const control = id ? element.ownerDocument.getElementById(id) : null;+ return control && String(control.type || "").toLowerCase() === "file"+ ? control+ : null;+ };+ const keyName = (event) => {+ if (["Control", "Alt", "Shift", "Meta"].includes(event.key)) return null;+ if (+ event.key.length === 1 &&+ ((event.ctrlKey && event.altKey) || (!event.ctrlKey && !event.metaKey))+ )+ return null;+ const modifiers = [];+ if (event.ctrlKey) modifiers.push("Control");+ if (event.altKey) modifiers.push("Alt");+ if (event.shiftKey) modifiers.push("Shift");+ if (event.metaKey) modifiers.push("Meta");+ if (modifiers.length) return [...modifiers, event.key].join("+");+ return KEY_NAMES.has(event.key) ? event.key : null;+ };++ const watchedRoots = new WeakSet();+ const fieldsIn = (node) => {+ if (node.nodeType !== 1) return [];+ return node.matches(FIELD_SELECTOR)+ ? [node]+ : Array.from(node.querySelectorAll(FIELD_SELECTOR));+ };+ const rememberPasswords = (node) => {+ for (const field of fieldsIn(node)) {+ if (inputTypeOf(field) === "password") knownPasswords.add(field);+ }+ };+ // A show-password toggle can swap in a new input node (v-if/v-else, a re-rendered wrapper), sometimes after a+ // leave transition. The first lone field added under the container a lone password just left inherits it.+ const vacatedPasswordSlots = new Map();+ const VACATED_SLOT_MS = 1000;+ const passwordTypeObserver = new MutationObserver((mutations) => {+ const now = Date.now();+ const added = new Set();+ for (const mutation of mutations) {+ if (mutation.type === "attributes") {+ if (+ String(mutation.oldValue || "").toLowerCase() === "password" ||+ inputTypeOf(mutation.target) === "password"+ ) {+ knownPasswords.add(mutation.target);+ }+ continue;+ }+ for (const node of mutation.removedNodes) {+ const fields = fieldsIn(node);+ if (fields.length === 1 && isPasswordField(fields[0])) {+ vacatedPasswordSlots.set(mutation.target, now);+ }+ }+ for (const node of mutation.addedNodes) added.add(node);+ }+ for (const [slot, at] of vacatedPasswordSlots) {+ if (now - at > VACATED_SLOT_MS) vacatedPasswordSlots.delete(slot);+ }+ for (const node of added) {+ // A node inserted inside another node from this batch is covered by that node's subtree walk.+ if (added.has(node.parentNode)) continue;+ watchShadowRoots(node);+ rememberPasswords(node);+ const fields = fieldsIn(node);+ if (fields.length !== 1) continue;+ for (const slot of vacatedPasswordSlots.keys()) {+ if (slot.contains(fields[0])) {+ knownPasswords.add(fields[0]);+ vacatedPasswordSlots.delete(slot);+ break;+ }+ }+ }+ });+ // Shadow trees are invisible to a document observer, so each open root gets its own.+ const watchRoot = (root) => {+ if (watchedRoots.has(root)) return;+ watchedRoots.add(root);+ for (const field of root.querySelectorAll('input[type="password"]')) {+ knownPasswords.add(field);+ }+ passwordTypeObserver.observe(root, {+ subtree: true,+ childList: true,+ attributes: true,+ attributeFilter: ["type"],+ attributeOldValue: true,+ });+ watchShadowRoots(root);+ };+ const watchShadowRoots = (node) => {+ if (!node || !node.querySelectorAll) return;+ if (node.shadowRoot) watchRoot(node.shadowRoot);+ for (const element of node.querySelectorAll("*")) {+ if (element.shadowRoot) watchRoot(element.shadowRoot);+ }+ };+ watchRoot(document);+ // A shadow root attached after its host was inserted raises no mutation, and a light-DOM button can toggle a+ // password inside it, so every click or Enter/Space looks for new open roots before the page handles it.+ const watchEventRoots = (event) => {+ for (const node of event.composedPath ? event.composedPath() : []) {+ const root = node.getRootNode && node.getRootNode();+ if (root && root !== document && root.host) watchRoot(root);+ }+ watchShadowRoots(document);+ };+ add(globalThis, "pointerdown", (event) => {+ if (!event.isTrusted) return;+ watchEventRoots(event);+ const raw = eventElement(event);+ const field = textField(raw);+ if (typing && !typing.element.contains(raw)) commitTyping();+ cancelClick();+ if (event.button !== 0) return;+ const select = nativeSelect(raw);+ if (select) {+ selectStart = snapshot(select);+ return;+ }+ if (field || unsupportedValueField(raw)) return;+ const target = clickElement(raw);+ if (labelTextField(target)) return;+ clickStart = snapshot(target);+ if (clickStart) {+ clickPointer = {+ id: event.pointerId,+ button: event.button,+ x: event.clientX,+ y: event.clientY,+ };+ beginHold();+ }+ });+ add(globalThis, "pointermove", (event) => {+ if (+ !event.isTrusted ||+ !clickPointer ||+ pointerUp ||+ event.pointerId !== clickPointer.id+ )+ return;+ if (+ Math.hypot(+ event.clientX - clickPointer.x,+ event.clientY - clickPointer.y,+ ) > CLICK_DRAG_THRESHOLD+ )+ cancelClick();+ });+ add(globalThis, "pointerup", (event) => {+ if (!event.isTrusted) return;+ if (+ !clickPointer ||+ event.pointerId !== clickPointer.id ||+ event.button !== clickPointer.button+ )+ return;+ if (+ Math.hypot(+ event.clientX - clickPointer.x,+ event.clientY - clickPointer.y,+ ) > CLICK_DRAG_THRESHOLD+ ) {+ cancelClick();+ return;+ }+ pointerUp = true;+ });+ add(globalThis, "pointercancel", (event) => {+ if (event.isTrusted && clickPointer && event.pointerId === clickPointer.id)+ cancelClick();+ });+ add(globalThis, "click", (event) => {+ if (!event.isTrusted) return;+ if (event.detail === 0) return;+ const raw = eventElement(event);+ if (nativeSelect(raw)) return;+ if (unsupportedValueField(raw)) return;+ const target = clickElement(raw);+ if (textField(target) || labelTextField(target)) return;+ const start = clickStart || snapshot(target);+ clickStart = null;+ clickPointer = null;+ pointerUp = false;+ endHold();+ emit(fileInputFor(target) ? "file_click" : "click", start);+ });+ add(globalThis, "focusin", (event) => {+ if (!event.isTrusted) return;+ watchEventRoots(event);+ const target = eventElement(event);+ const field = textField(target);+ if (field) {+ startTyping(field);+ return;+ }+ if (target && target.tagName && target.tagName.toLowerCase() === "select") {+ selectStart = snapshot(target);+ }+ });+ add(globalThis, "focusout", (event) => {+ if (!event.isTrusted) return;+ if (typing && typing.element === textField(eventElement(event)))+ commitTyping();+ });+ const isSelect = (element) =>+ Boolean(+ element && element.tagName && element.tagName.toLowerCase() === "select",+ );+ const inShadowRoot = (element) =>+ Boolean(element.getRootNode && element.getRootNode() !== document);+ const emitSelect = (target) => {+ const option = target.selectedOptions && target.selectedOptions[0];+ const start =+ selectStart && selectStart.element === target+ ? selectStart+ : snapshot(target);+ const field = secretFieldKind(target);+ if (field)+ emit("secret_select", start, {+ field,+ field_label: ownLabelText(target),+ });+ else+ emit("select", start, {+ value: String(target.value || ""),+ label: option ? String(option.textContent || "").trim() : "",+ });+ selectStart = null;+ };+ add(globalThis, "change", (event) => {+ if (!event.isTrusted) return;+ const target = eventElement(event);+ if (isSelect(target)) {+ emitSelect(target);+ return;+ }+ const unsupportedValue = unsupportedValueField(target);+ if (unsupportedValue) {+ emit("unsupported_input", snapshot(unsupportedValue), {+ input_type: String(unsupportedValue.type || ""),+ value: fieldValue(unsupportedValue),+ });+ return;+ }+ // Date-like inputs fire change per completed segment; their span ends on focus loss or Enter instead.+ const segmented = /^(date|time|datetime-local|month|week)$/.test(+ String(target && target.type),+ );+ const focused =+ target &&+ target.getRootNode &&+ target.getRootNode().activeElement === target;+ if (+ typing &&+ typing.element === textField(target) &&+ !(segmented && focused)+ )+ commitTyping();+ });+ add(globalThis, "paste", (event) => {+ if (!event.isTrusted || typing) return;+ const field =+ textField(eventElement(event)) || textField(document.activeElement);+ if (field) startTyping(field);+ });+ add(globalThis, "input", (event) => {+ if (!event.isTrusted) return;+ const target = eventElement(event);+ // change is not composed, so a select inside a shadow root reports only through input.+ if (isSelect(target) && inShadowRoot(target)) {+ emitSelect(target);+ return;+ }+ if (typing) return;+ const field =+ textField(eventElement(event)) || textField(document.activeElement);+ if (field) startTyping(field, null);+ });+ add(globalThis, "keydown", (event) => {+ if (!event.isTrusted || event.repeat) return;+ if (event.key === "Enter" || event.key === " ") watchEventRoots(event);+ const raw = eventElement(event);+ const target = clickElement(raw);+ if (+ nativeSelect(raw) ||+ nativeSelect(target) ||+ nativeSelect(document.activeElement)+ )+ return;+ const field = textField(raw) || textField(document.activeElement);+ if (!typing && field && !event.isComposing) startTyping(field);+ const name = keyName(event);+ const namedKey =+ KEY_NAMES.has(event.key) &&+ !event.ctrlKey &&+ !event.altKey &&+ !event.metaKey;+ const inSecretField =+ Boolean(typing && typing.secret) ||+ isSecretField(raw) ||+ isSecretField(document.activeElement);+ const suppressKey = inSecretField && (!namedKey || event.key === "Tab");+ if (typing) {+ if (event.isComposing) return;+ const multiline =+ typing.element.tagName.toLowerCase() === "textarea" ||+ typing.element.isContentEditable;+ if (event.key === "Enter" && multiline) return;+ const commitsAndEmits =+ event.key === "Enter" ||+ event.key === "Tab" ||+ ["ArrowUp", "ArrowDown", "Escape", "PageUp", "PageDown"].includes(+ event.key,+ ) ||+ Boolean(name && (event.ctrlKey || event.altKey || event.metaKey));+ if (event.key === "Tab" || commitsAndEmits) commitTyping();+ if (suppressKey || !commitsAndEmits || !name) return;+ emit("key", snapshot(field || raw), { key: name });+ if (event.key === "Enter") wakeNow();+ return;+ }+ if (suppressKey) return;+ const tag = target && target.tagName && target.tagName.toLowerCase();+ const spaceControl =+ event.key === " " &&+ target &&+ target.matches &&+ target.matches("button,input[type=checkbox],input[type=radio]");+ if (name || spaceControl)+ emit("key", tag === "body" || tag === "html" ? {} : snapshot(target), {+ key: spaceControl ? "Space" : name,+ });+ });+ const flush = () => {+ commitTyping();+ const start = pointerUp ? clickStart : null;+ clickStart = null;+ clickPointer = null;+ pointerUp = false;+ endHold();+ if (start) emit("click", start);+ else wake();+ };+ add(globalThis, "pagehide", flush);++ const rearm = () => {+ const priorDocument = installedDocument;+ for (const listener of listeners) {+ const target = listener[0] === priorDocument ? document : listener[0];+ target.addEventListener(listener[1], listener[2], true);+ listener[0] = target;+ }+ installedDocument = document;+ api.document = document;+ };+ // The tap never reads a secret value, so nothing can scrub it later: empty those fields once the person is done,+ // before Copilot can read the live page.+ const clearSecretElement = (element) => {+ if (element.isContentEditable) {+ if (!element.textContent) return;+ element.textContent = "";+ element.dispatchEvent(new Event("input", { bubbles: true }));+ return;+ }+ if (String(element.tagName || "").toLowerCase() === "select") {+ if (element.selectedIndex === -1) return;+ element.selectedIndex = -1;+ element.dispatchEvent(new Event("change", { bubbles: true }));+ return;+ }+ if (!element.value) return;+ const descriptor = Object.getOwnPropertyDescriptor(+ Object.getPrototypeOf(element),+ "value",+ );+ if (descriptor && descriptor.set) descriptor.set.call(element, "");+ else element.value = "";+ element.dispatchEvent(new Event("input", { bubbles: true }));+ };+ const clearSecretValues = () => {+ const fields = new Set(recordedSecretElements);+ // A show-password toggle can copy the typed password into a text input the person never focused.+ const typedPasswords = new Set(+ Array.from(recordedSecretElements)+ .filter(isPasswordField)+ .map((field) => String(field.value || ""))+ .filter(Boolean),+ );+ if (typedPasswords.size) {+ for (const field of document.querySelectorAll("input,textarea")) {+ if (typedPasswords.has(String(field.value || ""))) fields.add(field);+ }+ }+ for (const field of fields) {+ try {+ clearSecretElement(field);+ } catch (_error) {+ // A detached or page-locked element keeps nothing worth retrying.+ }+ }+ recordedSecretElements.clear();+ };+ const api = {+ document,+ get stopped() {+ return stopped;+ },+ drain: (maxWaitMs, acknowledgedStream, acknowledgedSeq) =>+ new Promise((resolve) => {+ rearm();+ if (acknowledgedStream === stream) {+ while (queue.length && queue[0].seq <= acknowledgedSeq) queue.shift();+ }+ if (stopped) {+ resolve(empty());+ return;+ }+ if (queue.length && !holding) {+ resolve(drained());+ return;+ }+ if (waiter) waiter(empty());+ let settled = false;+ const timer = setTimeout(+ () => {+ if (settled) return;+ settled = true;+ if (waiter === finish) waiter = null;+ resolve(queue.length && !holding ? drained() : empty());+ },+ Math.max(0, Number(maxWaitMs) || 0),+ );+ const finish = (events) => {+ if (settled) return;+ settled = true;+ clearTimeout(timer);+ resolve(events);+ };+ waiter = finish;+ }),+ selectorFor: getDurableSelector,+ flush,+ stop: () => {+ if (stopped) return [];+ flush();+ stopped = true;+ clearSecretValues();+ passwordTypeObserver.disconnect();+ for (const [target, type, listener] of listeners) {+ target.removeEventListener(type, listener, true);+ }+ listeners.length = 0;+ finishWaiter(empty());+ return drained();+ },+ };+ globalThis.__skyvernDemoTap = api;+})();diff --git a/skyvern/forge/sdk/copilot/demonstration_tap.py b/skyvern/forge/sdk/copilot/demonstration_tap.pynew file mode 100644index 000000000..3617c8190--- /dev/null+++ b/skyvern/forge/sdk/copilot/demonstration_tap.py@@ -0,0 +1,622 @@+from __future__ import annotations++import asyncio+import contextlib+import json+import time+from dataclasses import dataclass+from itertools import count+from pathlib import Path+from typing import Any, TypedDict++import structlog+from playwright.async_api import BrowserContext, CDPSession, Frame, Page++from skyvern.forge.sdk.copilot.composition_browser_expressions import scout_pre_action_packet_js++WORLD = "skyvern-demonstration"+TAP_SOURCE = scout_pre_action_packet_js() + Path(__file__).with_name("demonstration_tap.js").read_text()+USER_NAVIGATION_TRANSITIONS = {"typed", "address_bar", "generated", "auto_bookmark", "keyword"}+DEMONSTRATION_MAX_EVENTS = 1000++LOG = structlog.get_logger()+++@dataclass(frozen=True)+class TapEvent:+ page: Page+ session_key: str+ frame_id: str+ is_main_frame: bool+ frame_name: str+ payload: dict[str, Any]+++class TapSessionState(TypedDict):+ session: CDPSession+ page: Page+ target_frame: Frame | None+ is_page_session: bool+ script_id: str | None+ frames: dict[str, dict[str, Any]]+ main_frame_id: str+ seen_navigation_entry_ids: set[str]+ drain_acks: dict[str, tuple[str | None, int]]+ seen_event_sequences: dict[str, dict[str, int]]+ worlds: dict[str, int]+++class PageTap:+ def __init__(self, context: BrowserContext, *, max_events: int = DEMONSTRATION_MAX_EVENTS) -> None:+ self.context = context+ self._events: asyncio.Queue[TapEvent] = asyncio.Queue(maxsize=max_events)+ self._event_limit_reached = False+ self._sessions: dict[str, TapSessionState] = {}+ self._page_sessions: dict[int, str] = {}+ self._oopif_sessions: dict[Frame, str] = {}+ self._refused_frames: set[tuple[Frame, str]] = set()+ self._poll_tasks: dict[tuple[str, str], asyncio.Task[None]] = {}+ self._background_tasks: set[asyncio.Task[Any]] = set()+ self._installing_pages: set[int] = set()+ self._session_ids = count(1)+ self._scan_task: asyncio.Task[None] | None = None+ self._page_listener: Any = None+ self._intercept_file_chooser = False+ self._control_navigation_depth = 0+ self._stopped = False++ @property+ def event_limit_reached(self) -> bool:+ return self._event_limit_reached++ def _enqueue_event(self, event: TapEvent) -> bool:+ if self._event_limit_reached:+ return False+ try:+ self._events.put_nowait(event)+ except asyncio.QueueFull:+ self._event_limit_reached = True+ return False+ return True++ def _spawn(self, coroutine: Any) -> asyncio.Task[Any]:+ task = asyncio.create_task(coroutine)+ self._background_tasks.add(task)+ task.add_done_callback(self._background_tasks.discard)+ return task++ async def start(self, pages: Any) -> None:+ if self._stopped:+ return+ if self._page_listener is None:++ def on_page(page: Page) -> None:+ self._spawn(self._install_page(page))++ self._page_listener = on_page+ self.context.on("page", on_page)+ await asyncio.gather(*(self._install_page(page) for page in pages))+ await self._scan_once()+ if self._scan_task is None or self._scan_task.done():+ self._scan_task = asyncio.create_task(self.poll_new_frames())++ async def _install_page(self, page: Page) -> None:+ page_key = id(page)+ if self._stopped or page_key in self._page_sessions or page_key in self._installing_pages:+ return+ self._installing_pages.add(page_key)+ session: CDPSession | None = None+ key: str | None = None+ try:+ session = await self.context.new_cdp_session(page)+ key = await self._install_session(session, page, target_frame=None, is_page_session=True)+ if key is not None:+ self._page_sessions[page_key] = key+ except asyncio.CancelledError:+ raise+ except Exception: # noqa: BLE001+ return+ finally:+ if session is not None and key is None:+ with contextlib.suppress(Exception):+ await session.detach()+ self._installing_pages.discard(page_key)++ async def _install_oopif(self, page: Page, frame: Frame) -> None:+ refused_key = (frame, frame.url)+ if self._stopped or frame in self._oopif_sessions or refused_key in self._refused_frames:+ return+ self._refused_frames = {key for key in self._refused_frames if key[0] is not frame}+ session: CDPSession | None = None+ key: str | None = None+ try:+ session = await self.context.new_cdp_session(frame)+ key = await self._install_session(session, page, target_frame=frame, is_page_session=False)+ if key is not None:+ self._oopif_sessions[frame] = key+ except asyncio.CancelledError:+ raise+ except Exception: # noqa: BLE001+ self._refused_frames.add(refused_key)+ return+ finally:+ if session is not None and key is None:+ with contextlib.suppress(Exception):+ await session.detach()++ async def _install_session(+ self,+ session: CDPSession,+ page: Page,+ *,+ target_frame: Frame | None,+ is_page_session: bool,+ ) -> str | None:+ try:+ await session.send("Page.enable")+ if self._intercept_file_chooser:+ with contextlib.suppress(Exception):+ await session.send("Page.setInterceptFileChooserDialog", {"enabled": True})+ installed = await session.send(+ "Page.addScriptToEvaluateOnNewDocument",+ {"source": TAP_SOURCE, "worldName": WORLD, "runImmediately": True},+ )+ tree = await session.send("Page.getFrameTree")+ seen_navigation_entry_ids: set[str] = set()+ if is_page_session:+ entries = (await session.send("Page.getNavigationHistory")).get("entries")+ if isinstance(entries, list):+ seen_navigation_entry_ids.update(str(entry["id"]) for entry in entries)+ except Exception: # noqa: BLE001+ return None+ if self._stopped:+ with contextlib.suppress(Exception):+ await session.send(+ "Page.removeScriptToEvaluateOnNewDocument", {"identifier": installed.get("identifier")}+ )+ return None+ key = f"tap-{next(self._session_ids)}"+ self._sessions[key] = {+ "session": session,+ "page": page,+ "target_frame": target_frame,+ "is_page_session": is_page_session,+ "script_id": installed.get("identifier"),+ "frames": {},+ "main_frame_id": str(tree["frameTree"]["frame"]["id"]),+ "seen_navigation_entry_ids": seen_navigation_entry_ids,+ "drain_acks": {},+ "seen_event_sequences": {},+ "worlds": {},+ }+ self._reconcile_frames(key, tree)+ session.on("Page.frameNavigated", lambda event: self._frame_navigated(key, event))+ session.on("Page.frameRequestedNavigation", lambda event: self._frame_requested_navigation(key, event))+ return key++ def _frame_requested_navigation(self, session_key: str, event: dict[str, Any]) -> None:+ state = self._sessions.get(session_key)+ frame_id = str(event.get("frameId") or "")+ if state is not None and frame_id in state["frames"] and event.get("disposition") == "currentTab":+ self._spawn(+ self._drain_frame(+ session_key,+ frame_id,+ max_wait_ms=0,+ context_id=state["worlds"].get(frame_id),+ )+ )++ def _frame_navigated(self, session_key: str, event: dict[str, Any]) -> None:+ state = self._sessions.get(session_key)+ frame = event.get("frame")+ if state is None or not isinstance(frame, dict):+ return+ frame_id = str(frame.get("id") or "")+ previous_url = str(state["frames"].get(frame_id, {}).get("url") or "")+ state["frames"][frame_id] = {**state["frames"].get(frame_id, {}), **frame}+ state["worlds"].pop(frame_id, None)+ if not state["is_page_session"] or frame_id != state["main_frame_id"]:+ return+ self._spawn(+ self._enqueue_navigation_event(+ session_key,+ state,+ frame_id,+ previous_url=previous_url,+ timestamp_ms=time.time() * 1000,+ suppressed=bool(self._control_navigation_depth),+ )+ )++ async def _enqueue_navigation_event(+ self,+ session_key: str,+ state: TapSessionState,+ frame_id: str,+ *,+ previous_url: str,+ timestamp_ms: float,+ suppressed: bool,+ ) -> None:+ try:+ history = await state["session"].send("Page.getNavigationHistory")+ current_index = history.get("currentIndex")+ entries = history.get("entries")+ if not isinstance(current_index, int) or not isinstance(entries, list):+ return+ entry = entries[current_index]+ except Exception: # noqa: BLE001+ return+ entry_id = str(entry.get("id") or "")+ if not entry_id or entry_id in state["seen_navigation_entry_ids"]:+ return+ state["seen_navigation_entry_ids"].add(entry_id)+ if suppressed:+ return+ transition_type = entry.get("transitionType")+ url = entry.get("url")+ if transition_type not in USER_NAVIGATION_TRANSITIONS or not isinstance(url, str) or not url:+ return+ self._enqueue_event(+ TapEvent(+ page=state["page"],+ session_key=session_key,+ frame_id=frame_id,+ is_main_frame=True,+ frame_name="",+ payload={+ "kind": "navigate",+ "ts": timestamp_ms,+ "url": url,+ "source_url": previous_url,+ "transition_type": transition_type,+ },+ )+ )++ def begin_control_navigation(self) -> None:+ self._control_navigation_depth += 1++ def end_control_navigation(self) -> None:+ self._control_navigation_depth = max(0, self._control_navigation_depth - 1)++ @staticmethod+ def _frame_records(tree: dict[str, Any]) -> dict[str, dict[str, Any]]:+ records: dict[str, dict[str, Any]] = {}++ def visit(node: dict[str, Any]) -> None:+ frame = node["frame"]+ records[str(frame["id"])] = frame+ for child in node.get("childFrames", []):+ visit(child)++ visit(tree["frameTree"])+ return records++ def _reconcile_frames(self, session_key: str, tree: dict[str, Any]) -> None:+ state = self._sessions.get(session_key)+ if state is None:+ return+ records = self._frame_records(tree)+ state["frames"] = records+ state["main_frame_id"] = str(tree["frameTree"]["frame"]["id"])+ for frame_id in records:+ task_key = (session_key, frame_id)+ if task_key not in self._poll_tasks or self._poll_tasks[task_key].done():+ task = asyncio.create_task(self._poll_frame(session_key, frame_id))+ self._poll_tasks[task_key] = task+ for task_key, task in list(self._poll_tasks.items()):+ if task_key[0] == session_key and task_key[1] not in records:+ task.cancel()+ self._poll_tasks.pop(task_key, None)++ async def _scan_once(self) -> None:+ for page in list(self.context.pages):+ await self._install_page(page)+ for key in list(self._sessions):+ state = self._sessions.get(key)+ if state is None:+ continue+ try:+ tree = await asyncio.wait_for(state["session"].send("Page.getFrameTree"), timeout=3)+ except TimeoutError:+ continue+ except Exception as exc: # noqa: BLE001+ if _is_detached_session_error(exc):+ await self._drop_session(key)+ continue+ self._reconcile_frames(key, tree)+ for page in list(self.context.pages):+ for frame in list(page.frames):+ if frame.parent_frame is not None:+ await self._install_oopif(page, frame)++ async def poll_new_frames(self) -> None:+ while not self._stopped:+ await asyncio.sleep(0.5)+ try:+ await self._scan_once()+ except asyncio.CancelledError:+ raise+ except Exception:+ LOG.exception("copilot_demonstration_frame_scan_failed")++ async def _poll_frame(self, session_key: str, frame_id: str) -> None:+ while not self._stopped:+ state = self._sessions.get(session_key)+ if state is None or frame_id not in state["frames"]:+ return+ if not await self._drain_frame(session_key, frame_id, max_wait_ms=1000):+ await asyncio.sleep(0.1)++ async def _drain_frame(+ self,+ session_key: str,+ frame_id: str,+ *,+ max_wait_ms: int,+ flush: bool = False,+ context_id: int | None = None,+ ) -> bool:+ state = self._sessions.get(session_key)+ if state is None or frame_id not in state["frames"]:+ return False+ acknowledged_stream, acknowledged_seq = state["drain_acks"].get(frame_id, (None, 0))+ drain = (+ f"globalThis.__skyvernDemoTap.drain({max_wait_ms}, {json.dumps(acknowledged_stream)}, {acknowledged_seq})"+ )+ expression = (+ f"globalThis.__skyvernDemoTap ? (globalThis.__skyvernDemoTap.flush(), {drain}) : null"+ if flush+ else f"globalThis.__skyvernDemoTap ? {drain} : null"+ )+ try:+ if context_id is None:+ world = await state["session"].send(+ "Page.createIsolatedWorld",+ {"frameId": frame_id, "worldName": WORLD, "grantUniveralAccess": False},+ )+ context_id = world["executionContextId"]+ state["worlds"][frame_id] = context_id+ result = await asyncio.wait_for(+ state["session"].send(+ "Runtime.evaluate",+ {+ "expression": expression,+ "contextId": context_id,+ "awaitPromise": True,+ "returnByValue": True,+ },+ ),+ timeout=3,+ )+ except asyncio.CancelledError:+ raise+ except Exception: # noqa: BLE001+ return False+ if not self._stopped:+ self._enqueue_events(session_key, state, frame_id, result.get("result", {}).get("value"))+ return True++ def _enqueue_events(self, session_key: str, state: TapSessionState, frame_id: str, result: Any) -> None:+ if not isinstance(result, dict) or not isinstance(result.get("stream"), str):+ return+ stream = result["stream"]+ payloads = result.get("events")+ if not isinstance(payloads, list):+ return+ seen_sequences = state["seen_event_sequences"].setdefault(frame_id, {})+ last_seen = seen_sequences.get(stream, 0)+ record = state["frames"].get(frame_id, {})+ for payload in payloads:+ if isinstance(payload, dict) and isinstance(payload.get("seq"), int) and payload["seq"] > last_seen:+ queued = self._enqueue_event(+ TapEvent(+ page=state["page"],+ session_key=session_key,+ frame_id=frame_id,+ is_main_frame=bool(state["is_page_session"] and frame_id == state["main_frame_id"]),+ frame_name=str(record.get("name") or ""),+ payload=payload,+ )+ )+ if not queued:+ break+ last_seen = payload["seq"]+ seen_sequences[stream] = last_seen+ state["drain_acks"][frame_id] = (stream, last_seen)+ while len(seen_sequences) > 2:+ seen_sequences.pop(next(iter(seen_sequences)))++ async def _drop_session(self, session_key: str) -> None:+ state = self._sessions.pop(session_key, None)+ if state is None:+ return+ for task_key, task in list(self._poll_tasks.items()):+ if task_key[0] == session_key:+ task.cancel()+ self._poll_tasks.pop(task_key, None)+ for page_key, key in list(self._page_sessions.items()):+ if key == session_key:+ self._page_sessions.pop(page_key, None)+ for frame_key, key in list(self._oopif_sessions.items()):+ if key == session_key:+ self._oopif_sessions.pop(frame_key, None)+ with contextlib.suppress(Exception):+ await state["session"].detach()++ def drain_events(self) -> list[TapEvent]:+ events = [self._events.get_nowait() for _ in range(self._events.qsize())]+ return sorted(events, key=lambda event: event.payload.get("ts", 0))++ async def flush(self) -> None:+ for session_key, state in list(self._sessions.items()):+ for frame_id in list(state["frames"]):+ with contextlib.suppress(Exception):+ await self._drain_frame(session_key, frame_id, max_wait_ms=0, flush=True)+ await asyncio.sleep(0)++ async def set_intercept_file_chooser_dialog(self, enabled: bool) -> None:+ self._intercept_file_chooser = enabled+ await asyncio.gather(+ *(+ state["session"].send("Page.setInterceptFileChooserDialog", {"enabled": enabled})+ for state in self._sessions.values()+ )+ )++ async def frame_owner_selector(self, event: TapEvent) -> str | None:+ if event.is_main_frame:+ return None+ event_state = self._sessions.get(event.session_key)+ if event_state is None:+ return None+ frame_record = event_state["frames"].get(event.frame_id, {})+ parent_id = str(frame_record.get("parentId") or "")+ page_key = self._page_sessions.get(id(event.page))+ page_state = self._sessions.get(page_key or "")+ if page_state is None or parent_id != page_state["main_frame_id"]:+ return None+ return await self._frame_owner_selector_on_session(page_state, event.frame_id, parent_id)++ async def frame_path(self, event: TapEvent) -> list[dict[str, str]] | None:+ if event.is_main_frame:+ return []+ event_state = self._sessions.get(event.session_key)+ page_key = self._page_sessions.get(id(event.page))+ page_state = self._sessions.get(page_key or "")+ if event_state is None or page_state is None:+ return None+ frame_id = event.frame_id+ reverse_path: list[dict[str, str]] = []+ while frame_id != page_state["main_frame_id"]:+ state = event_state if frame_id in event_state["frames"] else page_state+ frame_record = state["frames"].get(frame_id)+ if frame_record is None:+ return None+ parent_id = str(frame_record.get("parentId") or "")+ if not parent_id:+ return None+ owner_state = page_state if parent_id == page_state["main_frame_id"] else state+ owner = await self._frame_owner_metadata_on_session(owner_state, frame_id, parent_id)+ if owner is None:+ return None+ name = owner["name"]+ if name:+ reverse_path.append({"name": name})+ else:+ reverse_path.append({"owner_selector": owner["selector"]})+ frame_id = parent_id+ return list(reversed(reverse_path))++ async def _frame_owner_selector_on_session(+ self, state: TapSessionState, frame_id: str, parent_id: str+ ) -> str | None:+ metadata = await self._frame_owner_metadata_on_session(state, frame_id, parent_id)+ return metadata["selector"] if metadata is not None else None++ async def _frame_owner_metadata_on_session(+ self, state: TapSessionState, frame_id: str, parent_id: str+ ) -> dict[str, str] | None:+ session = state["session"]+ try:+ try:+ owner = await session.send("DOM.getFrameOwner", {"frameId": frame_id})+ except Exception: # noqa: BLE001+ await session.send("DOM.enable")+ owner = await session.send("DOM.getFrameOwner", {"frameId": frame_id})+ world = await session.send(+ "Page.createIsolatedWorld",+ {"frameId": parent_id, "worldName": WORLD, "grantUniveralAccess": False},+ )+ resolved = await session.send(+ "DOM.resolveNode",+ {+ "backendNodeId": owner["backendNodeId"],+ "executionContextId": world["executionContextId"],+ },+ )+ result = await session.send(+ "Runtime.callFunctionOn",+ {+ "functionDeclaration": (+ "function() { if (!globalThis.__skyvernDemoTap) return null; "+ "return {selector: globalThis.__skyvernDemoTap.selectorFor(this), "+ "name: this.getAttribute('name') || ''}; }"+ ),+ "objectId": resolved["object"]["objectId"],+ "returnByValue": True,+ },+ )+ except Exception: # noqa: BLE001+ return None+ value = result.get("result", {}).get("value")+ if not isinstance(value, dict) or not isinstance(value.get("selector"), str):+ return None+ return {"selector": value["selector"], "name": str(value.get("name") or "")}++ async def stop(self) -> None:+ if self._stopped:+ return+ await self.flush()+ self._stopped = True+ if self._page_listener is not None:+ with contextlib.suppress(Exception):+ self.context.remove_listener("page", self._page_listener)+ sessions = list(self._sessions.items())++ async def stop_session(session_key: str, state: TapSessionState) -> None:+ session = state["session"]+ for frame_id in list(state["frames"]):+ with contextlib.suppress(Exception):+ world = await session.send(+ "Page.createIsolatedWorld",+ {"frameId": frame_id, "worldName": WORLD, "grantUniveralAccess": False},+ )+ result = await asyncio.wait_for(+ session.send(+ "Runtime.evaluate",+ {+ "expression": ("globalThis.__skyvernDemoTap && globalThis.__skyvernDemoTap.stop()"),+ "contextId": world["executionContextId"],+ "returnByValue": True,+ },+ ),+ timeout=3,+ )+ self._enqueue_events(session_key, state, frame_id, result.get("result", {}).get("value"))+ if state["script_id"]:+ with contextlib.suppress(Exception):+ await session.send("Page.removeScriptToEvaluateOnNewDocument", {"identifier": state["script_id"]})++ await asyncio.gather(*(stop_session(key, state) for key, state in sessions))+ tasks = list(self._poll_tasks.values()) + list(self._background_tasks)+ if self._scan_task is not None:+ tasks.append(self._scan_task)+ for task in tasks:+ task.cancel()+ if tasks:+ await asyncio.gather(*tasks, return_exceptions=True)+ for _, state in sessions:+ with contextlib.suppress(Exception):+ await state["session"].detach()+ self._sessions.clear()+ self._poll_tasks.clear()+ self._page_sessions.clear()+ self._oopif_sessions.clear()+ self._refused_frames.clear()+++def _is_detached_session_error(exc: Exception) -> bool:+ message = str(exc).lower()+ return any(+ token in message+ for token in (+ "target closed",+ "session closed",+ "session is detached",+ "connection closed",+ "browser has been closed",+ )+ )diff --git a/skyvern/forge/sdk/copilot/enforcement.py b/skyvern/forge/sdk/copilot/enforcement.pyindex f04b058a8..2adeeab05 100644--- a/skyvern/forge/sdk/copilot/enforcement.py+++ b/skyvern/forge/sdk/copilot/enforcement.py@@ -13,7 +13,7 @@ import reimport timefrom collections.abc import Mapping, Sequencefrom dataclasses import dataclass-from typing import TYPE_CHECKING, Any+from typing import TYPE_CHECKING, Any, TypeVarfrom urllib.parse import urlparseimport structlog@@ -23,11 +23,13 @@ from agents.memory.session import Sessionfrom agents.run import Runnerfrom agents.run_config import RunConfigfrom agents.run_context import RunContextWrapper+from agents.tool import Toolfrom openai.types.responses import ResponseFunctionToolCallfrom PIL import Imagefrom skyvern.config import settingsfrom skyvern.forge.sdk.copilot import streaming_adapter+from skyvern.forge.sdk.copilot.ask_user import ACCOUNT_GROUP_SUBMIT_TOOL_NAMEfrom skyvern.forge.sdk.copilot.budget_expiry import (BudgetExpiredReplyObservation,BudgetExpirySource,@@ -65,7 +67,7 @@ from skyvern.forge.sdk.copilot.config import (DEFAULT_TOKEN_BUDGET,CopilotConfig,)-from skyvern.forge.sdk.copilot.context import REPLY_TOOL_NAME+from skyvern.forge.sdk.copilot.context import BLOCK_RUNNING_TOOLS, REPLY_TOOL_NAME, WORKFLOW_MUTATION_TOOLSfrom skyvern.forge.sdk.copilot.credential_fill_fields import LIVE_SCOUT_CREDENTIAL_FIELDSfrom skyvern.forge.sdk.copilot.credential_pause import maybe_credential_pause, release_credential_pause_gatefrom skyvern.forge.sdk.copilot.diagnosis_repair_contract import RepairNextAction, effective_proxy_label@@ -107,7 +109,11 @@ from skyvern.forge.sdk.copilot.result_evidence import (from skyvern.forge.sdk.copilot.runtime import (AgentContext,)-from skyvern.forge.sdk.copilot.screenshot_utils import ScreenshotActionRelation, ScreenshotEntry+from skyvern.forge.sdk.copilot.screenshot_utils import (+ CarriesPendingScreenshots,+ ScreenshotActionRelation,+ ScreenshotEntry,+)from skyvern.forge.sdk.copilot.steer import take_steer_input, watch_for_steerfrom skyvern.forge.sdk.copilot.terminal_predicates import (artifact_health_blocked,@@ -145,6 +151,8 @@ LOG = structlog.get_logger()TOTAL_TIMEOUT_SECONDS = settings.WORKFLOW_COPILOT_TOTAL_TIMEOUT_SECONDS or 900BUDGET_DRAIN_HEADROOM = 4+ToolT = TypeVar("ToolT", bound=Tool)+BUDGET_DRAIN_EXCLUDED_TOOLS = WORKFLOW_MUTATION_TOOLS | BLOCK_RUNNING_TOOLS | {ACCOUNT_GROUP_SUBMIT_TOOL_NAME}HARD_BACKSTOP_ALLOWANCE_SECONDS = 90# A turn whose budget is already spent may have no backstop time left for its one final reply, so# that reply gets at least this long. Budget-drain reports that finished took a median 14s, p90 35s.@@ -483,6 +491,12 @@ def _same_page(left: str | None, right: str | None) -> bool:return left_path == right_path+def pending_screenshot_entry(ctx: object) -> ScreenshotEntry | None:+ if not isinstance(ctx, CarriesPendingScreenshots) or not ctx.pending_screenshots:+ return None+ return ctx.pending_screenshots[0]++def pending_screenshot_message(ctx: Any, *, sharp: bool = False) -> dict[str, Any] | None:"""Build the synthetic user message for the staged frame without draining it.@@ -1433,18 +1447,15 @@ async def _run_streamed_with_deadline(return result-async def _unrun_call_outputs(agent: Agent, ctx: CopilotContext) -> dict[str, str] | None:- """Results by call id when the response in progress names a tool the request did not offer, else None.+def budget_drain_tools(tools: Sequence[ToolT]) -> list[ToolT]:+ return [tool for tool in tools if tool.name not in BUDGET_DRAIN_EXCLUDED_TOOLS]+++def unrun_call_outputs(calls: Sequence[ResponseFunctionToolCall], offered: set[str]) -> dict[str, str] | None:+ """Results by call id when a response names a tool the request did not offer, else None.The SDK rejects such a response before running any of its calls, so every call gets a result."""- response = ctx.last_model_response- if response is None:- return None- calls = [item for item in response.output if isinstance(item, ResponseFunctionToolCall)]- offered = {tool.name for tool in await agent.get_all_tools(RunContextWrapper(ctx))}- unknown = sorted({call.name for call in calls if call.name not in offered})- if not unknown:+ if all(call.name in offered for call in calls):return None- LOG.warning("copilot_unknown_tool_call", tool_names=unknown, call_count=len(calls))return {call.call_id: json.dumps({@@ -1460,6 +1471,22 @@ async def _unrun_call_outputs(agent: Agent, ctx: CopilotContext) -> dict[str, st}+async def _unrun_call_outputs(agent: Agent, ctx: CopilotContext) -> dict[str, str] | None:+ response = ctx.last_model_response+ if response is None:+ return None+ calls = [item for item in response.output if isinstance(item, ResponseFunctionToolCall)]+ offered = {tool.name for tool in await agent.get_all_tools(RunContextWrapper(ctx))}+ outputs = unrun_call_outputs(calls, offered)+ if outputs is not None:+ LOG.warning(+ "copilot_unknown_tool_call",+ tool_names=sorted({call.name for call in calls if call.name not in offered}),+ call_count=len(calls),+ )+ return outputs++async def _save_unrun_call_results(agent: Agent, ctx: CopilotContext, session: Session | None, stream: EventSourceStream) -> bool:@@ -1572,7 +1599,7 @@ async def _run_budget_drain()try:result = await _run_drain(- agent,+ agent.clone(tools=budget_drain_tools(agent.tools)),observation,ctx,session,@@ -2147,6 +2174,9 @@ async def run_with_enforcement(**runner_kwargs: Any,) -> RunResultStreaming:"""Run agent with enforcement nudges, preserving conversation history."""+ # A retry after a failed drain would otherwise restart the full tool set on a fresh clock.+ if ctx.budget_expiry_state.drain_attempted:+ raise MaxTurnsExceeded("Budget drain already attempted")session = runner_kwargs.pop("session", None)copilot_config = runner_kwargs.pop("copilot_config", None) or CopilotConfig()current_input: str | list = initial_input@@ -2381,9 +2411,16 @@ async def run_with_enforcement(# OpenAI rejects images in tool messages, so a queued post-run# screenshot rides as its own user message just before the nudge.+ entry = pending_screenshot_entry(ctx)screenshot_msg = _consume_pending_screenshots(ctx)if screenshot_msg is not None:- LOG.info("Injecting screenshot user message", count=len(screenshot_msg["content"]) - 1)+ LOG.info(+ "Injecting screenshot user message",+ count=len(screenshot_msg["content"]) - 1,+ source_tool=entry.provenance.source_tool if entry else None,+ capture_id=entry.capture_id if entry else None,+ capture_event_id=entry.capture_event_id if entry else None,+ )with copilot_span("enforcement_nudge", data={"nudge_type": nudge_type, "iteration": iteration}):extra_msgs = _assemble_enforcement_messages(screenshot_msg, decision.message)diff --git a/skyvern/forge/sdk/copilot/human_input_wait.py b/skyvern/forge/sdk/copilot/human_input_wait.pyindex ea2a61fd7..1fa3a1ced 100644--- a/skyvern/forge/sdk/copilot/human_input_wait.py+++ b/skyvern/forge/sdk/copilot/human_input_wait.py@@ -40,7 +40,7 @@ def pause_human_input(ctx: CopilotContext, kind: Literal["question", "credentialelapsed = time.monotonic() - wait.started_atif wait.owner == "question":ctx.copilot_question_pause_seconds += elapsed- else:+ elif wait.owner == "credential":ctx.copilot_credential_pause_seconds += elapseddeadline = ctx.model_stream_deadlineif deadline is not None and wait.deadline_when is not None and not deadline.expired():diff --git a/skyvern/forge/sdk/copilot/mcp_adapter.py b/skyvern/forge/sdk/copilot/mcp_adapter.pyindex 3a2eea6d1..a8dba4001 100644--- a/skyvern/forge/sdk/copilot/mcp_adapter.py+++ b/skyvern/forge/sdk/copilot/mcp_adapter.py@@ -75,6 +75,7 @@ from skyvern.forge.sdk.copilot.runtime import (browser_session_turn,close_browser_session_quietly,current_call_browser_session_override,+ current_demonstrated_action,effective_browser_session_id,ensure_browser_session,mcp_browser_context,@@ -1833,17 +1834,20 @@ class SkyvernOverlayMCPServer(MCPServer):propagated_error: BaseExceptioncopilot_ctx = self._context_provider()overlay = self._overlays.get(tool_name, SchemaOverlay())+ demonstrated_action = current_demonstrated_action()# Bound out here so one call's browser reaches everything it touches: pre-hooks, session# preparation, dispatch and post-hooks. Binding only the dispatch would collect evidence# from one browser, act in another, then record the result from the first.# Resolved once. A sibling call in the same turn can record or clear the run session, so# resolving again for dispatch could bind the hooks to one browser and the dispatch to another.call_binding = (- resolve_browser_session_binding(copilot_ctx, arguments or {}) if overlay.requires_browser else None+ (resolve_browser_session_binding(copilot_ctx, arguments or {}) if overlay.requires_browser else None)+ if demonstrated_action is None+ else None)owning_call_id = (meta or {}).get(_TOOL_CALL_ID_META_KEY)try:- if overlay.requires_browser:+ if overlay.requires_browser and demonstrated_action is None:async with browser_session_turn(copilot_ctx, session_id=call_binding.session_id_for(copilot_ctx) if call_binding else None):@@ -1892,6 +1896,71 @@ class SkyvernOverlayMCPServer(MCPServer):mcp_name = self._alias_map.get(tool_name, tool_name)if self._enforce_allowlist_on_dispatch and mcp_name not in self._allowlist:raise ValueError(f"MCP tool is not available on this Copilot surface: {tool_name}")+ held_failures = HeldRequestFailures()+ demonstrated_action = current_demonstrated_action()+ if demonstrated_action is not None:+ existing_interactions = {id(item) for item in copilot_ctx.scouted_interactions}+ existing_trajectory = {id(item) for item in copilot_ctx.scout_trajectory}++ def stamp_user_actor() -> None:+ for item in copilot_ctx.scouted_interactions:+ if id(item) not in existing_interactions:+ item["actor"] = "user"+ for item in copilot_ctx.scout_trajectory:+ if id(item) not in existing_trajectory:+ item["actor"] = "user"++ if overlay.pre_hook is not None:+ refusal = await overlay.pre_hook(arguments, copilot_ctx)+ if refusal is not None:+ LOG.info("copilot_demonstration_replay_refused", tool=tool_name)+ stamp_user_actor()+ projected_refusal = scrub_model_facing_tool_result(copilot_ctx, refusal, tool_name=mcp_name)+ projected_refusal = _scrub_stamped_tool_result(+ copilot_ctx, held_failures, projected_refusal, tool_name=mcp_name+ )+ record_tool_step_result_for_ctx(copilot_ctx, tool_name, arguments, projected_refusal)+ return _copilot_to_call_tool_result(projected_refusal, tool_name)++ raw_mcp = scrub_model_facing_tool_result(+ copilot_ctx,+ deepcopy(demonstrated_action.synthesized_result),+ tool_name=mcp_name,+ )+ copilot_result = mcp_to_copilot(raw_mcp) if raw_mcp else {}+ if overlay.post_hook is not None:+ async with browser_evidence_commit_lock(copilot_ctx):+ base_copilot_result = deepcopy(copilot_result)+ ctx_snapshot = _snapshot_post_hook_context(copilot_ctx)+ held_token = POST_HOOK_HELD_REQUEST_FAILURES.set(held_failures)+ try:+ copilot_result = await overlay.post_hook(copilot_result, raw_mcp, copilot_ctx)+ except asyncio.CancelledError:+ _restore_post_hook_context(copilot_ctx, ctx_snapshot)+ raise+ except Exception: # noqa: BLE001+ _restore_post_hook_context(copilot_ctx, ctx_snapshot)+ LOG.warning(+ "MCP post-hook failed",+ tool=tool_name,+ fails_closed=overlay.post_hook_fails_closed,+ )+ copilot_result = (+ {+ "ok": False,+ "error": f"{tool_name} result could not be prepared for display and is withheld.",+ }+ if overlay.post_hook_fails_closed+ else base_copilot_result+ )+ finally:+ POST_HOOK_HELD_REQUEST_FAILURES.reset(held_token)+ stamp_user_actor()+ copilot_result = _scrub_stamped_tool_result(copilot_ctx, held_failures, copilot_result, tool_name=mcp_name)+ record_tool_step_result_for_ctx(copilot_ctx, tool_name, arguments, copilot_result)+ held_failures.release()+ return _copilot_to_call_tool_result(copilot_result, tool_name)+observed_continuity_generation = copilot_ctx.browser_session_continuity_generationattempt_browser_session_id = current_call_browser_session_override() or getattr(copilot_ctx, "browser_session_id", None@@ -1903,8 +1972,6 @@ class SkyvernOverlayMCPServer(MCPServer):if binding is None:binding = resolve_browser_session_binding(copilot_ctx, arguments or {})- held_failures = HeldRequestFailures()-async def _stamp_page_state(result: dict[str, Any], *, probe: bool = True) -> None:reader = self._page_state_readerif reader is None or not result or not overlay.requires_browser or "page_state" in result:@@ -2524,6 +2591,22 @@ class SkyvernOverlayMCPServer(MCPServer):call_session_override = current_call_browser_session_override()attempt_browser_session_id = requested_session_id or call_session_override or ctx.browser_session_iduses_shared_browser_outcome = mcp_tool_name in _SHARED_BROWSER_OUTCOME_TOOLS+ if current_demonstrated_action() is not None:+ result = {"ok": False, "error": "Browser access is unavailable while replaying a demonstration."}+ if uses_shared_browser_outcome:+ outcome = _not_dispatched_browser_call_outcome(+ raw_tool_name=mcp_tool_name,+ source_browser_session_id=attempt_browser_session_id,+ source_browser_session_generation=observed_continuity_generation,+ raw_result=result,+ ctx=ctx,+ )+ outcome = _scrub_browser_call_outcome(ctx, outcome)+ return _InternalToolCallResult(+ result=_project_browser_call_outcome(outcome, display_tool_name=copilot_name),+ browser_outcome=outcome,+ )+ return _InternalToolCallResult(result=scrub_model_facing_tool_result(ctx, result, tool_name=mcp_tool_name))dispatch_started = Falseif not self._client:_log_mcp_timing(ctx, copilot_name, mcp_tool_name, phases, {}, "internal", "not_connected")diff --git a/skyvern/forge/sdk/copilot/output_extraction_plan.py b/skyvern/forge/sdk/copilot/output_extraction_plan.pyindex f495f526a..18e005165 100644--- a/skyvern/forge/sdk/copilot/output_extraction_plan.py+++ b/skyvern/forge/sdk/copilot/output_extraction_plan.py@@ -920,8 +920,10 @@ def value_designation_probe_expression(value_text: str, label: str) -> str:" owned = labelNodes.filter((n) => anchor.contains(n)); if (owned.length) break; }"" const valueLike = /\\d/.test(label) && !/[A-Za-z]{3}/.test(label);"" if (anchor && !valueLike && owned.length === 1 && runs(owned[0]).includes(label)) {"- " const base = shape(anchor); const labelShape = shape(owned[0]); const inner = shape(chosen);"+ " const base = shape(anchor); const inner = shape(chosen);"" const quoted = '\"' + label.split('\\\\').join('\\\\\\\\').split('\"').join('\\\\\"') + '\"';"+ # The label's classes are named only where its bare tag is ambiguous, as in the page-evidence twin.+ " for (const labelShape of new Set([owned[0].tagName.toLowerCase(), shape(owned[0])])) {"" const anchorSelector = base + ':has(' + labelShape + ':text-is(' + quoted + ')) ' + inner;"" const anchors = new Set(); const matched = [];"" for (const root of roots) for (const node of root.querySelectorAll(labelShape)) {"@@ -929,8 +931,9 @@ def value_designation_probe_expression(value_text: str, label: str) -> str:" for (let p = up(node); p; p = up(p)) if (p.matches && p.matches(base)) anchors.add(p); }"" for (const root of roots) for (const node of root.querySelectorAll(inner)) {"" for (let p = up(node); p; p = up(p)) if (anchors.has(p)) { matched.push(node); break; } }"- " if (matched.length === 1 && matched[0] === chosen && anchorSelector.length <= 160)"- " selectorCandidates.push({ selector: anchorSelector, source: 'text_anchor', match_count: 1, position: 0 }); } }"+ " if (matched.length === 1 && matched[0] === chosen && anchorSelector.length <= 160) {"+ " selectorCandidates.push({ selector: anchorSelector, source: 'text_anchor', match_count: 1, position: 0 });"+ " break; } } } }"" const all = Array.from(document.querySelectorAll(selector));"" const position = all.indexOf(chosen);"" if (position < 0) return { error: 'path-unstable', text: target, url: location.href };"diff --git a/skyvern/forge/sdk/copilot/output_utils.py b/skyvern/forge/sdk/copilot/output_utils.pyindex b36917ad9..e8bb6b809 100644--- a/skyvern/forge/sdk/copilot/output_utils.py+++ b/skyvern/forge/sdk/copilot/output_utils.py@@ -43,6 +43,7 @@ from skyvern.forge.sdk.copilot.build_test_outcome import ()from skyvern.forge.sdk.copilot.composition_evidence import (INTERNAL_VALIDATION_FAILURE_PREFIX,+ fit_authoring_facts,model_visible_composition_evidence,)from skyvern.forge.sdk.copilot.context import (@@ -1642,8 +1643,15 @@ def _fit_post_run_page_evidence(result: dict[str, Any]) -> None:packet = data.get("post_run_page_evidence") if isinstance(data, dict) else Noneif not isinstance(data, dict) or not isinstance(packet, dict):return- if len(json.dumps(result)) > _RECENT_TOOL_OUTPUT_CHAR_CAP:- data["post_run_page_evidence"] = model_visible_composition_evidence(packet)+ if len(json.dumps(result)) <= _RECENT_TOOL_OUTPUT_CHAR_CAP:+ return++ def fits(candidate: dict[str, Any]) -> bool:+ return len(json.dumps({**result, "data": {**data, "post_run_page_evidence": candidate}})) <= (+ _RECENT_TOOL_OUTPUT_CHAR_CAP+ )++ data["post_run_page_evidence"] = fit_authoring_facts(packet, model_visible_composition_evidence(packet), fits)def sanitize_tool_result_for_llm(tool_name: str, result: dict[str, Any]) -> dict[str, Any]:diff --git a/skyvern/forge/sdk/copilot/request_policy.py b/skyvern/forge/sdk/copilot/request_policy.pyindex 9b759e12d..bdc42f3cf 100644--- a/skyvern/forge/sdk/copilot/request_policy.py+++ b/skyvern/forge/sdk/copilot/request_policy.py@@ -392,7 +392,7 @@ def _redact_cited_secrets(message: str, citations: list[str]) -> str:return "".join(pieces)-async def _screen_raw_secret_safety(+async def screen_raw_secret_safety(user_message: str,handler: LLMAPIHandler | None,*,@@ -897,7 +897,17 @@ class SteerMessageSiteURLSource:kind: Literal["steer_message"] = field(default="steer_message", init=False)-SiteURLSource = UserMessageSiteURLSource | QuestionResponseSiteURLSource | SteerMessageSiteURLSource+@dataclass(frozen=True)+class DemonstrationNavigationSiteURLSource:+ kind: Literal["demonstration_navigation"] = field(default="demonstration_navigation", init=False)+++SiteURLSource = (+ UserMessageSiteURLSource+ | QuestionResponseSiteURLSource+ | SteerMessageSiteURLSource+ | DemonstrationNavigationSiteURLSource+)@dataclass(frozen=True)@@ -1025,6 +1035,13 @@ class RequestPolicy:return_project_user_provided_sites(self, _steer_message_url_texts(steer), reset=False)+ def project_demonstration_navigation(self, url: str) -> None:+ _project_user_provided_sites(+ self,+ [_SiteURLText(text=url, source=DemonstrationNavigationSiteURLSource())],+ reset=False,+ )+@propertydef raw_secret_redacted_draft(self) -> bool:return self.raw_secret_detected and self.raw_secret_handling == "redacted_draft"@@ -4901,7 +4918,7 @@ async def build_request_policy_trust_floor(selected_connected_account_id: str | None = None,) -> RequestPolicy:del config- safety = await _screen_raw_secret_safety(user_message, handler, organization_id=organization_id)+ safety = await screen_raw_secret_safety(user_message, handler, organization_id=organization_id)credential_ids = _credential_ids(user_message)raw_secret_present = safety.status == "detected"# Only an unavailable screen withholds the turn. A screen that ran and cited redacts what itdiff --git a/skyvern/forge/sdk/copilot/runtime.py b/skyvern/forge/sdk/copilot/runtime.pyindex 8a8ac44a3..1964f5e84 100644--- a/skyvern/forge/sdk/copilot/runtime.py+++ b/skyvern/forge/sdk/copilot/runtime.py@@ -345,6 +345,7 @@ class ScoutedSelectorCandidate(TypedDict):source: strmatch_count: int | Nonevisible_match_count: NotRequired[int]+ first_match_visible: NotRequired[bool]class ScoutedInteraction(TypedDict):@@ -352,6 +353,7 @@ class ScoutedInteraction(TypedDict):# those named in context._TURN_EPHEMERAL_INTERACTION_FIELDS. A field added here that holds a# raw value — a literal, page text — has to be listed there too; nothing enforces the pair.tool_name: str+ actor: NotRequired[Literal["user"]]# Set only when the action was demonstrated somewhere other than the chat's own browser, so# absence means the chat's browser. Without it a demonstration in the last run's browser is# indistinguishable from one the chat drove.@@ -944,6 +946,31 @@ async def _resolve_self_heal_browser_state_inner(ctx: AgentContext) -> tuple[str_CALL_BROWSER_SESSION_ID: ContextVar[str | None] = ContextVar("copilot_call_browser_session_id", default=None)+@dataclass(frozen=True)+class DemonstratedAction:+ selector: str+ source_url: str+ packet: dict[str, Any] | None+ synthesized_result: dict[str, Any]+ readback: str | None = None+++_DEMONSTRATED_ACTION: ContextVar[DemonstratedAction | None] = ContextVar("copilot_demonstrated_action", default=None)+++@contextmanager+def staged_demonstrated_action(action: DemonstratedAction) -> Iterator[None]:+ token = _DEMONSTRATED_ACTION.set(action)+ try:+ yield+ finally:+ _DEMONSTRATED_ACTION.reset(token)+++def current_demonstrated_action() -> DemonstratedAction | None:+ return _DEMONSTRATED_ACTION.get()++@contextmanagerdef bound_call_browser_session(session_id: str | None) -> Iterator[None]:"""Bind one tool call's browser for everything it reaches: hooks, preparation, dispatch.@@ -1098,6 +1125,8 @@ async def resolve_browser_state_for_context(*,session_id: str | None = None,) -> BrowserState | None:+ if current_demonstrated_action() is not None:+ return Noneresolved_session_id = session_id if session_id is not None else _CALL_BROWSER_SESSION_ID.get()if resolved_session_id is None:resolved_session_id = ctx.browser_session_iddiff --git a/skyvern/forge/sdk/copilot/screenshot_utils.py b/skyvern/forge/sdk/copilot/screenshot_utils.pyindex aa8d4e3a8..7c469bc28 100644--- a/skyvern/forge/sdk/copilot/screenshot_utils.py+++ b/skyvern/forge/sdk/copilot/screenshot_utils.py@@ -124,12 +124,25 @@ def capturing_tool_call(tool_call_id: str | None) -> Iterator[None]:_capturing_tool_call_id.reset(token)+def capturing_tool_call_id() -> str | None:+ return _capturing_tool_call_id.get()++@runtime_checkableclass CarriesChatScreenshots(Protocol):pending_chat_screenshots: list[ChatScreenshotFrame]chat_screenshot_capture_ids: set[str]+@runtime_checkable+class CarriesPendingScreenshots(Protocol):+ pending_screenshots: list[ScreenshotEntry]+++class CarriesPendingFrames(CarriesPendingScreenshots, Protocol):+ pending_frame_lease: PendingFrameLease | None++def screenshot_result_facts(result: dict[str, Any],*,@@ -308,6 +321,17 @@ def enqueue_screenshot(return True+def withdraw_frame_captured_before(ctx: CarriesPendingFrames, cutoff: float) -> ScreenshotEntry | None:+ """Drop the pending frame if it was captured before `cutoff`, releasing a lease on it, and return it."""+ pending = ctx.pending_screenshots+ if not pending or pending[0].captured_at >= cutoff:+ return None+ entry = pending.pop(0)+ if ctx.pending_frame_lease is not None and ctx.pending_frame_lease.capture_event_id == entry.capture_event_id:+ ctx.pending_frame_lease = None+ return entry++def consume_screenshot_artifact(result: dict[str, Any]) -> bytes | None:data = result.get("data")path = data.get("path") if isinstance(data, dict) else Nonediff --git a/skyvern/forge/sdk/copilot/session_factory.py b/skyvern/forge/sdk/copilot/session_factory.pyindex 7501a660a..23866135f 100644--- a/skyvern/forge/sdk/copilot/session_factory.py+++ b/skyvern/forge/sdk/copilot/session_factory.py@@ -28,6 +28,7 @@ from skyvern.forge.sdk.copilot.enforcement import (_RECENT_TOOL_OUTPUT_CHAR_CAP,_TOOL_OUTPUT_HEAD_TRUNCATION_SUFFIX,KEEP_RECENT_TOOL_OUTPUTS,+ NUDGE_SENTINEL,SCREENSHOT_PLACEHOLDER,TOKEN_BUDGET,_summarize_tool_arguments,@@ -38,6 +39,7 @@ from skyvern.forge.sdk.copilot.enforcement import (is_screenshot_message,is_synthetic_user_message,log_recent_tool_output_truncation,+ pending_screenshot_entry,pending_screenshot_message,unread_tool_output_indices,)@@ -227,8 +229,7 @@ def _copilot_call_model_input_filter(data: CallModelData[Any], *, token_budget:# Read-only peek: the end-of-turn drain in enforcement stays the only clear, so a provider# retry or model fallback re-running this filter still carries the frame.screenshot_msg = pending_screenshot_message(ctx)- pending = getattr(ctx, "pending_screenshots", None)- entry = pending[0] if isinstance(pending, list) and pending else None+ entry = pending_screenshot_entry(ctx)if (screenshot_msg is not Noneand entry is not None@@ -238,8 +239,7 @@ def _copilot_call_model_input_filter(data: CallModelData[Any], *, token_budget:lease = getattr(ctx, "pending_frame_lease", None)if isinstance(lease, PendingFrameLease) and lease.capture_event_id == entry.capture_event_id:if lease.input_fingerprint != fingerprint:- if isinstance(pending, list):- pending.clear()+ ctx.pending_screenshots.clear()ctx.pending_frame_lease = Nonescreenshot_msg = NoneLOG.info(@@ -264,6 +264,9 @@ def _copilot_call_model_input_filter(data: CallModelData[Any], *, token_budget:"Injecting screenshot user message",count=len(screenshot_msg["content"]) - 1,path="model_input_filter",+ source_tool=entry.provenance.source_tool if entry else None,+ capture_id=entry.capture_id if entry else None,+ capture_event_id=entry.capture_event_id if entry else None,)items.append(screenshot_msg)@@ -431,7 +434,33 @@ def _filter_to_budget(items: list[Any], instructions: str | None, *, token_budgebudget=token_budget,unread_count=len(unread_tool_output_indices(items)),)+ demonstration_call_ids = {+ call_id+ for item in items+ if isinstance(call_id := get_agent_message_field(item, "call_id"), str)+ and call_id.startswith("call_demo_")+ and get_agent_message_field(item, "type") == "function_call"+ }items = aggressive_prune(items)+ retained_demonstration_call_ids = {+ call_id+ for item in items+ if isinstance(call_id := get_agent_message_field(item, "call_id"), str)+ and call_id in demonstration_call_ids+ and get_agent_message_field(item, "type") == "function_call"+ }+ omitted_demonstration_calls = len(demonstration_call_ids - retained_demonstration_call_ids)+ if omitted_demonstration_calls:+ items.append(+ {+ "role": "user",+ "content": (+ NUDGE_SENTINEL+ + f"{omitted_demonstration_calls} earlier demonstrated step fact packet(s) were omitted "+ "because the model input exceeded its token budget."+ ),+ }+ )est = estimate_tokens(items)LOG.info("Final token estimate after aggressive prune", tokens=est)diff --git a/skyvern/forge/sdk/copilot/tools/__init__.py b/skyvern/forge/sdk/copilot/tools/__init__.pyindex 5f7031e52..716546ae4 100644--- a/skyvern/forge/sdk/copilot/tools/__init__.py+++ b/skyvern/forge/sdk/copilot/tools/__init__.py@@ -44,7 +44,12 @@ from skyvern.forge.sdk.copilot.composition_evidence import (from skyvern.forge.sdk.copilot.composition_evidence import workflow_target_url as workflow_target_urlfrom skyvern.forge.sdk.copilot.config import AuthoringCapability, BlockAuthoringPolicyfrom skyvern.forge.sdk.copilot.context import (+ BLOCK_RUNNING_TOOLS,+ EDIT_BLOCK_AND_RUN_TOOL_NAME,+ EDIT_BLOCK_TOOL_NAME,REPLY_TOOL_NAME,+ UPDATE_AND_RUN_BLOCKS_TOOL_NAME,+ UPDATE_WORKFLOW_TOOL_NAME,USER_FACING_REASON_PARAM,USER_FACING_REASON_SCHEMA,CopilotContext,@@ -128,10 +133,7 @@ from skyvern.utils.yaml_loader import dump_workflow_yamlfrom ._shared import _COMPOSITION_STRIPPED_HTML_MAX_CHARS as _COMPOSITION_STRIPPED_HTML_MAX_CHARSfrom ._shared import _DISCOVERY_PER_CALL_TIMEOUT_SECONDS as _DISCOVERY_PER_CALL_TIMEOUT_SECONDSfrom ._shared import _FAILED_BLOCK_STATUSES as _FAILED_BLOCK_STATUSES-from ._shared import BLOCK_RUNNING_TOOLS as BLOCK_RUNNING_TOOLS-from ._shared import EDIT_BLOCK_AND_RUN_TOOL_NAME, EDIT_BLOCK_TOOL_NAMEfrom ._shared import RUN_BLOCKS_SAFETY_CEILING_SECONDS as RUN_BLOCKS_SAFETY_CEILING_SECONDS-from ._shared import UPDATE_AND_RUN_BLOCKS_TOOL_NAME, UPDATE_WORKFLOW_TOOL_NAMEfrom ._shared import AdmittedOutputRead as AdmittedOutputReadfrom ._shared import RequestedOutputRead as RequestedOutputReadfrom ._shared import _composition_get_html as _composition_get_html@@ -184,10 +186,12 @@ from .composition_capture import _same_inspect_target as _same_inspect_targetfrom .credential_fill import _credential_fill_authority_error as _credential_fill_authority_errorfrom .credential_fill import _credential_fill_prerequisite_error as _credential_fill_prerequisite_errorfrom .credential_fill import (- _fill_credential_field_impl,_request_credential,)from .credential_fill import _resolve_credential_fill_value as _resolve_credential_fill_value+from .credential_fill import (+ fill_credential_field,+)from .credentials import (LIST_CREDENTIALS_MAX_PAGE_SIZE,)@@ -2239,7 +2243,7 @@ async def fill_credential_field_tool(# Never fall back to the chat's browser: a credential filled there lands on a page the model did not name.return json.dumps({"ok": False, "error": binding.unavailable_reason, **binding.provenance()})with bound_call_browser_session(binding.session_id_override):- result = await _fill_credential_field_impl(ctx.context, selector, credential_id, field, submit_selector)+ result = await fill_credential_field(ctx.context, selector, credential_id, field, submit_selector)return json.dumps(scrub_secrets_from_structure(ctx.context, {**result, **binding.provenance()}))diff --git a/skyvern/forge/sdk/copilot/tools/_shared.py b/skyvern/forge/sdk/copilot/tools/_shared.pyindex ce31abd66..608b91b37 100644--- a/skyvern/forge/sdk/copilot/tools/_shared.py+++ b/skyvern/forge/sdk/copilot/tools/_shared.py@@ -285,22 +285,6 @@ def _has_meaningful_registered_output_payload(data: Mapping[str, Any]) -> bool:)-EDIT_BLOCK_TOOL_NAME = "edit_block"-EDIT_BLOCK_AND_RUN_TOOL_NAME = "edit_block_and_run"-UPDATE_AND_RUN_BLOCKS_TOOL_NAME = "update_and_run_blocks"-UPDATE_WORKFLOW_TOOL_NAME = "update_workflow"-RUN_BLOCKS_TOOL_NAME = "run_blocks_and_collect_debug"-BLANK_BROWSER_TEST_TOOL_NAME = "test_workflow_from_blank_browser"--BLOCK_RUNNING_TOOLS = frozenset(- {RUN_BLOCKS_TOOL_NAME, UPDATE_AND_RUN_BLOCKS_TOOL_NAME, EDIT_BLOCK_AND_RUN_TOOL_NAME, BLANK_BROWSER_TEST_TOOL_NAME}-)--WORKFLOW_MUTATION_TOOLS = frozenset(- {UPDATE_WORKFLOW_TOOL_NAME, UPDATE_AND_RUN_BLOCKS_TOOL_NAME, EDIT_BLOCK_AND_RUN_TOOL_NAME}-)--CREDENTIAL_METADATA_TOOLS = frozenset({"list_credentials"})diff --git a/skyvern/forge/sdk/copilot/tools/banned_blocks.py b/skyvern/forge/sdk/copilot/tools/banned_blocks.pyindex 431a86f12..2ebbca608 100644--- a/skyvern/forge/sdk/copilot/tools/banned_blocks.py+++ b/skyvern/forge/sdk/copilot/tools/banned_blocks.py@@ -401,7 +401,7 @@ def _code_only_browser_schema_guidance(*, agent_blocks: bool = False, image_ocr:"A saved run executes this block against a page it loads itself, without the interactions performed while scouting. Whatever the page requires before the target is reachable is part of what the block does, not a condition it inherits.",_code_only_browser_validation_guidance(agent_blocks=agent_blocks),"Keep block outputs JSON-safe and include visible evidence text when extracting records, products, totals, confirmations, or identifiers.",- "Wait for the value the block returns, not for a URL or a navigation. A page reaches its final URL while it is still rendering, so a URL check passes before the value exists and a navigation wait fails on a page that has already arrived.",+ "Wait for the value the block returns, not for a URL or a navigation. A page reaches its final URL while it is still rendering, so a URL check passes before the value exists and a navigation wait fails on a page that has already arrived. In a saved run each statement starts as soon as the previous one returns.",_saved_credential_guidance(),_secret_credential_guidance(),"The Code runtime provides await solve_captcha(page) for a platform-managed verification challenge observed while scouting.",diff --git a/skyvern/forge/sdk/copilot/tools/browser_code.py b/skyvern/forge/sdk/copilot/tools/browser_code.pyindex ddea77418..5df807b87 100644--- a/skyvern/forge/sdk/copilot/tools/browser_code.py+++ b/skyvern/forge/sdk/copilot/tools/browser_code.py@@ -6,6 +6,7 @@ import asyncioimport hashlibimport jsonimport math+import timeimport uuidfrom collections.abc import Callablefrom typing import Any@@ -15,7 +16,7 @@ from agents import FunctionToolfrom agents.run_context import RunContextWrapperfrom playwright.async_api import Page-from skyvern.cli.core.session_manager import get_page+from skyvern.cli.core.session_manager import get_page, resolve_browserfrom skyvern.forge import appfrom skyvern.forge.sdk.copilot.browser_code_contract import (DEMONSTRATED_OPERATIONS,@@ -34,7 +35,11 @@ from skyvern.forge.sdk.copilot.browser_target import (last_run_facts,resolve_browser_session_binding,)-from skyvern.forge.sdk.copilot.enforcement import HARD_BACKSTOP_ALLOWANCE_SECONDS, TOTAL_TIMEOUT_SECONDS+from skyvern.forge.sdk.copilot.enforcement import (+ HARD_BACKSTOP_ALLOWANCE_SECONDS,+ TOTAL_TIMEOUT_SECONDS,+ pending_screenshot_entry,+)from skyvern.forge.sdk.copilot.loop_detection import record_tool_step_result_for_ctxfrom skyvern.forge.sdk.copilot.mcp_adapter import (_browser_call_outcome_from_mapping,@@ -57,11 +62,19 @@ from skyvern.forge.sdk.copilot.runtime import (effective_browser_session_id,mcp_browser_context,)+from skyvern.forge.sdk.copilot.screenshot_utils import capturing_tool_call_id, withdraw_frame_captured_before+from skyvern.utils.contained_effects import contained_effectfrom skyvern.webeye.browser_errors import BrowserAutomationError+from ._shared import _DISCOVERY_PER_CALL_TIMEOUT_SECONDSfrom .errors import reporting_withheld_browserfrom .guardrails import _authority_tool_error-from .scouting import _mark_pending_browser_interaction_observation, _record_scouted_interaction+from .scouting import (+ _capture_post_interaction_screenshot,+ _mark_pending_browser_interaction_observation,+ _record_scouted_interaction,+ _take_viewport_frame,+)LOG = structlog.get_logger()@@ -90,13 +103,19 @@ expression is returned as `value`; `print` output is returned as `stdout`.Browser API (async, Playwright-shaped):- `page` is the current tab. Locators: `page.locator(css)`, `page.get_by_role(role, name=...)`,- `get_by_text`, `get_by_label`, `get_by_placeholder`, `.first`, `.last`, `.nth(i)`, `.filter(...)`.+ `get_by_text`, `get_by_label`, `get_by_placeholder`, `.first`, `.last`, `.nth(i)`, `.filter(...)`,+ `.or_(other_locator)`.- Actions: `.click()`, `.fill(text)`, `.press(key)`, `.check()`, `.select_option(...)`, `.hover()`,- `page.keyboard.press(key)`.+ `page.keyboard.press(key)`. `.click()` and `.press(key)` wait for a navigation they start until the new+ page starts loading, not until it has loaded; the other actions, `page.keyboard.press`, and a navigation+ the page's script starts later, such as after a timer or a network request, are not waited for.- Waits: `page.wait_for_selector(...)`, `page.wait_for_url(...)`, `page.wait_for_load_state(...)`,- `locator.wait_for(...)`.+ `locator.wait_for(...)`. `wait_for_selector` and `locator.wait_for` keep polling while a navigation+ replaces the page, until the element matches or the timeout ends.- DOM reads: `.text_content()`, `.inner_text()`, `.get_attribute(name)`, `.input_value()`, `.count()`,- `.is_visible()`, `page.content()`, `page.evaluate(js)`, `page.title()`, and `page.url`.+ `.is_visible()`, `page.content()`, `page.evaluate(js)`, `page.title()`, and `page.url`. `.count()`+ and `.is_visible()` answer once, for the page present at the call: while a navigation replaces it,+ `.count()` raises `Execution context was destroyed` and `.is_visible()` returns False.- Navigation: `page.goto(url)`, `page.go_back()`, `page.reload()`. Internal, private-network, andnon-web destinations are refused.- Frames: `page.frames`, `page.main_frame`, `page.frame_locator(css)`.@@ -310,8 +329,17 @@ def _cell_payload(cell: BrowserCodeCellResult) -> dict[str, Any]:def _take_interruption_note(host: BrowserCodeHost) -> dict[str, Any]:+ notes: dict[str, Any] = {}+ unreturned = host.unreturned_cell+ if unreturned is not None:+ host.unreturned_cell = None+ payload = _cell_payload(unreturned)+ notes["previous_call_completed"] = {+ "state": "the previous run_browser_code call finished; its result was not returned",+ **{key: payload[key] for key in ("ok", "error_code", "current_url", "operations") if key in payload},+ }if not host.interrupted:- return {}+ return notesoperation = host.interrupted_operationhost.interrupted = Falsehost.interrupted_operation = None@@ -320,7 +348,8 @@ def _take_interruption_note(host: BrowserCodeHost) -> dict[str, Any]:note["last_operation"] = _operation_fact(operation)if operation.succeeded is None:note["page_after_call"] = "unknown: that operation reached the browser without a reply"- return {"previous_call_interrupted": note}+ notes["previous_call_interrupted"] = note+ return notesasync def _discard_session(host: BrowserCodeHost) -> None:@@ -539,6 +568,7 @@ async def _run_bound_cell(browser_evidence_commit_lock(copilot_ctx),browser_session_recovery(copilot_ctx),):+ cell_started = time.monotonic()notes = _take_interruption_note(host)session_to_reset = host.session if fresh_namespace else Nonebrowser_session_id = effective_browser_session_id(copilot_ctx)@@ -583,6 +613,8 @@ async def _run_bound_cell(cell = await _run_cell(host, session, code, timeout, deny_pixels=pixels_denied)except BrowserCodeSessionUnavailableError as exc:if exc.session_ended:+ if entered_browser:+ withdraw_frame_captured_before(copilot_ctx, cell_started)# Dropped now, or the retry this refusal asks for is spent rediscovering that the# session is over, and only the call after that reopens one.await _discard_session(host)@@ -591,6 +623,8 @@ async def _run_bound_cell(refusal["retry_after_seconds"] = exc.retry_after_secondsreturn finish(refusal)except (CopilotBrowserGenerationRetired, CopilotBrowserSessionUnavailable) as exc:+ if entered_browser:+ withdraw_frame_captured_before(copilot_ctx, cell_started)disposition = await _browser_session_error_disposition(copilot_ctx, exc, tool_name=TOOL_NAME, call_path="model")@@ -661,14 +695,94 @@ async def _run_bound_cell(if last.succeeded is None:ended["page_after_call"] = "unknown: that operation reached the browser without a reply"notes["session_ended_during_call"] = ended- await _discard_session(host)- result = finish(- {**_cell_payload(cell), "executed_source_reference": source_reference, **notes, **binding.provenance()}- )- _record_outcome(copilot_ctx, cell, result, browser_session_id=browser_session_id, generation=generation)+ try:+ result = finish(+ {**_cell_payload(cell), "executed_source_reference": source_reference, **notes, **binding.provenance()}+ )+ _record_outcome(copilot_ctx, cell, result, browser_session_id=browser_session_id, generation=generation)+ try:+ await _stage_cell_frame(+ copilot_ctx,+ cell,+ session,+ browser_session_id=browser_session_id,+ session_id_override=binding.session_id_override,+ cell_started=cell_started,+ )+ finally:+ if not cell.session_alive:+ await asyncio.shield(_discard_session(host))+ except asyncio.CancelledError:+ host.unreturned_cell = cell+ raisereturn result+async def _stage_cell_frame(+ copilot_ctx: AgentContext,+ cell: BrowserCodeCellResult,+ session: BrowserCodeSession,+ *,+ browser_session_id: str | None,+ session_id_override: str | None,+ cell_started: float,+) -> None:+ started = time.monotonic()+ staged = False+ reason = "cancelled"+ try:+ # A session that ended under the cell, as a timeout ends it, can still have confirmed the tab it left.+ page = session.confirmed_page+ if page is None:+ reason = "tab_unconfirmed" if cell.session_alive else "session_ended"+ elif copilot_ctx.codeblock_redaction_parameters:+ reason = "self_heal"+ elif not copilot_ctx.supports_vision:+ reason = "no_vision"+ else:+ try:+ # A fresh lease, so a browser retired after the cell stops the capture instead of shooting a+ # closing tab.+ async with mcp_browser_context(copilot_ctx, session_id_override=session_id_override):+ # Checked here, not before the cell: a secret can reach this browser while the cell runs.+ if browser_session_id in copilot_ctx.secret_run_browser_session_ids:+ reason = "secret_run"+ elif page not in (await resolve_browser(session_id=browser_session_id))[0].pages:+ # The browser was replaced before this lease; its old tab can still answer a screenshot.+ reason = "browser_retired"+ else:+ frame = await _take_viewport_frame(+ copilot_ctx, timeout_seconds=_DISCOVERY_PER_CALL_TIMEOUT_SECONDS, page=page+ )+ if frame is None:+ reason = "screenshot_failed"+ else:+ staged = await _capture_post_interaction_screenshot(+ copilot_ctx, source_tool=TOOL_NAME, captured_url=cell.current_url, frame=frame+ )+ reason = "staged" if staged else "not_staged"+ except (CopilotBrowserGenerationRetired, CopilotBrowserSessionUnavailable):+ reason = "browser_retired"+ except Exception as exc:+ # The cell has already acted on the page; a failed picture must not turn its result into an error.+ reason = f"capture_error:{type(exc).__name__}"+ finally:+ # Otherwise a frame from before this cell would reach the model as if this cell had produced it.+ withdrawn = None if staged else withdraw_frame_captured_before(copilot_ctx, cell_started)+ with contained_effect("record browser-code cell frame", tool_name=TOOL_NAME):+ entry = pending_screenshot_entry(copilot_ctx) if staged else None+ LOG.info(+ "copilot_browser_code_cell_frame",+ staged=staged,+ reason=reason,+ tool_call_id=capturing_tool_call_id(),+ capture_id=entry.capture_id if entry else None,+ capture_event_id=entry.capture_event_id if entry else None,+ withdrawn_capture_id=withdrawn.capture_id if withdrawn else None,+ capture_ms=round((time.monotonic() - started) * 1000),+ )++async def close_browser_code_session(copilot_ctx: AgentContext) -> None:host = copilot_ctx.browser_code_hosttry:diff --git a/skyvern/forge/sdk/copilot/tools/composition_capture.py b/skyvern/forge/sdk/copilot/tools/composition_capture.pyindex a0211c523..c8a01efa2 100644--- a/skyvern/forge/sdk/copilot/tools/composition_capture.py+++ b/skyvern/forge/sdk/copilot/tools/composition_capture.py@@ -30,6 +30,7 @@ from skyvern.forge.sdk.copilot.composition_browser_expressions import (from skyvern.forge.sdk.copilot.composition_evidence import (_MAX_KEY_VALUE_RELATIONS,CONSENT_OBSTRUCTION_KIND,+ fit_authoring_facts,has_bounded_page_schema,has_satisfiable_collapsed_disclosure_path,merge_visual_composition_evidence,@@ -168,11 +169,17 @@ def _model_facing_inspect_result(result: dict[str, Any], *, copilot_ctx: AgentCoshaped = copy.deepcopy(result)data = shaped.get("data")if isinstance(data, dict):- for authoring_facts in (True, False):- shaped["data"] = model_visible_composition_evidence(data, authoring_facts=authoring_facts)- _attach_field_upload_routes(shaped["data"], upload_routes_for(copilot_ctx))- if len(json.dumps(shaped)) <= _RECENT_TOOL_OUTPUT_CHAR_CAP - _PAGE_STATE_RESERVE_CHARS:- break+ upload_routes = upload_routes_for(copilot_ctx)+ projected, bare = (model_visible_composition_evidence(data, authoring_facts=facts) for facts in (True, False))+ for packet in (projected, bare):+ _attach_field_upload_routes(packet, upload_routes)++ def fits(packet: dict[str, Any]) -> bool:+ return (+ len(json.dumps({**shaped, "data": packet})) <= _RECENT_TOOL_OUTPUT_CHAR_CAP - _PAGE_STATE_RESERVE_CHARS+ )++ shaped["data"] = fit_authoring_facts(projected, bare, fits)if len(json.dumps(shaped)) <= _RECENT_TOOL_OUTPUT_CHAR_CAP:return shapeddata = shaped.get("data")diff --git a/skyvern/forge/sdk/copilot/tools/credential_fill.py b/skyvern/forge/sdk/copilot/tools/credential_fill.pyindex 97ae9e2c3..c88a09d1e 100644--- a/skyvern/forge/sdk/copilot/tools/credential_fill.py+++ b/skyvern/forge/sdk/copilot/tools/credential_fill.py@@ -42,6 +42,7 @@ from skyvern.forge.sdk.copilot.credential_resolution import is_resolved_page_urlfrom skyvern.forge.sdk.copilot.loop_detection import record_tool_step_result_for_ctxfrom skyvern.forge.sdk.copilot.page_identity import safe_page_originfrom skyvern.forge.sdk.copilot.request_policy import (+ DemonstrationNavigationSiteURLSource,QuestionResponseSiteURLSource,RequestPolicy,SiteURLSource,@@ -454,6 +455,8 @@ def _log_fill_grant(source_fields = {"source_kind": source.kind, "source_interaction_id": source.interaction_id}elif isinstance(source, SteerMessageSiteURLSource):source_fields = {"source_kind": source.kind, "source_steer_id": source.steer_id}+ elif isinstance(source, DemonstrationNavigationSiteURLSource):+ source_fields = {"source_kind": source.kind}LOG.info("copilot credential fill grant",route=route,@@ -1328,7 +1331,7 @@ async def _submit_after_credential_fill()-async def _fill_credential_field_impl(+async def fill_credential_field(copilot_ctx: AgentContext,selector: str,credential_id: str,diff --git a/skyvern/forge/sdk/copilot/tools/mcp_hooks.py b/skyvern/forge/sdk/copilot/tools/mcp_hooks.pyindex cf86fa0d9..b7fabb0b6 100644--- a/skyvern/forge/sdk/copilot/tools/mcp_hooks.py+++ b/skyvern/forge/sdk/copilot/tools/mcp_hooks.py@@ -68,6 +68,7 @@ from skyvern.forge.sdk.copilot.runtime import (ScoutedInteraction,ScoutedSelectorCandidate,browser_valid_tab_count,+ current_demonstrated_action,tab_switch_refusal,)from skyvern.forge.sdk.copilot.screenshot_utils import viewport_visible_effect@@ -770,6 +771,7 @@ async def _get_workflow_knowledge_post_hook(_MODEL_SCOUT_FACT_KEYS = ("tool_name",+ "actor","executed_selector","selector_candidates","role",@@ -1566,7 +1568,13 @@ async def _type_text_post_hook(has_landing_probe = (isinstance(selector, str) and bool(selector.strip()) and isinstance(typed_length, int) and typed_length > 0)- field_value = await _read_scout_field_value(ctx, selector) if has_landing_probe else None+ demonstrated_action = current_demonstrated_action()+ if demonstrated_action is not None:+ field_value = demonstrated_action.readback+ elif has_landing_probe:+ field_value = await _read_scout_field_value(ctx, selector)+ else:+ field_value = Nonecontrol_readonly, control_disabled = (await _probe_scout_control_state(ctx, selector) if has_landing_probe else (None, None))diff --git a/skyvern/forge/sdk/copilot/tools/run_execution.py b/skyvern/forge/sdk/copilot/tools/run_execution.pyindex 23e7bbace..b8d3cd72a 100644--- a/skyvern/forge/sdk/copilot/tools/run_execution.py+++ b/skyvern/forge/sdk/copilot/tools/run_execution.py@@ -46,7 +46,6 @@ from skyvern.forge.sdk.copilot.budget_expiry import (BudgetExpiryState,RecordedBlockState,RecordedRunState,- budget_run_denial,)from skyvern.forge.sdk.copilot.build_test_connect_failure import (SUPERSEDED_BY_NEWER_TEST_REASON,@@ -3744,8 +3743,6 @@ async def _run_blocks_and_collect_debug() -> dict[str, Any]:"""With ``new_exit``, run in a browser whose network exit is verified to differ from the most recent buildtest's; the model cannot choose or verify an exit, so the platform supplies it or says why it cannot."""- if ctx.budget_expiry_state.drain_active:- return dict(budget_run_denial(ctx.budget_expiry_state))new_exit_prior_session_id = ctx.last_run_blocks_browser_session_idif new_exit and new_exit_prior_session_id is None:return {@@ -7007,9 +7004,7 @@ def _is_undispatched_run(result: Mapping[str, object]) -> bool:def _skips_build_test_finalize(result: Mapping[str, object]) -> bool:data = result.get("data")return (- isinstance(data, dict)- and data.get("run_dispatched") is False- and (data.get("budget_expired") is True or data.get(_PROPOSAL_BIND_REFUSED_KEY) is True)+ isinstance(data, dict) and data.get("run_dispatched") is False and data.get(_PROPOSAL_BIND_REFUSED_KEY) is True)diff --git a/skyvern/forge/sdk/copilot/tools/scouting.py b/skyvern/forge/sdk/copilot/tools/scouting.pyindex 7d0567b0e..7cc5b8c28 100644--- a/skyvern/forge/sdk/copilot/tools/scouting.py+++ b/skyvern/forge/sdk/copilot/tools/scouting.py@@ -13,6 +13,7 @@ from urllib.parse import parse_qs, urlparse, urlsplitimport structlogfrom playwright.async_api import BrowserContext, Download, Frame, Page, Response+from skyvern.cli.core.browser_ops import do_screenshotfrom skyvern.config import settingsfrom skyvern.forge import appfrom skyvern.forge.sdk.copilot.browser_code_contract import BROWSER_SESSION_UNAVAILABLE_ERROR_CODE@@ -42,6 +43,7 @@ from skyvern.forge.sdk.copilot.composition_browser_expressions import (selector_match_count_expression as _selector_match_count_expression,)from skyvern.forge.sdk.copilot.composition_evidence import (+ CANDIDATE_COUNT_KEYS,SCOUT_INTERACTION_EVIDENCE_TOOL,has_actionable_steer_content,has_bounded_page_schema,@@ -59,6 +61,7 @@ from skyvern.forge.sdk.copilot.enforcement import ()from skyvern.forge.sdk.copilot.mcp_adapter import (BROWSER_SESSION_LOSS_ERROR_CODES,+ is_redaction_withheld,scrub_model_facing_tool_result,service_worker_blocked_context,)@@ -74,12 +77,14 @@ from skyvern.forge.sdk.copilot.runtime import (ScoutedInteraction,ScoutedSelectorCandidate,current_call_browser_session_override,+ current_demonstrated_action,effective_browser_session_id,live_working_page,raw_secret_browser_denied,resolve_browser_state_for_context,)from skyvern.forge.sdk.copilot.screenshot_utils import (+ ProvenanceBinding,ScreenshotActionRelation,ScreenshotProvenance,ViewportFrame,@@ -177,6 +182,10 @@ async def _live_working_page_url(ctx: AgentContext) -> str | None:async def _capture_scout_source_url(ctx: AgentContext) -> None:# Pre-action: a navigating click/Enter would leave only the destination URL, not the page the selector acted on.+ demonstrated_action = current_demonstrated_action()+ if demonstrated_action is not None:+ ctx.pending_scout_source_url = demonstrated_action.source_url+ returnsource_url = await _live_working_page_url(ctx)ctx.pending_scout_source_url = source_url@@ -392,6 +401,11 @@ async def _capture_scout_pre_action(ctx: AgentContext, selector: str | None) ->selector = _selector_text(selector)if not selector:return+ demonstrated_action = current_demonstrated_action()+ if demonstrated_action is not None and selector == demonstrated_action.selector:+ if demonstrated_action.packet is not None:+ _apply_scout_pre_action_packet(ctx, selector, demonstrated_action.packet, source="demonstration")+ returnparsed = _role_name_from_selector(selector)source = "selector" if parsed is not None else "page_read"parsed_role, parsed_name = (parsed[0].strip(), parsed[1].strip()) if parsed is not None else ("", "")@@ -546,17 +560,23 @@ async def _capture_post_interaction_screenshot(frame = await _take_viewport_frame(ctx, timeout_seconds=timeout_seconds)if frame is None:return False+ # The provenance reaches the model as text, so its URLs pass the same scrub as a tool result's.+ urls = scrub_model_facing_tool_result(+ ctx, {"captured_url": frame.producer_url, "dispatch_url": captured_url}, tool_name=source_tool+ )+ if is_redaction_withheld(urls):+ urls = {}return enqueue_screenshot(ctx,base64.b64encode(frame.png).decode("ascii"),provenance=ScreenshotProvenance(source_tool=source_tool,- captured_url=frame.producer_url,+ captured_url=urls.get("captured_url"),observation_step=observation_step,browser_session_id=frame.producer_session_id,workflow_run_id=None,action_relation=ScreenshotActionRelation.AFTER_SOURCE_ACTION,- dispatch_url=captured_url,+ dispatch_url=urls.get("dispatch_url"),dispatch_browser_session_id=frame.dispatch_session_id,producer_browser_session_id=frame.producer_session_id,session_binding=frame.session_binding,@@ -566,15 +586,33 @@ async def _capture_post_interaction_screenshot()-async def _take_viewport_frame(ctx: AgentContext, *, timeout_seconds: float) -> ViewportFrame | None:- """Read the viewport without staging it; withheld during self-heal."""+async def _take_viewport_frame(+ ctx: AgentContext, *, timeout_seconds: float, page: Page | None = None+) -> ViewportFrame | None:+ """Read the viewport without staging it; withheld during self-heal. A given `page` is shot directly,+ because the broker's current tab does not follow a worker's `switch_tab`."""if ctx.codeblock_redaction_parameters:return None+ started_at = time.monotonic()+ dispatch_session_id = effective_browser_session_id(ctx)+ if page is not None:+ try:+ shot = await asyncio.wait_for(do_screenshot(page), timeout=timeout_seconds)+ except Exception as exc:+ LOG.debug("Copilot viewport frame of the given page failed", error_type=type(exc).__name__)+ return None+ return ViewportFrame(+ png=shot.data,+ dispatch_session_id=dispatch_session_id,+ producer_url=page.url,+ producer_session_id=dispatch_session_id,+ session_binding=ProvenanceBinding.AGREE,+ started_at=started_at,+ viewport_css_px=shot.viewport_css_px,+ )server = ctx.discovery_mcp_serverif server is None:return None- started_at = time.monotonic()- dispatch_session_id = effective_browser_session_id(ctx)screenshot_arguments = {"session_id": dispatch_session_id} if dispatch_session_id else {}try:result = await asyncio.wait_for(@@ -606,7 +644,7 @@ def _model_visible_selector_candidates(value: Any) -> list[Any]:if not isinstance(value, list):return []return [- {key: item for key, item in candidate.items() if key not in {"match_count", "visible_match_count"}}+ {key: item for key, item in candidate.items() if key not in CANDIDATE_COUNT_KEYS}if isinstance(candidate, dict)else candidatefor candidate in value@@ -702,6 +740,45 @@ def _record_scout_trajectory_fact(ctx: AgentContext, artifact: ScoutedInteractioreturn recorded+def record_scouted_interaction_fact(ctx: AgentContext, artifact: ScoutedInteraction) -> None:+ redacted_artifact = _scrub_retained_interaction(ctx, _redact_codeblock_value(ctx, artifact))+ if not isinstance(redacted_artifact, dict) or "tool_name" not in redacted_artifact:+ return+ artifact = cast(ScoutedInteraction, redacted_artifact)+ interactions = [+ item+ for item in ctx.scouted_interactions+ if not (+ item.get("tool_name") == artifact["tool_name"]+ and item.get("selector") == artifact.get("selector")+ and item.get("source_url") == artifact.get("source_url")+ and item.get("credential_field") == artifact.get("credential_field")+ )+ ]+ interactions.append(artifact)+ ctx.scouted_interactions = _capped_with_eviction_accounting(interactions, collection="scouted_interactions")++ recorded = _record_scout_trajectory_fact(ctx, artifact)+ if artifact["tool_name"] == "click":+ click_records: list[ScoutedInteraction] = [artifact]+ if recorded is not None:+ click_records.append(recorded)+ ctx.pending_scout_click_records = click_records++ LOG.info(+ "copilot_scout_interaction_captured",+ tool_name=artifact["tool_name"],+ selector=artifact.get("selector"),+ source_url=artifact.get("source_url"),+ role=artifact.get("role"),+ credential_field=artifact.get("credential_field"),+ credential_id=artifact.get("credential_id"),+ total_scouted_interactions=len(ctx.scouted_interactions),+ total_scout_trajectory=len(ctx.scout_trajectory),+ )+ record_reached_terminal_action_observation(ctx)++def _observed_control_readiness(ctx: AgentContext, selector: str, source_url: str) -> tuple[bool, bool]:"""Return (hidden, disabled) when bounded same-page evidence observed this exact selector unready."""packets: list[dict[str, Any]] = []@@ -917,44 +994,7 @@ def _record_scouted_interaction(artifact["element_fingerprint_probed"] = element_fingerprint_probedif ambiguous:artifact["ambiguous"] = True- redacted_artifact = _scrub_retained_interaction(ctx, _redact_codeblock_value(ctx, artifact))- if not isinstance(redacted_artifact, dict) or "tool_name" not in redacted_artifact:- return- artifact = cast(ScoutedInteraction, redacted_artifact)- interactions = [- item- for item in ctx.scouted_interactions- if not (- item.get("tool_name") == artifact["tool_name"]- and item.get("selector") == artifact.get("selector")- and item.get("source_url") == artifact.get("source_url")- and item.get("credential_field") == artifact.get("credential_field")- )- ]- interactions.append(artifact)- ctx.scouted_interactions = _capped_with_eviction_accounting(interactions, collection="scouted_interactions")-- recorded = _record_scout_trajectory_fact(ctx, artifact)- if artifact["tool_name"] == "click":- # Both collections hold their own object for this click, and an effect observed after the- # fact updates these rather than re-finding them by an identity the scrub may have removed.- click_records: list[ScoutedInteraction] = [artifact]- if recorded is not None:- click_records.append(recorded)- ctx.pending_scout_click_records = click_records-- LOG.info(- "copilot_scout_interaction_captured",- tool_name=artifact["tool_name"],- selector=artifact.get("selector"),- source_url=artifact.get("source_url"),- role=artifact.get("role"),- credential_field=artifact.get("credential_field"),- credential_id=artifact.get("credential_id"),- total_scouted_interactions=len(ctx.scouted_interactions),- total_scout_trajectory=len(ctx.scout_trajectory),- )- record_reached_terminal_action_observation(ctx)+ record_scouted_interaction_fact(ctx, artifact)def _attach_scout_observation_step(diff --git a/skyvern/forge/sdk/experimentation/copilot_demonstration.py b/skyvern/forge/sdk/experimentation/copilot_demonstration.pynew file mode 100644index 000000000..edfb3ee3e--- /dev/null+++ b/skyvern/forge/sdk/experimentation/copilot_demonstration.py@@ -0,0 +1,33 @@+import structlog++from skyvern.config import settings+from skyvern.forge import app+from skyvern.forge.sdk.cache.base import BaseCache++LOG = structlog.get_logger()++COPILOT_DEMONSTRATION_FLAG = "WORKFLOW_COPILOT_DEMONSTRATION"+++def shared_copilot_demonstration_cache() -> BaseCache | None:+ cache = app.CACHE+ return cache if cache is not None and cache.is_shared else None+++async def copilot_demonstration_enabled(organization_id: str | None) -> bool:+ if shared_copilot_demonstration_cache() is None:+ return False+ if settings.WORKFLOW_COPILOT_DEMONSTRATION_ENABLED or not organization_id:+ return settings.WORKFLOW_COPILOT_DEMONSTRATION_ENABLED+ try:+ return (+ await app.EXPERIMENTATION_PROVIDER.is_feature_enabled_cached(+ COPILOT_DEMONSTRATION_FLAG,+ organization_id,+ properties={"organization_id": organization_id},+ )+ is True+ )+ except Exception:+ LOG.warning("Failed to resolve Copilot demonstration feature flag; failing closed", exc_info=True)+ return Falsediff --git a/skyvern/forge/sdk/routes/workflow_copilot.py b/skyvern/forge/sdk/routes/workflow_copilot.pyindex 1f9a6a931..0dd1feb09 100644--- a/skyvern/forge/sdk/routes/workflow_copilot.py+++ b/skyvern/forge/sdk/routes/workflow_copilot.py@@ -76,6 +76,19 @@ from skyvern.forge.sdk.copilot.credential_pause import (start_manual_sign_in,)from skyvern.forge.sdk.copilot.credential_resolution import safe_admitted_url+from skyvern.forge.sdk.copilot.demonstration_recorder import (+ DemonstrationRecorder,+ DemonstrationRejection,+ active_demonstration,+ append_demonstration_control,+ claim_demonstration_recording,+ demonstration_action_labels,+ demonstration_limit_reached,+ demonstration_owner_is_live,+ demonstration_time_limit_reached,+ load_demonstration_record,+ validate_demonstration_for_replay,+)from skyvern.forge.sdk.copilot.enforcement import EXPIRED_TURN_REPLY_SECONDS, TOTAL_TIMEOUT_SECONDSfrom skyvern.forge.sdk.copilot.goal_suggestion import suggest_goal_from_codefrom skyvern.forge.sdk.copilot.interruption import (@@ -93,7 +106,7 @@ from skyvern.forge.sdk.copilot.recoverable_failure import (merge_failure_into_context,)from skyvern.forge.sdk.copilot.repair_origin_run import RepairOriginRefusal, resolve_repair_origin_binding-from skyvern.forge.sdk.copilot.request_policy import _screen_raw_secret_safety+from skyvern.forge.sdk.copilot.request_policy import screen_raw_secret_safetyfrom skyvern.forge.sdk.copilot.review_gate import parse_execution_receipts, serialize_execution_receiptsfrom skyvern.forge.sdk.copilot.runtime import close_browser_session_quietlyfrom skyvern.forge.sdk.copilot.steer import STEER_DOORBELL_TTL, copilot_steer_key@@ -134,6 +147,10 @@ from skyvern.forge.sdk.db.exceptions import (DuplicateCopilotTurnError,NotFoundError,)+from skyvern.forge.sdk.experimentation.copilot_demonstration import (+ copilot_demonstration_enabled,+ shared_copilot_demonstration_cache,+)from skyvern.forge.sdk.forge_log import _generated_log_valuefrom skyvern.forge.sdk.routes.browser_profiles import (_hard_delete_created_profile_after_store_failure,@@ -183,6 +200,11 @@ from skyvern.forge.sdk.schemas.workflow_copilot import (WorkflowCopilotCredentialGenerateResult,WorkflowCopilotCredentialResponseRequest,WorkflowCopilotCredentialResponseResult,+ WorkflowCopilotDemonstrationActiveResult,+ WorkflowCopilotDemonstrationControlRequest,+ WorkflowCopilotDemonstrationControlResult,+ WorkflowCopilotDemonstrationInsertion,+ WorkflowCopilotDemonstrationRecordRequest,WorkflowCopilotDisableAutoAcceptRequest,WorkflowCopilotGoalSuggestionRequest,WorkflowCopilotGoalSuggestionResponse,@@ -632,11 +654,16 @@ async def _resolve_copilot_request_config() -> CopilotConfig:copilot_config = await app.AGENT_FUNCTION.get_copilot_config_for_request(organization_id,+ # A recording becomes code; a demonstration follows the org's authoring dial like any other turn.code_block_mode=True if chat_request.product_action == "refine_recording" else chat_request.code_block,)return copilot_config or CopilotConfig()+async def _resolve_copilot_api_key(request: Request, organization_id: str) -> str | None:+ return request.headers.get("x-api-key") or await app.AGENT_FUNCTION.resolve_org_api_key(organization_id)++def _with_current_copilot_code_mode_metadata(turn_outcome: TurnOutcome | None,*,@@ -726,7 +753,7 @@ async def _watch_for_cancel(return-RouteExit = Literal["completed", "error", "cancelled", "duplicate"]+RouteExit = Literal["completed", "error", "cancelled", "duplicate", "refused"]class _FirstEventTimingStream:@@ -2885,9 +2912,7 @@ async def _new_copilot_chat_post(chat_request.workflow_permanent_id,organization.organization_id,)- api_key = request.headers.get("x-api-key")- if not api_key:- api_key = await app.AGENT_FUNCTION.resolve_org_api_key(organization.organization_id)+ api_key = await _resolve_copilot_api_key(request, organization.organization_id)if not api_key:failure_kind_for_log = "configuration"@@ -3155,6 +3180,37 @@ async def _new_copilot_chat_post(returncapture_code_mode_opt_out_after_persist()route_exit = "completed"+ except DemonstrationRejection as rejection:+ if turn_started:+ await _recover_from_route_exception(+ rejection,+ restore_log_context="demonstration replay handler",+ translated_log_message="Demonstration replay refusal translated to recoverable workflow copilot reply",+ none_chat_log_message="Demonstration replay refused after the Copilot turn started",+ none_chat_user_message=rejection.detail,+ failure_kind="server",+ )+ return+ route_exit = "refused"+ terminal_frame_emitted = True+ LOG.warning(+ "Copilot demonstration replay claim refused",+ status_code=rejection.status_code,+ organization_id=organization.organization_id,+ workflow_copilot_chat_id=(+ chat.workflow_copilot_chat_id if chat is not None else chat_request.workflow_copilot_chat_id+ ),+ turn_id=turn_id,+ )+ await stream.send(+ WorkflowCopilotStreamErrorUpdate(+ type=WorkflowCopilotStreamMessageType.ERROR,+ error=rejection.detail,+ failure_kind=None,+ turn_id=turn_id,+ narrative_summary=None,+ )+ )except HTTPException as exc:failure_kind_for_log = _http_exception_failure_kind(exc)http_exception_log = LOG.error if exc.status_code >= 500 else LOG.warning@@ -3680,6 +3736,24 @@ async def workflow_copilot_chat_post(organization: Organization = Depends(org_auth_service.get_current_org),) -> EventSourceResponse:eval_entrypoint_url = _validated_eval_entrypoint_url(request, chat_request, organization)+ if chat_request.product_action == "demonstrate":+ cache = shared_copilot_demonstration_cache()+ if not await copilot_demonstration_enabled(organization.organization_id) or cache is None:+ raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Copilot demonstration is disabled")+ if not chat_request.demonstration_id:+ raise HTTPException(+ status_code=status.HTTP_400_BAD_REQUEST,+ detail="demonstration_id is required for a Copilot demonstration",+ )+ try:+ await validate_demonstration_for_replay(+ cache,+ organization_id=organization.organization_id,+ workflow_permanent_id=chat_request.workflow_permanent_id,+ demonstration_id=chat_request.demonstration_id,+ )+ except DemonstrationRejection as rejection:+ raise HTTPException(status_code=rejection.status_code, detail=rejection.detail) from rejectionraw_eval_mode = request.headers.get("x-copilot-eval-mode")if raw_eval_mode is not None:try:@@ -4199,7 +4273,7 @@ async def workflow_copilot_question_response(handler = await resolve_raw_secret_safety_handler(chat.workflow_permanent_id, organization.organization_id)async def screen_text(text: str) -> str:- safety = await _screen_raw_secret_safety(text, handler, organization_id=organization.organization_id)+ safety = await screen_raw_secret_safety(text, handler, organization_id=organization.organization_id)if safety.status == "blocked":raise HTTPException(status_code=503, detail="The safety screen is unavailable. Please retry your answer."@@ -4372,7 +4446,7 @@ async def workflow_copilot_steer(recorded = await record()if recorded is None:handler = await resolve_raw_secret_safety_handler(chat.workflow_permanent_id, organization.organization_id)- safety = await _screen_raw_secret_safety(+ safety = await screen_raw_secret_safety(steer_request.message, handler, organization_id=organization.organization_id)if safety.status == "blocked":@@ -4628,6 +4702,148 @@ async def workflow_copilot_credential_response(return WorkflowCopilotCredentialResponseResult(result="accepted")+async def _demonstration_cache(organization_id: str) -> BaseCache:+ cache = shared_copilot_demonstration_cache()+ if not await copilot_demonstration_enabled(organization_id) or cache is None:+ raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Copilot demonstration is disabled")+ return cache+++@base_router.post("/workflow/copilot/demonstration/record", include_in_schema=False)+async def workflow_copilot_demonstration_record(+ request: Request,+ record_request: WorkflowCopilotDemonstrationRecordRequest,+ organization: Organization = Depends(org_auth_service.get_current_org),+) -> EventSourceResponse:+ cache = await _demonstration_cache(organization.organization_id)+ api_key = await _resolve_copilot_api_key(request, organization.organization_id)+ copilot_config = await app.AGENT_FUNCTION.get_copilot_config_for_request(+ organization.organization_id,+ code_block_mode=None,+ )+ copilot_config = copilot_config or CopilotConfig()+ browser_session = await app.DATABASE.browser_sessions.get_persistent_browser_session(+ record_request.browser_session_id, organization.organization_id+ )+ workflow = await app.DATABASE.workflows.get_workflow_by_permanent_id(+ workflow_permanent_id=record_request.workflow_permanent_id,+ organization_id=organization.organization_id,+ )+ if browser_session is None or workflow is None:+ raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Unknown workflow or browser session")+ try:+ claim = await claim_demonstration_recording(+ cache,+ organization_id=organization.organization_id,+ workflow_permanent_id=record_request.workflow_permanent_id,+ browser_session_id=record_request.browser_session_id,+ demonstration_id=record_request.demonstration_id,+ insertion=record_request.insertion.model_dump() if record_request.insertion is not None else None,+ )+ except DemonstrationRejection as rejection:+ raise HTTPException(status_code=rejection.status_code, detail=rejection.detail) from rejection++ async def handler(stream: EventSourceStream) -> None:+ recorder = DemonstrationRecorder(+ cache=cache,+ organization_id=organization.organization_id,+ workflow_permanent_id=record_request.workflow_permanent_id,+ browser_session_id=record_request.browser_session_id,+ claim=claim,+ stream=stream,+ api_key=api_key,+ authoring_capability=copilot_config.authoring_capability,+ )+ await recorder.run()++ return FastAPIEventSourceStream.create(request, handler)+++@base_router.post(+ "/workflow/copilot/demonstration/control",+ include_in_schema=False,+ response_model=WorkflowCopilotDemonstrationControlResult,+)+async def workflow_copilot_demonstration_control(+ control_request: WorkflowCopilotDemonstrationControlRequest,+ organization: Organization = Depends(org_auth_service.get_current_org),+) -> WorkflowCopilotDemonstrationControlResult:+ cache = await _demonstration_cache(organization.organization_id)+ if control_request.action == "navigate":+ try:+ parsed = urlparse(control_request.url or "")+ except ValueError as exc:+ raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="url must be http(s)") from exc+ if parsed.scheme not in {"http", "https"} or not parsed.hostname:+ raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="url must be http(s)")+ if control_request.action == "fill_credential" and not (+ control_request.credential_id and control_request.selector and control_request.field+ ):+ raise HTTPException(+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,+ detail="credential_id, selector, and field are required when action is 'fill_credential'",+ )+ try:+ seq = await append_demonstration_control(+ cache,+ organization_id=organization.organization_id,+ request=control_request,+ )+ except DemonstrationRejection as rejection:+ raise HTTPException(status_code=rejection.status_code, detail=rejection.detail) from rejection+ if control_request.action == "stop":+ deadline = time.monotonic() + 10+ while time.monotonic() < deadline:+ record = await load_demonstration_record(+ cache, organization.organization_id, control_request.demonstration_id+ )+ # A dead recorder never commits; Done's preflight settles the record instead of waiting here.+ if record is None or record.status != "stopping" or not demonstration_owner_is_live(record):+ break+ await asyncio.sleep(0.1)+ if record is not None and record.status == "stopped":+ return WorkflowCopilotDemonstrationControlResult(seq=seq, action_count=record.event_count)+ return WorkflowCopilotDemonstrationControlResult(seq=seq)+++@base_router.get("/workflow/copilot/demonstration/active", include_in_schema=False)+async def workflow_copilot_demonstration_active(+ workflow_permanent_id: str,+ browser_session_id: str,+ organization: Organization = Depends(org_auth_service.get_current_org),+) -> WorkflowCopilotDemonstrationActiveResult | None:+ cache = await _demonstration_cache(organization.organization_id)+ active = await active_demonstration(+ cache,+ organization_id=organization.organization_id,+ workflow_permanent_id=workflow_permanent_id,+ browser_session_id=browser_session_id,+ )+ if active is None:+ return None+ demonstration_id, record = active+ copilot_config = await app.AGENT_FUNCTION.get_copilot_config_for_request(+ organization.organization_id,+ code_block_mode=None,+ )+ copilot_config = copilot_config or CopilotConfig()+ return WorkflowCopilotDemonstrationActiveResult(+ demonstration_id=demonstration_id,+ status=record.status,+ action_count=record.event_count,+ action_labels=await demonstration_action_labels(+ cache, organization.organization_id, demonstration_id, record.event_count+ ),+ can_fill_credentials=copilot_config.authoring_capability.code_blocks,+ can_resume=not demonstration_limit_reached(record) and not demonstration_time_limit_reached(record),+ insertion=(+ WorkflowCopilotDemonstrationInsertion.model_validate(record.insertion)+ if record.insertion is not None+ else None+ ),+ )++_READBACK_PAGE_SIZE = 100diff --git a/skyvern/forge/sdk/schemas/workflow_copilot.py b/skyvern/forge/sdk/schemas/workflow_copilot.pyindex 1b843c5eb..3e9c7deef 100644--- a/skyvern/forge/sdk/schemas/workflow_copilot.py+++ b/skyvern/forge/sdk/schemas/workflow_copilot.py@@ -392,6 +392,11 @@ class WorkflowCopilotChatMessage(BaseModel):modified_at: datetime = Field(..., description="When the message was last modified")+class WorkflowCopilotDemonstrationInsertion(BaseModel):+ after_label: str | None = Field(default=None, max_length=200)+ before_label: str | None = Field(default=None, max_length=200)++class WorkflowCopilotChatRequest(BaseModel):workflow_permanent_id: str = Field(..., description="Workflow permanent ID for the chat")workflow_id: str = Field(..., description="Workflow ID (mutable version ID)")@@ -494,15 +499,21 @@ class WorkflowCopilotChatRequest(BaseModel):"a new proposal, so the client can keep rendering an actionable review gate."),)- product_action: Literal["test_end_to_end", "diagnose_run", "refine_recording"] | None = Field(+ product_action: Literal["test_end_to_end", "diagnose_run", "refine_recording", "demonstrate"] | None = Field(None,description=("Structured product action for this turn, dispatched by the server instead of the agent. ""'test_end_to_end' runs every block of the pending proposal in a browser session minted ""for that run. 'diagnose_run' opens repair on the finished run named by workflow_run_id. "- "'refine_recording' infers a reusable workflow from recording_evidence."+ "'refine_recording' infers a reusable workflow from recording_evidence. "+ "'demonstrate' replays a completed browser demonstration before the first model call."),)+ demonstration_id: str | None = Field(default=None, max_length=64)+ demonstration_insertion: WorkflowCopilotDemonstrationInsertion | None = Field(+ None,+ description="Optional placement bounds for blocks authored after a demonstration.",+ )recording_evidence: RecordingEvidencePacket | None = Field(None,description=(@@ -580,6 +591,37 @@ class WorkflowCopilotCredentialResponseRequest(BaseModel):credential_id: str | None = Field(None, description="Saved credential ID; required when action is 'connected'")+class WorkflowCopilotDemonstrationControlRequest(BaseModel):+ demonstration_id: str = Field(max_length=64)+ action: Literal["navigate", "fill_credential", "stop", "cancel"]+ url: str | None = Field(default=None, max_length=2048)+ selector: str | None = Field(default=None, max_length=2000)+ credential_id: str | None = Field(default=None, max_length=128)+ field: Literal["password", "totp"] | None = None+++class WorkflowCopilotDemonstrationControlResult(BaseModel):+ seq: int+ action_count: int | None = None+++class WorkflowCopilotDemonstrationRecordRequest(BaseModel):+ workflow_permanent_id: str = Field(max_length=128)+ browser_session_id: str = Field(max_length=128)+ demonstration_id: str | None = Field(default=None, max_length=64)+ insertion: WorkflowCopilotDemonstrationInsertion | None = None+++class WorkflowCopilotDemonstrationActiveResult(BaseModel):+ demonstration_id: str+ status: Literal["recording", "stopping", "stopped"]+ action_count: int+ action_labels: list[str]+ can_fill_credentials: bool+ can_resume: bool+ insertion: WorkflowCopilotDemonstrationInsertion | None = None++class WorkflowCopilotCredentialGenerateRequest(BaseModel):turn_id: str = Field(..., description="turn_id from the matching credential_required frame")workflow_copilot_chat_id: str = Field(..., description="chat ID from the matching credential_required frame")diff --git a/skyvern/forge/sdk/services/route_authorization.py b/skyvern/forge/sdk/services/route_authorization.pyindex a44e92eac..f7cc8c944 100644--- a/skyvern/forge/sdk/services/route_authorization.py+++ b/skyvern/forge/sdk/services/route_authorization.py@@ -131,7 +131,7 @@ ROUTE_ACTIONS: Mapping[str, str] = {"skyvern.forge.sdk.routes.credentials.delete_credential": "credential.delete","skyvern.forge.sdk.routes.credentials.delete_credential_folder": "credential_folder.delete","skyvern.forge.sdk.routes.credentials.get_azure_client_secret_credential": "vault_provider.read",- "skyvern.forge.sdk.routes.credentials.get_bitwarden_credential": "vault_provider.read",+ "skyvern.forge.sdk.routes.credentials.get_bitwarden_credential": "vault_provider.status","skyvern.forge.sdk.routes.credentials.get_credential": "credential.read","skyvern.forge.sdk.routes.credentials.get_credential_folder": "credential_folder.read","skyvern.forge.sdk.routes.credentials.get_credential_folders": "credential_folder.list",@@ -139,7 +139,7 @@ ROUTE_ACTIONS: Mapping[str, str] = {"skyvern.forge.sdk.routes.credentials.get_credentials": "credential.list","skyvern.forge.sdk.routes.credentials.get_custom_credential_service_config": "vault_provider.read","skyvern.forge.sdk.routes.credentials.get_onepassword_token": "vault_provider.read",- "skyvern.forge.sdk.routes.credentials.get_onepassword_token_status": "vault_provider.read",+ "skyvern.forge.sdk.routes.credentials.get_onepassword_token_status": "vault_provider.status","skyvern.forge.sdk.routes.credentials.get_test_credential_status": "credential.list","skyvern.forge.sdk.routes.credentials.get_totp_codes": "credential.list","skyvern.forge.sdk.routes.credentials.list_bitwarden_items": "vault_provider.list",@@ -261,6 +261,9 @@ ROUTE_ACTIONS: Mapping[str, str] = {"skyvern.forge.sdk.routes.workflow_copilot.workflow_copilot_credential_deletion": "credential.delete","skyvern.forge.sdk.routes.workflow_copilot.workflow_copilot_credential_generate": "credential.create","skyvern.forge.sdk.routes.workflow_copilot.workflow_copilot_credential_response": "workflow_copilot.manage",+ "skyvern.forge.sdk.routes.workflow_copilot.workflow_copilot_demonstration_active": "workflow_copilot.manage",+ "skyvern.forge.sdk.routes.workflow_copilot.workflow_copilot_demonstration_control": "workflow_copilot.manage",+ "skyvern.forge.sdk.routes.workflow_copilot.workflow_copilot_demonstration_record": "workflow_copilot.manage","skyvern.forge.sdk.routes.workflow_copilot.workflow_copilot_disable_auto_accept": "workflow_copilot.manage","skyvern.forge.sdk.routes.workflow_copilot.workflow_copilot_message_feedback": "workflow_copilot.manage","skyvern.forge.sdk.routes.workflow_copilot.workflow_copilot_question_response": "workflow_copilot.manage",diff --git a/skyvern/forge/sdk/trace/base.py b/skyvern/forge/sdk/trace/base.pydeleted file mode 100644index 0af8f8a02..000000000--- a/skyvern/forge/sdk/trace/base.py+++ /dev/null@@ -1,53 +0,0 @@-from abc import ABC, abstractmethod-from typing import Any, Awaitable, Callable, ParamSpec, TypeVar--P = ParamSpec("P")-R = TypeVar("R")---class BaseTrace(ABC):- @abstractmethod- def traced(- self,- name: str | None = None,- metadata: dict[str, Any] | None = None,- tags: list[str] | None = None,- **kwargs: Any,- ) -> Callable[[Callable[P, R]], Callable[P, R]]:- pass-- @abstractmethod- def traced_async(- self,- name: str | None = None,- metadata: dict[str, Any] | None = None,- tags: list[str] | None = None,- **kwargs: Any,- ) -> Callable[[Callable[P, Awaitable[R]]], Callable[P, Awaitable[R]]]:- pass-- def add_task_completion_tag(self, status: str) -> None:- """Add a completion tag to the current trace based on task/workflow status."""-- def add_experiment_metadata(self, experiment_data: dict[str, Any]) -> None:- """Add experiment metadata to the current trace."""---class NoOpTrace(BaseTrace):- def traced(- self,- name: str | None = None,- metadata: dict[str, Any] | None = None,- tags: list[str] | None = None,- **kwargs: Any,- ) -> Callable[[Callable[P, R]], Callable[P, R]]:- return lambda func: func-- def traced_async(- self,- name: str | None = None,- metadata: dict[str, Any] | None = None,- tags: list[str] | None = None,- **kwargs: Any,- ) -> Callable[[Callable[P, Awaitable[R]]], Callable[P, Awaitable[R]]]:- return lambda func: funcdiff --git a/skyvern/forge/sdk/trace/experiment_utils.py b/skyvern/forge/sdk/trace/experiment_utils.pydeleted file mode 100644index 2f03786ef..000000000--- a/skyvern/forge/sdk/trace/experiment_utils.py+++ /dev/null@@ -1,98 +0,0 @@-"""Utilities for collecting and formatting experiment data for tracing."""--from typing import TYPE_CHECKING, Any--import structlog--from skyvern.forge.sdk.core import skyvern_context--if TYPE_CHECKING:- from skyvern.forge.sdk.experimentation.providers import BaseExperimentationProvider--LOG = structlog.get_logger()---async def collect_experiment_metadata_safely(- experimentation_provider: "BaseExperimentationProvider",-) -> dict[str, Any]:- """- Safely collect experiment-related metadata from the current context.-- This is a safe wrapper around collect_experiment_metadata() that ensures- any exceptions are caught and handled gracefully.-- Args:- experimentation_provider: The experimentation provider to use for fetching experiment data.-- Returns:- Dictionary containing experiment data, or empty dict if collection fails.- """- try:- return await collect_experiment_metadata(experimentation_provider)- except Exception:- LOG.warning("Failed to collect experiment metadata", exc_info=True)- return {}---async def collect_experiment_metadata(- experimentation_provider: "BaseExperimentationProvider",-) -> dict[str, Any]:- """- Collect experiment-related metadata from the current context.-- Args:- experimentation_provider: The experimentation provider to use for fetching experiment data.-- Returns:- Dictionary containing experiment data that can be added to traces.- """- # Get the current context- context = skyvern_context.current()- if not context or not context.run_id:- return {}-- # Use run_id as the distinct_id for experiments- distinct_id = context.run_id- organization_id = context.organization_id-- if not distinct_id or not organization_id:- return {}-- experiment_metadata: dict[str, Any] = {}-- try:- # Only collect critical experiment flags that are relevant for tracing- experiment_flags = [- "LLM_NAME",- "LLM_SECONDARY_NAME",- # Add more experiment flags as needed- "PROMPT_CACHING_OPTIMIZATION",- "THINKING_BUDGET_OPTIMIZATION",- "GEMINI_3_REASONING_EFFORT",- ]-- for flag in experiment_flags:- try:- # Get the experiment value (already cached by experimentation provider)- value = await experimentation_provider.get_value_cached(- flag, distinct_id, properties={"organization_id": organization_id}- )-- # Get the payload if available (already cached by experimentation provider)- payload = await experimentation_provider.get_payload_cached(- flag, distinct_id, properties={"organization_id": organization_id}- )-- # Only include if we have actual experiment data- if value is not None or payload is not None:- experiment_metadata[f"experiment_{flag}"] = {"value": value, "payload": payload}-- except Exception:- # Silently skip failed experiments- continue-- except Exception:- # Silently fail if experimentation provider is not available- pass-- return experiment_metadatadiff --git a/skyvern/forge/sdk/trace/lmnr.py b/skyvern/forge/sdk/trace/lmnr.pydeleted file mode 100644index 17f21962d..000000000--- a/skyvern/forge/sdk/trace/lmnr.py+++ /dev/null@@ -1,65 +0,0 @@-from typing import Any, Awaitable, Callable, ParamSpec, TypeVar--import litellm-from lmnr import Instruments, Laminar, LaminarLiteLLMCallback, observe--from skyvern.forge.sdk.trace.base import BaseTrace--P = ParamSpec("P")-R = TypeVar("R")---class LaminarTrace(BaseTrace):- def __init__(self, api_key: str) -> None:- Laminar.initialize(project_api_key=api_key, disabled_instruments={Instruments.SKYVERN, Instruments.PATCHRIGHT})- litellm.callbacks.append(LaminarLiteLLMCallback())-- def traced(- self,- name: str | None = None,- metadata: dict[str, Any] | None = None,- tags: list[str] | None = None,- **kwargs: Any,- ) -> Callable[[Callable[P, R]], Callable[P, R]]:- return observe(name=name, ignore_output=True, metadata=metadata, tags=tags, **kwargs)-- def traced_async(- self,- name: str | None = None,- metadata: dict[str, Any] | None = None,- tags: list[str] | None = None,- **kwargs: Any,- ) -> Callable[[Callable[P, Awaitable[R]]], Callable[P, Awaitable[R]]]:- return observe(name=name, ignore_output=True, metadata=metadata, tags=tags, **kwargs)-- def add_task_completion_tag(self, status: str) -> None:- """Add a completion tag to the current trace based on task/workflow status."""- try:- # Get the current trace ID- trace_id = Laminar.get_trace_id()- if trace_id is None:- return-- # Map status to appropriate tag- status_tag_map = {- "completed": "COMPLETED",- "failed": "FAILURE",- "timed_out": "TIMEOUT",- "canceled": "CANCELED",- "terminated": "TERMINATED",- }-- tag = status_tag_map.get(status, "FAILURE")- Laminar.set_span_tags([tag])- except Exception:- # Silently fail if tracing is not available or there's an error- pass-- def add_experiment_metadata(self, experiment_data: dict[str, Any]) -> None:- """Add experiment metadata to the current trace."""- try:- # Add experiment metadata to the current trace- Laminar.set_trace_metadata(experiment_data)- except Exception:- # Silently fail if tracing is not available or there's an error- passdiff --git a/skyvern/forge/sdk/workflow/models/block.py b/skyvern/forge/sdk/workflow/models/block.pyindex 699eb816c..63bbd5177 100644--- a/skyvern/forge/sdk/workflow/models/block.py+++ b/skyvern/forge/sdk/workflow/models/block.py@@ -1220,22 +1220,7 @@ class Block(BaseModel, abc.ABC):# Every arm that reports a block failure lands here -- the raise path and the ones that# return an unsuccessful result -- so this is where the reason is scrubbed. It is persisted# to workflow_run_blocks.failure_reason and lifted onto the run, neither redacted downstream.- if failure_reason:- # Best-effort: this builder runs for every block result, including on paths that never- # start a ForgeApp, and a secret lookup must never be what turns a block result into a- # failure. Degrades to the unredacted reason, which is the pre-existing behaviour.- try:- context = skyvern_context.current()- block_run_id = context.workflow_run_id if context is not None else None- block_run_secrets = (- app.WORKFLOW_CONTEXT_MANAGER.get_secret_values_for_run(block_run_id)- if app.WORKFLOW_CONTEXT_MANAGER.artifact_redaction_enabled(block_run_id)- else app.WORKFLOW_CONTEXT_MANAGER.runtime_secret_values_for_artifacts()- )- if block_run_secrets:- failure_reason = redact_secrets_from_text(failure_reason, block_run_secrets)- except Exception:- LOG.warning("failed to redact a block failure reason", exc_info=True)+ failure_reason = _redact_run_secrets(failure_reason)# TODO: update workflow run block status and failure reasonif isinstance(output_parameter_value, str):@@ -5076,11 +5061,12 @@ class _StagedFrame(NamedTuple):@dataclassclass _HealAttemptTrace:- """Carries the recovery task id and pinned credential key back to the episode writer, which runs- after the attempt returns."""+ """Carries the recovery task id, pinned credential key and the recovery's own failure sentence back to+ the episode writer, which runs after the attempt returns."""escalation_task_id: str | None = Nonecredential_parameter_key: str | None = None+ recovery_failure_reason: str | None = NoneCODE_BLOCK_GENERIC_FAILURE_REASON = "Failed to execute code block."@@ -5403,6 +5389,26 @@ def _context_navigation_output(return {"associated_navigation": associated_navigation_output(context, workflow_run_context, parameters)}+def _redact_run_secrets(text: str | None) -> str | None:+ if not text:+ return text+ # Best-effort: every block result passes through here, including on paths that never start a ForgeApp,+ # and a secret lookup must never be what turns a block result into a failure. Degrades to the+ # unredacted text, which is the pre-existing behaviour.+ try:+ context = skyvern_context.current()+ run_id = context.workflow_run_id if context is not None else None+ run_secrets = (+ app.WORKFLOW_CONTEXT_MANAGER.get_secret_values_for_run(run_id)+ if app.WORKFLOW_CONTEXT_MANAGER.artifact_redaction_enabled(run_id)+ else app.WORKFLOW_CONTEXT_MANAGER.runtime_secret_values_for_artifacts()+ )+ return redact_secrets_from_text(text, run_secrets) if run_secrets else text+ except Exception:+ LOG.warning("failed to redact a block failure reason", exc_info=True)+ return text++def _redact_codeblock_failure_text(value: str | None,parameters: dict[str, Any],@@ -7998,11 +8004,21 @@ async def wrapper({default_args}):trace: _HealAttemptTrace | None = None,failed_nav_error_code: str | None = None,authored_code: str | None,+ code_failure_reason: str | None,) -> BlockResult | None:"""Run one bounded agent mini-run on the same workflow-run browser to finish the block's goal(narrowed to the failing step when one is confidently matched). Returns a BlockResult when aheal was attempted, or None to fall through to the caller's fail-closed path."""resolved_redaction_parameters = redaction_parameters or {}++ def block_failure_reason(recovery_reason: str | None) -> str | None:+ if trace is not None:+ trace.recovery_failure_reason = recovery_reason+ if not code_failure_reason:+ return recovery_reason+ code_reason = workflow_run_context.mask_secrets_in_data(code_failure_reason)+ return f"{code_reason}\nAI fallback: {recovery_reason}" if recovery_reason else code_reason+if not await self._ai_fallback_enabled(workflow_run_context):return Noneeffective_classification = classification@@ -8252,9 +8268,11 @@ async def wrapper({default_args}):)return await self.build_block_result(success=False,- failure_reason=_redact_codeblock_failure_text(- f"Self-heal escalation did not reach a final status for block {self.label}",- resolved_redaction_parameters,+ failure_reason=block_failure_reason(+ _redact_codeblock_failure_text(+ f"Self-heal escalation did not reach a final status for block {self.label}",+ resolved_redaction_parameters,+ )),output_parameter_value=None,status=BlockStatus.failed,@@ -8286,7 +8304,7 @@ async def wrapper({default_args}):)return await self.build_block_result(success=False,- failure_reason=unsatisfied_reason,+ failure_reason=block_failure_reason(unsatisfied_reason),output_parameter_value=None,status=BlockStatus.failed,workflow_run_block_id=workflow_run_block_id,@@ -8314,9 +8332,13 @@ async def wrapper({default_args}):organization_id=organization_id,)recovery_status = TASK_TO_BLOCK_STATUS.get(updated_task.status, BlockStatus.failed)- recovery_failure_reason = _redact_codeblock_failure_text(- updated_task.failure_reason or f"Self-heal escalation finished with status {updated_task.status}",- resolved_redaction_parameters,+ # Redacted here, not only by build_block_result, because the child row and the heal episode+ # also persist this sentence.+ recovery_failure_reason = _redact_run_secrets(+ _redact_codeblock_failure_text(+ updated_task.failure_reason or f"Self-heal escalation finished with status {updated_task.status}",+ resolved_redaction_parameters,+ ))# Driver codes ride behind the recovery's own codes because its sentence cannot name them. The# recovery block repeats this failure_reason and is newer, so readers attribute codes from it.@@ -8329,7 +8351,7 @@ async def wrapper({default_args}):)return await self.build_block_result(success=False,- failure_reason=recovery_failure_reason,+ failure_reason=block_failure_reason(recovery_failure_reason),output_parameter_value=None,status=recovery_status,workflow_run_block_id=workflow_run_block_id,@@ -8933,6 +8955,7 @@ async def wrapper({default_args}):# Read by the caller before this runs, since AI fallback can navigate away from the failing tab.sign_in_form_visible: bool = False,authored_code: str | None,+ code_failure_reason: str | None,) -> BlockResult:resolved_redaction_parameters = redaction_parameters or {}staged_frame: _StagedFrame | None = None@@ -9186,6 +9209,7 @@ async def wrapper({default_args}):trace=trace,failed_nav_error_code=failed_nav_error_code,authored_code=authored_code,+ code_failure_reason=code_failure_reason,)recovery_wall_clock_ms = int((monotonic() - recovery_started_at) * 1000)await self._write_heal_episode_safe(@@ -9203,7 +9227,9 @@ async def wrapper({default_args}):failing_line=failing_line,matched_step_index=matched_step_index,failure_message=scrub_failure_value(- recovery_result.failure_reason if recovery_result is not None else "floor_no_result"+ "floor_no_result"+ if recovery_result is None+ else trace.recovery_failure_reason or recovery_result.failure_reason),wall_clock_ms=recovery_wall_clock_ms,action_count=await _recovery_action_count(trace.escalation_task_id),@@ -9722,6 +9748,12 @@ async def wrapper({default_args}):if recovery_nav_code and is_registered_secret(recovery_nav_code, workflow_run_context):recovery_nav_code = None+ # The runner already scrubbed this text against the same serialized+ # parameters and guarded the fail-closed case, so re-scrubbing can only+ # no-op or empty it; fall back to what the runner reported.+ runner_reason = secure_failure.failure_reason or "Code block failed in the secure runner"+ secure_failure_reason = scrub_failure_reason(runner_reason, fallback=runner_reason) or ""+async def build_secure_failure_result() -> BlockResult:engine_block_result = secure_code_block_result.block_resultif engine_block_result is not None:@@ -9746,11 +9778,6 @@ async def wrapper({default_args}):organization_id=organization_id,)return engine_block_result- # The runner already scrubbed this text against the same serialized- # parameters and guarded the fail-closed case, so re-scrubbing can only- # no-op or empty it; fall back to what the runner reported.- runner_reason = secure_failure.failure_reason or "Code block failed in the secure runner"- secure_failure_reason = scrub_failure_reason(runner_reason, fallback=runner_reason) or ""secure_error_code = scrub_failure_reason(secure_failure.error_code, fallback="")secure_error_codes = [secure_error_code] if secure_error_code else []if secure_failure.nav_error_code and not is_registered_secret(@@ -9798,6 +9825,7 @@ async def wrapper({default_args}):secure_sign_in_form_visible = await page_shows_sign_in_form(page)return await self._resolve_failure_with_heal(authored_code=authored_code,+ code_failure_reason=secure_failure_reason,exception=None,failing_line=secure_failure.failing_line,build_failure_result=build_secure_failure_result,@@ -10186,6 +10214,7 @@ async def wrapper({default_args}):return await self._resolve_failure_with_heal(authored_code=authored_code,+ code_failure_reason=None,exception=e,failing_line=failing_line,build_failure_result=build_declared_failure_result,@@ -10381,6 +10410,7 @@ async def wrapper({default_args}):return await self._resolve_failure_with_heal(authored_code=authored_code,+ code_failure_reason=failure_reason,exception=e,failing_line=failing_line,build_failure_result=build_legacy_failure_result,diff --git a/skyvern/forge/taskv3/engine.py b/skyvern/forge/taskv3/engine.pyindex 22efa4e18..551cb6b1b 100644--- a/skyvern/forge/taskv3/engine.py+++ b/skyvern/forge/taskv3/engine.py@@ -326,6 +326,7 @@ async def run_task_v3_agent_loop(block_completion_judge: GoalJudge | None = None,# Appended to the goal, whose Code outline section is last, only as far as the request has room for them.code_typed_values: tuple[CodeTypedValue, ...] = (),+ anthropic_cache_markers: bool = False,) -> LoopOutcome:"""Run one Task V3 task to completion against `page`, returning the loop outcome.@@ -728,6 +729,7 @@ async def run_task_v3_agent_loop(target_action_reserve=target_action_reserve,block_completion_check=_block_completion_check if block_check_on else None,unseen_text=unseen_text,+ anthropic_cache_markers=anthropic_cache_markers,)finally:# The context outlives this run; a signal raised as the loop was cancelled must not firediff --git a/skyvern/forge/taskv3/loop.py b/skyvern/forge/taskv3/loop.pyindex 17178b276..65b550be4 100644--- a/skyvern/forge/taskv3/loop.py+++ b/skyvern/forge/taskv3/loop.py@@ -736,6 +736,13 @@ def _extract_text(response: Any) -> str | None:return _get(message, "content")+def _extract_finish_reason(response: Any) -> str | None:+ choices = _get(response, "choices") or []+ if not choices:+ return None+ return _get(choices[0], "finish_reason")++def _extract_reasoning_summary(response: Any) -> str | None:"""The provider's reasoning summary (message.reasoning_content), when litellm's chat->responsesbridge returned one -- surfaces the provider's reasoning summary where available; a tool call@@ -3159,9 +3166,11 @@ class _PerceptionStore:self.entries[index] = entryreturn content- def compact(self, messages: list[dict[str, Any]]) -> None:- """Run before every LLM call. The unread round (after the last assistant message) is never elided- and never charged: one batched turn must not evict every earlier read."""+ def compact(self, messages: list[dict[str, Any]]) -> int | None:+ """Run before every LLM call; returns the lowest message index it rewrote. The unread round (after the+ last assistant message) is never elided and never charged: one batched turn must not evict every+ earlier read."""+ before = [m.get("content") for m in messages]last_assistant_idx = -1for i in range(len(messages) - 1, -1, -1):if messages[i].get("role") == "assistant":@@ -3199,6 +3208,7 @@ class _PerceptionStore:charged -= len(messages[i]["content"])self._elide(messages, i)self.retained_chars_peak = max(self.retained_chars_peak, charged)+ return next((i for i, content in enumerate(before) if messages[i].get("content") != content), None)def _floor(self, read: list[int], newest: dict[tuple[str, str, str], int]) -> set[int]:"""The newest reads the model has seen, one per key, that an epoch never elides."""@@ -3478,6 +3488,7 @@ async def run_agent_tool_loop(# frame cannot be read: evidence the action did something, for the step-cap completion offer.action_effect_fingerprint: Callable[[], Awaitable[str | None]] | None = None,unseen_text: UnseenPageText | None = None,+ anthropic_cache_markers: bool = False,) -> LoopOutcome:tool_by_name = {tool.name: tool for tool in tools}st = LoopState(@@ -3821,23 +3832,25 @@ async def run_agent_tool_loop(active_call_kwargs["caller_owns_exhaustion_receipt"] = Truedef _degrade_call_params(exc: BaseException) -> bool:- """Drop the optional reasoning-summary dict and report whether the turn is worth re-issuing.+ """Drop one optional call param (the reasoning-summary dict, then the Claude cache markers for the rest+ of the run) and report whether the turn is worth re-issuing.Called only when the turn is otherwise about to end the run, so the cost is one extra callon a run that was already failing. A context-window overflow is excluded because dropping aparameter provably cannot fix it. Dropping reasoning_effort reverts to the config's ownvalue, so a provider that rejects the dict form cannot end the run on turn 1."""+ nonlocal anthropic_cache_markersif isinstance(exc, SkyvernContextWindowExceededError):return False- if active_call_kwargs.pop("reasoning_effort", None) is not None:- LOG.warning(- "taskv3 loop retrying without optional call param",- dropped="reasoning_effort",- turn=st.turns,- exc_info=True,- )- return True+ for param in ("reasoning_effort", "cache_control_injection_points"):+ if active_call_kwargs.pop(param, None) is not None:+ if param == "cache_control_injection_points":+ anthropic_cache_markers = False+ LOG.warning(+ "taskv3 loop retrying without optional call param", dropped=param, turn=st.turns, exc_info=True+ )+ return Truereturn False# Images produced by an on-demand `look` this turn, to show the model on the NEXT call only. Passed@@ -4080,8 +4093,26 @@ async def run_agent_tool_loop(# Elide superseded perception results before re-sending the transcript, so a perception-heavy# run can't balloon the context to the token backstop (the pre-compaction runaway mode).- st.reads.compact(st.messages)+ rewritten_at = st.reads.compact(st.messages)llm_caller.message_history = list(st.messages)+ if anthropic_cache_markers:+ # Not -1: a look image rides after the transcript for this call only. The mark before the newest+ # observe writes the entry that the call superseding it reads back at the rewrite mark.+ marked = {0, len(st.messages) - 1}+ newest_observe = max(+ (i for i, e in st.reads.entries.items() if e.tool == "observe" and e.shown), default=None+ )+ for read_at in (rewritten_at, newest_observe):+ if read_at is None:+ continue+ # litellm drops a mark placed on an assistant turn with no text.+ mark_at = read_at - 1+ while st.messages[mark_at]["role"] == "assistant":+ mark_at -= 1+ marked.add(mark_at)+ active_call_kwargs["cache_control_injection_points"] = [+ {"location": "message", "index": i} for i in sorted(marked)+ ]# Consume any pending look image into THIS call only, then clear: the image rides one request# and is never appended to `messages`, so the turn after carries zero image blocks.screenshots_for_call = st.pending_screenshots or None@@ -4172,7 +4203,9 @@ async def run_agent_tool_loop(if not tool_calls:st.no_tool_call_turns += 1- LOG.info("taskv3 loop turn produced no tool call", turn=st.turns)+ LOG.info(+ "taskv3 loop turn produced no tool call", turn=st.turns, finish_reason=_extract_finish_reason(response)+ )st.messages.append({"role": "user", "content": NO_TOOL_CALL_NUDGE})continuediff --git a/skyvern/webeye/utils/page.py b/skyvern/webeye/utils/page.pyindex 452743d0b..7a5f07012 100644--- a/skyvern/webeye/utils/page.py+++ b/skyvern/webeye/utils/page.py@@ -1467,6 +1467,20 @@ def _merge_images_by_position(images: list[Image.Image], positions: list[int]) -return merged_img+# The browser driver's per-call stack capture can leave caller frames (and the screenshot bytes they hold) in a+# reference cycle until a full collection. That collection pauses the whole event loop, so rate-limit it.+_last_stitch_gc_at = float("-inf")+++def _collect_after_stitch() -> None:+ global _last_stitch_gc_at+ now = time.monotonic()+ if now - _last_stitch_gc_at < SettingsManager.get_settings().SCREENSHOT_STITCH_GC_MIN_INTERVAL_SECONDS:+ return+ _last_stitch_gc_at = now+ gc.collect()++def _stitch_screenshots_to_png(screenshots: list[bytes], positions: list[int]) -> bytes:images: list[Image.Image] = []merged_img: Image.Image | None = None@@ -1484,12 +1498,9 @@ def _stitch_screenshots_to_png(screenshots: list[bytes], positions: list[int]) -buffer.seek(0)return buffer.read()finally:- # The decoded images, stitched image, and PNG buffer land in reference cycles that- # gen-0 GC defers, leaving ~100 MB/event resident until a full collection; release- # them explicitly then force one, scoped to the multi-viewport stitch that accumulates them._close_screenshot_stitch_resources(images, merged_img, buffer)if len(images) > 1:- gc.collect()+ _collect_after_stitch()def _stitch_partial_frames(partial_frames: list[tuple[bytes, int]]) -> bytes | None:diff --git a/tests/unit/copilot_test_helpers.py b/tests/unit/copilot_test_helpers.pyindex c7c3184da..eadc0691e 100644--- a/tests/unit/copilot_test_helpers.py+++ b/tests/unit/copilot_test_helpers.py@@ -1724,6 +1724,7 @@ class ScriptedModel(Model):self.inputs: list[list[Any]] = []self.served_instructions: str | None = Noneself.served_tools: list[Any] = []+ self.served_tool_names_by_call: list[set[str]] = []async def get_response(self, *args: Any, **kwargs: Any) -> ModelResponse:raise AssertionError("streamed execution required")@@ -1739,6 +1740,7 @@ class ScriptedModel(Model):):self.served_instructions = system_instructionsself.served_tools = tools+ self.served_tool_names_by_call.append({tool.name for tool in tools})self.inputs.append(list(input))output = self.script[len(self.inputs) - 1]if output is None:@@ -1761,8 +1763,9 @@ async def run_scripted_turn(max_turns: int = 10,output_guardrails: list[OutputGuardrail[CopilotContext]] | None = None,saved_items: list[TResponseInputItem] | None = None,+ ctx: CopilotContext | None = None,) -> tuple[RunResultStreaming, CopilotContext, FakeCopilotStream]:- ctx = make_copilot_ctx(+ ctx = ctx or make_copilot_ctx(organization_id="org_test", workflow_permanent_id="wpid_test", workflow_id=None, workflow_yaml=None, stream=None)stream = FakeCopilotStream()diff --git a/tests/unit/force_stub_app.py b/tests/unit/force_stub_app.pyindex 5a26171c4..05db191da 100644--- a/tests/unit/force_stub_app.py+++ b/tests/unit/force_stub_app.py@@ -68,6 +68,8 @@ def create_forge_stub_app() -> ForgeApp:# would otherwise auto-mock it as an (awaitable) AsyncMock, breaking sync callers.fake_app_module.BROWSER_MANAGER.get_for_task = MagicMock(return_value=None)fake_app_module.PERSISTENT_SESSIONS_MANAGER = _LazyNamespace()+ # A synchronous capability check; auto-mocked, its un-awaited coroutine leaks into later tests.+ fake_app_module.PERSISTENT_SESSIONS_MANAGER.supports_evict_and_reconnect = MagicMock(return_value=False)fake_app_module.ARTIFACT_MANAGER = _LazyNamespace()fake_app_module.AGENT_FUNCTION = _LazyNamespace()fake_app_module.AGENT_FUNCTION.is_block_scoped_workflow_run = AsyncMock(return_value=False)diff --git a/tests/unit/forge/sdk/copilot/test_browser_code_tool.py b/tests/unit/forge/sdk/copilot/test_browser_code_tool.pyindex 6f29b6f66..e0719ba7c 100644--- a/tests/unit/forge/sdk/copilot/test_browser_code_tool.py+++ b/tests/unit/forge/sdk/copilot/test_browser_code_tool.py@@ -1,6 +1,8 @@from __future__ import annotationsimport asyncio+import base64+import ioimport jsonfrom collections.abc import AsyncIterator, Callablefrom contextlib import asynccontextmanager@@ -10,7 +12,10 @@ from typing import Anyfrom unittest.mock import AsyncMock, MagicMockimport pytest+from agents.tool_context import ToolContext+from PIL import Imagefrom playwright.async_api import Page+from structlog.testing import capture_logsfrom skyvern.forge import appfrom skyvern.forge.agent_functions import AgentFunction@@ -29,11 +34,18 @@ from skyvern.forge.sdk.copilot.browser_code_contract import (BrowserCodeSessionUnavailableError,)from skyvern.forge.sdk.copilot.context import CopilotContext+from skyvern.forge.sdk.copilot.enforcement import pending_screenshot_messagefrom skyvern.forge.sdk.copilot.request_policy import RequestPolicyfrom skyvern.forge.sdk.copilot.runtime import (CopilotBrowserGenerationRetired,browser_session_recovery,)+from skyvern.forge.sdk.copilot.screenshot_utils import (+ PendingFrameLease,+ ScreenshotActionRelation,+ ScreenshotProvenance,+ resize_screenshot_b64,+)from skyvern.forge.sdk.copilot.secret_scrub import clear_session_scrub_values, register_secret_scrub_valuefrom skyvern.forge.sdk.copilot.tools import (_build_skyvern_mcp_overlays,@@ -135,6 +147,7 @@ class _LeaseProbeSession:self._denials_sink = denials_sinkself.session_id = "bcs_probe"self.page = SimpleNamespace(url="https://example.com/")+ self.confirmed_page: SimpleNamespace | None = Noneself.closed = Falseasync def rebind(self, page: Page) -> None:@@ -437,6 +450,7 @@ async def test_a_replaced_browser_session_reopens_the_interpreter_rather_than_redef __init__(self) -> None:self.session_id = "bcs_probe"self.page = SimpleNamespace(url="https://example.com/")+ self.confirmed_page = Noneself.closed = Falseasync def rebind(self, page: Page) -> None:@@ -1246,7 +1260,9 @@ async def test_run_browser_code_preserves_opaque_source_reference_during_secret_"get_page",AsyncMock(return_value=(SimpleNamespace(page=object()), None)),)- monkeypatch.setattr(browser_code_module, "_session_for_page", AsyncMock(return_value=object()))+ monkeypatch.setattr(+ browser_code_module, "_session_for_page", AsyncMock(return_value=SimpleNamespace(confirmed_page=None))+ )monkeypatch.setattr(browser_code_module,"_run_cell",@@ -1877,3 +1893,389 @@ async def test_a_browser_code_result_the_scrub_cannot_redact_is_withheld_and_carreported = ["page_request_failures"] if failures_redactable else []assert list(result) == ["page_state", *reported, "ok", "error", "data"]assert logged_request_failures(log) == ([] if failures_redactable else [failure])+++def _frame_png() -> bytes:+ buffer = io.BytesIO()+ Image.new("RGB", (8, 6), "white").save(buffer, format="PNG")+ return buffer.getvalue()+++def _ended_cell(*, ok: bool, url: str, session_alive: bool = True) -> BrowserCodeCellResult:+ return replace(_cell(ok=ok), current_url=url, session_alive=session_alive)+++def _ended_on(url: str) -> SimpleNamespace:+ return SimpleNamespace(url=url, screenshot=AsyncMock(return_value=_frame_png()))+++class _FramedSession(_LeaseProbeSession):+ def __init__(+ self,+ cells: list[BrowserCodeCellResult],+ ended_on: list[SimpleNamespace | None],+ during_cell: Callable[[], None] = lambda: None,+ ) -> None:+ super().__init__()+ self._cells = cells+ self._ended_on = ended_on+ self._during_cell = during_cell++ async def run_cell(self, code: str, *, timeout_seconds: float, deny_pixels: bool = False) -> BrowserCodeCellResult:+ self._during_cell()+ self.confirmed_page = self._ended_on.pop(0)+ return self._cells.pop(0)+++def _patch_framed_runtime(+ monkeypatch: pytest.MonkeyPatch,+ session: _FramedSession,+ *,+ retire_lease: int | None = None,+ replace_browser_lease: int | None = None,+) -> None:+ leases = 0+ browser_replaced = False++ @asynccontextmanager+ async def lease(_ctx: object, **_kwargs: object) -> AsyncIterator[None]:+ nonlocal leases, browser_replaced+ leases += 1+ if leases == retire_lease:+ raise CopilotBrowserGenerationRetired("pbs_1", BrowserRetirementReason.session_ending)+ browser_replaced = browser_replaced or leases == replace_browser_lease+ yield++ async def current_page(session_id: str | None = None) -> tuple[SimpleNamespace, None]:+ return SimpleNamespace(page=session.page), None++ async def admitted_browser(session_id: str | None = None) -> tuple[SimpleNamespace, None]:+ return SimpleNamespace(pages=[] if browser_replaced else [session.confirmed_page]), None++ async def open_session(**_kwargs: object) -> BrowserCodeSession:+ return session++ async def prepared(*_args: object, **_kwargs: object) -> tuple[None, None, None]:+ return None, None, None++ monkeypatch.setattr(browser_code_module, "mcp_browser_context", lease)+ monkeypatch.setattr(browser_code_module, "get_page", current_page)+ monkeypatch.setattr(browser_code_module, "resolve_browser", admitted_browser)+ monkeypatch.setattr(browser_code_module, "_prepare_browser_session_for_dispatch", prepared)+ monkeypatch.setattr(app.AGENT_FUNCTION, "open_copilot_browser_code_session", open_session)+++def _stale_frame(ctx: CopilotContext) -> None:+ ctx.pending_screenshots.append(+ resize_screenshot_b64(+ base64.b64encode(_frame_png()).decode("ascii"),+ provenance=ScreenshotProvenance.unknown(source_tool="get_browser_screenshot"),+ captured_at=0.0,+ )+ )+++ ("ok", "session_alive"),+ [(True, True), (False, True), (False, False)],+ ids=["ok-cell", "failed-cell", "timed-out-cell"],+)+async def test_a_finished_cell_replaces_the_stale_frame_with_one_of_the_page_it_ended_on(+ monkeypatch: pytest.MonkeyPatch, ok: bool, session_alive: bool+) -> None:+ ended_on = _ended_on("https://fixture.test/checkout")+ session = _FramedSession(+ [_ended_cell(ok=ok, url="https://fixture.test/checkout", session_alive=session_alive)], [ended_on]+ )+ _patch_framed_runtime(monkeypatch, session)+ ctx = make_copilot_context()+ ctx.browser_session_id = "pbs_1"+ _stale_frame(ctx)++ result = await browser_code_module.run_browser_code(ctx, "1")++ assert result["ok"] is ok+ assert len(ctx.pending_screenshots) == 1+ provenance = ctx.pending_screenshots[0].provenance+ assert provenance.source_tool == "run_browser_code"+ assert provenance.action_relation is ScreenshotActionRelation.AFTER_SOURCE_ACTION+ assert provenance.dispatch_url == "https://fixture.test/checkout"+ assert provenance.captured_url == "https://fixture.test/checkout"+++async def test_a_cell_frame_scrubs_its_urls_like_the_cell_result(monkeypatch: pytest.MonkeyPatch) -> None:+ secret = "tok-5f1e9a"+ url = f"https://fixture.test/callback?code={secret}"+ _patch_framed_runtime(monkeypatch, _FramedSession([_ended_cell(ok=True, url=url)], [_ended_on(url)]))+ ctx = make_copilot_context()+ ctx.browser_session_id = "pbs_1"+ register_secret_scrub_value(ctx, secret)+ try:+ result = await browser_code_module.run_browser_code(ctx, "1")+ finally:+ clear_session_scrub_values(ctx.browser_session_id)++ provenance = ctx.pending_screenshots[0].provenance+ assert provenance.dispatch_url == provenance.captured_url == result["current_url"]+ assert secret not in str(pending_screenshot_message(ctx))++[email protected]("withheld_by", ["secret-run", "self-heal", "non-vision", "unconfirmed-tab"])+async def test_a_cell_frame_is_withheld_by_the_trust_floor_or_an_unconfirmed_page(+ monkeypatch: pytest.MonkeyPatch, withheld_by: str+) -> None:+ ended_on = _ended_on("https://fixture.test/account")+ cell = _ended_cell(ok=True, url="https://fixture.test/account")+ session = _FramedSession([cell], [None if withheld_by == "unconfirmed-tab" else ended_on])+ _patch_framed_runtime(monkeypatch, session)+ ctx = make_copilot_context()+ ctx.browser_session_id = "pbs_1"+ if withheld_by == "secret-run":+ ctx.secret_run_browser_session_ids = {"pbs_1"}+ elif withheld_by == "self-heal":+ ctx.codeblock_redaction_parameters = {"password": "hunter2"}+ elif withheld_by == "non-vision":+ ctx.supports_vision = False++ await browser_code_module.run_browser_code(ctx, "1")++ assert ctx.pending_screenshots == []+ ended_on.screenshot.assert_not_awaited()+++async def test_a_secret_that_reaches_the_browser_during_the_cell_withholds_its_frame(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ ctx = make_copilot_context()+ ctx.browser_session_id = "pbs_1"+ ended_on = _ended_on("https://fixture.test/account")+ session = _FramedSession(+ [_ended_cell(ok=True, url="https://fixture.test/account")],+ [ended_on],+ during_cell=lambda: ctx.secret_run_browser_session_ids.add("pbs_1"),+ )+ _patch_framed_runtime(monkeypatch, session)++ await browser_code_module.run_browser_code(ctx, "1")++ assert ctx.pending_screenshots == []+ ended_on.screenshot.assert_not_awaited()+++ ("second_ends_on", "reason"),+ [("unconfirmed-tab", "tab_unconfirmed"), ("failed-screenshot", "screenshot_failed")],+)+async def test_a_cell_that_stages_no_frame_withdraws_the_earlier_cell_frame(+ monkeypatch: pytest.MonkeyPatch, second_ends_on: str, reason: str+) -> None:+ url = "https://fixture.test/listing"+ second = _ended_on(url)+ second.screenshot = AsyncMock(side_effect=RuntimeError("Target closed"))+ session = _FramedSession(+ [_ended_cell(ok=True, url=url), _ended_cell(ok=True, url=url)],+ [_ended_on(url), None if second_ends_on == "unconfirmed-tab" else second],+ )+ _patch_framed_runtime(monkeypatch, session)+ ctx = make_copilot_context()+ ctx.browser_session_id = "pbs_1"++ await browser_code_module.run_browser_code(ctx, "1")+ first_frame = ctx.pending_screenshots[0]+ ctx.pending_frame_lease = PendingFrameLease(+ capture_event_id=first_frame.capture_event_id, capture_id=first_frame.capture_id, input_fingerprint="f"+ )+ with capture_logs() as logs:+ result = await browser_code_module.run_browser_code(ctx, "2")++ assert result["ok"] is True and result["value"] == {"observed": True}+ assert pending_screenshot_message(ctx) is None+ assert ctx.pending_frame_lease is None+ [line] = [line for line in logs if line["event"] == "copilot_browser_code_cell_frame"]+ assert (line["reason"], line["withdrawn_capture_id"]) == (reason, first_frame.capture_id)+++async def test_a_frame_captured_after_the_cell_started_is_kept_when_the_cell_stages_no_frame(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ ctx = make_copilot_context()+ ctx.browser_session_id = "pbs_1"+ _stale_frame(ctx)++ def hook_frame_during_cell() -> None:+ ctx.pending_screenshots.clear()+ ctx.pending_screenshots.append(+ resize_screenshot_b64(+ base64.b64encode(_frame_png()).decode("ascii"),+ provenance=ScreenshotProvenance.unknown(source_tool="click"),+ )+ )++ session = _FramedSession(+ [_ended_cell(ok=True, url="https://fixture.test/x")], [None], during_cell=hook_frame_during_cell+ )+ _patch_framed_runtime(monkeypatch, session)++ await browser_code_module.run_browser_code(ctx, "1")++ assert [entry.provenance.source_tool for entry in ctx.pending_screenshots] == ["click"]+++async def test_a_frame_staged_while_the_cell_waits_for_the_browser_is_withdrawn_when_the_cell_stages_none(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ _patch_framed_runtime(monkeypatch, _FramedSession([_ended_cell(ok=True, url="https://fixture.test/x")], [None]))+ waiting = asyncio.Event()++ async def prepared(*_args: object, **_kwargs: object) -> tuple[None, None, None]:+ waiting.set()+ return None, None, None++ monkeypatch.setattr(browser_code_module, "_prepare_browser_session_for_dispatch", prepared)+ ctx = make_copilot_context()+ ctx.browser_session_id = "pbs_1"++ async with ctx.credential_fill_lock:+ call = asyncio.create_task(browser_code_module.run_browser_code(ctx, "1"))+ await waiting.wait()+ sibling = resize_screenshot_b64(+ base64.b64encode(_frame_png()).decode("ascii"),+ provenance=ScreenshotProvenance.unknown(source_tool="click"),+ )+ ctx.pending_screenshots.append(sibling)+ ctx.pending_frame_lease = PendingFrameLease(+ capture_event_id=sibling.capture_event_id, capture_id=sibling.capture_id, input_fingerprint="f"+ )+ await call++ assert pending_screenshot_message(ctx) is None+ assert ctx.pending_frame_lease is None++[email protected]("cancelled_in", ["frame-capture", "ended-session-close"])+async def test_a_cancel_in_the_cell_frame_or_session_close_reports_the_finished_cell_on_the_next_call(+ monkeypatch: pytest.MonkeyPatch, cancelled_in: str+) -> None:+ first_url = "https://fixture.test/first"+ first_ends_on = _ended_on(first_url)+ first_cell = replace(+ _ended_cell(ok=False, url=first_url, session_alive=cancelled_in == "frame-capture"),+ actions=(BrowserCodeOperation(operation="click", selector="#pay", succeeded=True),),+ )+ if cancelled_in == "frame-capture":+ first_ends_on.screenshot = AsyncMock(side_effect=asyncio.CancelledError)+ session = _FramedSession(+ [first_cell, _ended_cell(ok=True, url="https://fixture.test/2")],+ [first_ends_on, _ended_on("https://fixture.test/2")],+ )+ if cancelled_in == "ended-session-close":+ monkeypatch.setattr(session, "close", AsyncMock(side_effect=[asyncio.CancelledError, None]))+ _patch_framed_runtime(monkeypatch, session)+ ctx = make_copilot_context()+ ctx.browser_session_id = "pbs_1"+ _stale_frame(ctx)++ with pytest.raises(asyncio.CancelledError):+ await browser_code_module.run_browser_code(ctx, "1")+ expected_frames = [] if cancelled_in == "frame-capture" else [first_url]+ assert [entry.provenance.dispatch_url for entry in ctx.pending_screenshots] == expected_frames+ assert [(item["tool_name"], item["selector"]) for item in ctx.scouted_interactions] == [+ ("run_browser_code.click", "#pay")+ ]+ result = await browser_code_module.run_browser_code(ctx, "2")++ completed = result["previous_call_completed"]+ assert (completed["ok"], completed["error_code"], completed["current_url"]) == (False, "user_code_error", first_url)+ assert completed["operations"] == []+ assert "previous_call_interrupted" not in result+++async def test_a_cell_frame_stages_under_its_tool_call_on_the_add_mode_surface(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ surface = resolve_copilot_tool_surface(+ mode=None,+ native_tools=copilot_native_tools(+ supports_question_tool=True, browser_code_available=True, run_tools_available=True+ ),+ alias_map=get_skyvern_mcp_alias_map(),+ overlays=_build_skyvern_mcp_overlays(),+ browser_code_mode=CopilotBrowserCodeMode.ADD,+ )+ tool = next(tool for tool in surface.native_tools if tool.name == "run_browser_code")+ ended_on = _ended_on("https://fixture.test/listing")+ _patch_framed_runtime(+ monkeypatch, _FramedSession([_ended_cell(ok=True, url="https://fixture.test/listing")], [ended_on])+ )+ ctx = make_copilot_context()+ ctx.browser_session_id = "pbs_1"+ arguments = json.dumps({"code": "1", "user_facing_reason": "look"})++ await tool.on_invoke_tool(+ ToolContext(context=ctx, tool_name=tool.name, tool_call_id="call_cell", tool_arguments=arguments), arguments+ )++ assert "click" in surface.ordered_mcp_names+ assert [entry.provenance.source_tool for entry in ctx.pending_screenshots] == ["run_browser_code"]+ assert [frame.tool_call_id for frame in ctx.pending_chat_screenshots] == ["call_cell"]+++ "lost_before_capture", [{"retire_lease": 2}, {"replace_browser_lease": 2}], ids=["retired", "replaced"]+)+async def test_a_browser_retired_before_the_cell_frame_returns_the_result_without_a_frame(+ monkeypatch: pytest.MonkeyPatch, lost_before_capture: dict[str, int]+) -> None:+ ended_on = _ended_on("https://fixture.test/done")+ session = _FramedSession([_ended_cell(ok=True, url="https://fixture.test/done")], [ended_on])+ _patch_framed_runtime(monkeypatch, session, **lost_before_capture)+ ctx = make_copilot_context()+ ctx.browser_session_id = "pbs_1"++ result = await browser_code_module.run_browser_code(ctx, "1")++ assert result["ok"] is True and result["value"] == {"observed": True}+ assert ctx.pending_screenshots == []+ ended_on.screenshot.assert_not_awaited()+++ "error",+ [+ BrowserCodeSessionUnavailableError(+ "The code session's relay could not be reached.", error_code="relay_unavailable", session_ended=True+ ),+ CopilotBrowserGenerationRetired("pbs_1", BrowserRetirementReason.session_ending),+ ],+ ids=["relay-unavailable", "browser-retired"],+)+async def test_a_cell_that_loses_its_browser_withdraws_the_frame_from_before_it(+ monkeypatch: pytest.MonkeyPatch, error: Exception+) -> None:+ def lose_browser() -> None:+ raise error++ async def loss_disposition(*_args: object, **_kwargs: object) -> str:+ return "failed"++ _patch_framed_runtime(monkeypatch, _FramedSession([], [], during_cell=lose_browser))+ monkeypatch.setattr(browser_code_module, "_browser_session_error_disposition", loss_disposition)+ ctx = make_copilot_context()+ ctx.browser_session_id = "pbs_1"+ _stale_frame(ctx)++ result = await browser_code_module.run_browser_code(ctx, "1")++ assert result["ok"] is False+ assert pending_screenshot_message(ctx) is Nonediff --git a/tests/unit/test_agent_task_v3.py b/tests/unit/test_agent_task_v3.pyindex c82f61e1e..798685ca5 100644--- a/tests/unit/test_agent_task_v3.py+++ b/tests/unit/test_agent_task_v3.py@@ -90,6 +90,7 @@ from skyvern.forge.taskv3.run_arms import (run_arm_enabled,)from skyvern.forge.taskv3.tools import PageProvider, _record_frame_work+from skyvern.schemas.llm import LLMConfigfrom skyvern.schemas.runs import RunEnginefrom skyvern.schemas.workflows import BlockTypefrom skyvern.utils.secret_redaction import REDACTED_SECRET_PLACEHOLDER@@ -362,6 +363,33 @@ async def test_execute_task_v3_buckets_the_date_segment_aim_arm_per_run(monkeypa)+ ("model_name", "kill_switch", "marks"),+ [+ ("anthropic/claude-haiku-5-5", False, True),+ ("anthropic/claude-haiku-5-5", True, False),+ ("gpt-5.4", False, False),+ ],+)+async def test_execute_task_v3_sends_claude_cache_marks_unless_the_kill_switch_is_on(+ monkeypatch: pytest.MonkeyPatch, model_name: str, kill_switch: bool, marks: bool+) -> None:+ flags = AsyncMock(side_effect=lambda flag, *_a, **_k: kill_switch and flag == "DISABLE_ANTHROPIC_PROMPT_CACHING")+ monkeypatch.setattr(app.EXPERIMENTATION_PROVIDER, "is_feature_enabled_cached", flags)+ caller = MagicMock(llm_config=LLMConfig(model_name, [], True, False))++ _step, _task, loop_mock, _post = await _run_execute_task_v3(+ monkeypatch,+ LoopOutcome(status="completed", reason="done", billable_actions=[]),+ data_extraction_goal=None,+ extracted_information_schema=None,+ llm_caller_factory=MagicMock(return_value=caller),+ )++ assert loop_mock.call_args.kwargs["anthropic_cache_markers"] is marks++@pytest.mark.asyncio@pytest.mark.parametrize("variant, arm", [(None, "unrandomized"), ("control", "control"), ("treatment", "treatment")])async def test_execute_task_v3_holds_logins_only_on_the_login_pace_treatment(@@ -1214,11 +1242,24 @@ async def test_resolve_v3_llm_key_sends_wpid_flag_property(monkeypatch: pytest.M@pytest.mark.asyncio-async def test_resolve_v3_llm_key_ignores_unregistered_flag_value(monkeypatch: pytest.MonkeyPatch) -> None:+ ("variant", "registered"),+ [+ ("TYPO_KEY", {}),+ # LLMCaller sends a non-router ANTHROPIC key down its native Anthropic path, which cannot run a v3 transcript.+ (+ "ANTHROPIC_CLAUDE5_5_HAIKU",+ {"ANTHROPIC_CLAUDE5_5_HAIKU": LLMConfig("anthropic/claude-haiku-5-5", [], True, False)},+ ),+ ],+)+async def test_resolve_v3_llm_key_ignores_unusable_flag_value(+ monkeypatch: pytest.MonkeyPatch, variant: str, registered: dict[str, LLMConfig]+) -> None:monkeypatch.setattr(- "skyvern.forge.agent.app.EXPERIMENTATION_PROVIDER.get_value_cached", AsyncMock(return_value="TYPO_KEY")+ "skyvern.forge.agent.app.EXPERIMENTATION_PROVIDER.get_value_cached", AsyncMock(return_value=variant))- monkeypatch.setattr(agent_module.LLMConfigRegistry, "is_registered", lambda _k: False)+ monkeypatch.setattr(agent_module.LLMConfigRegistry, "_configs", registered)monkeypatch.setattr(agent_module, "is_custom_llm_key", lambda _k: False)monkeypatch.setattr(agent_module.settings, "TASK_V3_LLM_KEY", "")monkeypatch.setattr(agent_module.settings, "LLM_KEY", "DEFAULT_MODEL")diff --git a/tests/unit/test_api_handler_factory.py b/tests/unit/test_api_handler_factory.pyindex 2bcca95a0..c8528d2ab 100644--- a/tests/unit/test_api_handler_factory.py+++ b/tests/unit/test_api_handler_factory.py@@ -15,6 +15,7 @@ import litellm # type: ignore[import-not-found]import openaiimport pytest # type: ignore[import-not-found]import structlog+from litellm.llms.anthropic.chat.transformation import AnthropicConfig # type: ignore[import-not-found]from PIL import Imagefrom skyvern.forge.sdk.api.llm import api_handler_factory@@ -666,6 +667,7 @@ async def test_llm_caller_preserves_generated_cost_and_tokens_during_real_redactaccounting = next(record for record in records if "llm_cost" in record and "model" in record)assert accounting["llm_cost"] == 0.25 and type(accounting["llm_cost"]) is floatassert accounting["input_tokens"] == 7 and type(accounting["input_tokens"]) is int+ assert accounting["cache_creation_tokens"] == 2assert credential not in json.dumps(records)assert REDACTED_SECRET_PLACEHOLDER in accounting["prompt_name"]usage = next(record for record in records if record.get("log_code") == "copilot_model_usage")@@ -784,6 +786,7 @@ async def test_anthropic_call_stats_treats_missing_cache_read_tokens_as_zero("input_tokens": 100,"output_tokens": 10,"cache_read_input_tokens": None,+ "output_tokens_details": {"thinking_tokens": 4},},})@@ -791,9 +794,26 @@ async def test_anthropic_call_stats_treats_missing_cache_read_tokens_as_zero(stats = await caller.get_call_stats(response)assert stats.cached_tokens == 0+ assert stats.reasoning_tokens == 4assert stats.llm_cost == pytest.approx((3.0 * 100 + 15.0 * 10) / 1_000_000)+def test_anthropic_usage_takes_the_api_thinking_count_until_litellm_does() -> None:+ """litellm < 1.99 counts Claude's thinking text, empty on Claude 5.x, instead of the API's thinking_tokens. When the+ upstream assertion fails, litellm reads the count itself and use_anthropic_reported_thinking_tokens can go."""+ usage_object = {"input_tokens": 93, "output_tokens": 215, "output_tokens_details": {"thinking_tokens": 189}}+ config = AnthropicConfig()+ upstream = AnthropicConfig.calculate_usage.__wrapped__(config, usage_object, "") # type: ignore[attr-defined]+ patched = config.calculate_usage(usage_object, "")++ assert not upstream.completion_tokens_details.reasoning_tokens+ assert patched.completion_tokens == 215+ assert (patched.completion_tokens_details.reasoning_tokens, patched.completion_tokens_details.text_tokens) == (+ 189,+ 26,+ )++@pytest.mark.parametrize("pricing_case", ["bundled", "unknown", "lookup_error"])@pytest.mark.asyncioasync def test_anthropic_call_stats_uses_pricing_or_fallback(diff --git a/tests/unit/test_code_block_captcha.py b/tests/unit/test_code_block_captcha.pyindex c2f04de6f..755c38abd 100644--- a/tests/unit/test_code_block_captcha.py+++ b/tests/unit/test_code_block_captcha.py@@ -2257,6 +2257,7 @@ async def test_failure_on_a_sign_in_form_is_recorded_without_changing_any_other_await failure_tab.set_content(failure_tab_html)result = await block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=PlaywrightTimeoutError("timeout"),failing_line=1,build_failure_result=build_failure,diff --git a/tests/unit/test_code_block_heal_engine_classification.py b/tests/unit/test_code_block_heal_engine_classification.pyindex 2d0dec38b..4ce9cfc83 100644--- a/tests/unit/test_code_block_heal_engine_classification.py+++ b/tests/unit/test_code_block_heal_engine_classification.py@@ -166,6 +166,7 @@ async def test_attempt_self_heal_default_classification_uses_pinned_engine(result = await block._attempt_self_heal(authored_code=None,+ code_failure_reason=None,exception=exception,failing_line=None,recording_page=_recording_page(None),diff --git a/tests/unit/test_copilot_agent_helpers.py b/tests/unit/test_copilot_agent_helpers.pyindex 45d7fdcda..bffaf4d99 100644--- a/tests/unit/test_copilot_agent_helpers.py+++ b/tests/unit/test_copilot_agent_helpers.py@@ -98,7 +98,13 @@ from skyvern.forge.sdk.copilot.composition_evidence import (parse_composition_structured,)from skyvern.forge.sdk.copilot.config import BlockAuthoringPolicy, CopilotConfig-from skyvern.forge.sdk.copilot.context import AgentResult, CodeAuthoringRepairContext, CopilotContext+from skyvern.forge.sdk.copilot.context import (+ EDIT_BLOCK_TOOL_NAME,+ UPDATE_AND_RUN_BLOCKS_TOOL_NAME,+ AgentResult,+ CodeAuthoringRepairContext,+ CopilotContext,+)from skyvern.forge.sdk.copilot.diagnosis_repair_contract import (DiagnosisFailureType,DiagnosisInput,@@ -148,10 +154,6 @@ from skyvern.forge.sdk.copilot.runtime_authoring_repair import REPAIR_INSTRUCTIOfrom skyvern.forge.sdk.copilot.tools import _record_run_blocks_result, _run_blocks_and_collect_debugfrom skyvern.forge.sdk.copilot.tools import credentials as credentials_modulefrom skyvern.forge.sdk.copilot.tools import run_execution as run_execution_module-from skyvern.forge.sdk.copilot.tools._shared import (- EDIT_BLOCK_TOOL_NAME,- UPDATE_AND_RUN_BLOCKS_TOOL_NAME,-)from skyvern.forge.sdk.copilot.tools.completion import (_authored_output_contract_criteria,_completion_verification_criteria,@@ -651,6 +653,43 @@ async def test_a_plain_continuation_names_reusable_origin_inputs_before_any_run()+async def test_demonstrate_product_action_triggers_replay(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr(agent_module, "restore_pending_workflow_proposal", AsyncMock(return_value=None))+ monkeypatch.setattr(agent_module, "seed_repair_origin_run", AsyncMock(return_value=SimpleNamespace(finished=True)))+ monkeypatch.setattr(agent_module, "hydrate_prior_run_packet", AsyncMock(return_value=None))+ replay = AsyncMock(return_value=[])+ monkeypatch.setattr(agent_module, "replay_demonstration", replay)+ stub_copilot_agent_loop(+ monkeypatch, AsyncMock(return_value=_fake_run_result({"type": "REPLY", "user_response": "ok"}))+ )++ result = await agent_module.run_copilot_agent(+ stream=MagicMock(),+ organization_id="org-1",+ chat_request=WorkflowCopilotChatRequest(+ message="I've shown you the task in the browser. Build a workflow that does it.",+ workflow_id="wf-1",+ workflow_permanent_id="wfp-1",+ workflow_copilot_chat_id="chat-1",+ workflow_yaml="",+ product_action="demonstrate",+ demonstration_id="demo-1",+ ),+ chat_history=[],+ global_llm_context=None,+ llm_api_handler=SimpleNamespace(llm_key="PRIMARY"),+ raw_secret_safety_handler=AsyncMock(+ return_value={"version": "1", "state": "clean", "handling": "none", "citations": []}+ ),+ api_key="sk-test",+ config=CopilotConfig(),+ )++ assert result.user_response == "ok"+ replay.assert_awaited_once()++class TestFailedTestResponseNormalization:def test_paused_run_reply_is_not_rewritten_into_a_failed_test(self) -> None:from skyvern.forge.sdk.copilot.agent import _rewrite_failed_test_responsediff --git a/tests/unit/test_copilot_ask_user.py b/tests/unit/test_copilot_ask_user.pyindex 151e4210d..0f678ef52 100644--- a/tests/unit/test_copilot_ask_user.py+++ b/tests/unit/test_copilot_ask_user.py@@ -708,7 +708,7 @@ async def test_question_reply_persists_raw_secret_detection(sqlite_engine, monkectx.request_policy = RequestPolicy()monkeypatch.setattr(routes,- "_screen_raw_secret_safety",+ "screen_raw_secret_safety",AsyncMock(return_value=SimpleNamespace(status="detected",@@ -831,7 +831,7 @@ async def test_near_expiry_reply_renews_client_presence_and_rechecks_cancel(assert recorded.status == ("cancelled" if cancel_during_screen else "pending")return SimpleNamespace(status="clean", canonical_user_message=text)- monkeypatch.setattr(routes, "_screen_raw_secret_safety", screen)+ monkeypatch.setattr(routes, "screen_raw_secret_safety", screen)async with client:result = await client.post("/reply",diff --git a/tests/unit/test_copilot_budget_drain.py b/tests/unit/test_copilot_budget_drain.pyindex c40db957b..8d33f27ad 100644--- a/tests/unit/test_copilot_budget_drain.py+++ b/tests/unit/test_copilot_budget_drain.py@@ -12,10 +12,14 @@ import pytestfrom agents import Agent, ModelSettings, RunContextWrapper, SQLiteSession, function_toolfrom agents.exceptions import MaxTurnsExceededfrom agents.run_config import RunConfig+from litellm.exceptions import NotFoundError as LiteLLMNotFoundError+from litellm.exceptions import RateLimitError as LiteLLMRateLimitErrorfrom skyvern.forge import app as forge_app+from skyvern.forge.sdk.copilot import agent as agent_modulefrom skyvern.forge.sdk.copilot import tools as tools_modulefrom skyvern.forge.sdk.copilot.agent import (+ FINAL_REPLY_UNAPPLIED_CHANGE_NOTE,_build_narrative_payload,_build_timeout_exit_result,_format_chat_history,@@ -23,8 +27,11 @@ from skyvern.forge.sdk.copilot.agent import ()from skyvern.forge.sdk.copilot.ask_user import QuestionInteraction, QuestionPart, QuestionResponsefrom skyvern.forge.sdk.copilot.blocker_signal import CopilotToolBlockerSignal-from skyvern.forge.sdk.copilot.context import CopilotContext+from skyvern.forge.sdk.copilot.code_write_diff import CODE_WRITE_TOOL_NAMES+from skyvern.forge.sdk.copilot.config import CopilotConfig+from skyvern.forge.sdk.copilot.context import REPLY_TOOL_NAME, CopilotContextfrom skyvern.forge.sdk.copilot.enforcement import (+ BUDGET_DRAIN_EXCLUDED_TOOLS,BUDGET_DRAIN_HEADROOM,EXPIRED_TURN_REPLY_SECONDS,HARD_BACKSTOP_ALLOWANCE_SECONDS,@@ -43,11 +50,11 @@ from skyvern.forge.sdk.copilot.tools.run_execution import (from skyvern.forge.sdk.schemas.copilot_turn_outcome import ResponseKind, TurnOutcomefrom skyvern.forge.sdk.schemas.workflow_copilot import (WorkflowCopilotChatHistoryMessage,+ WorkflowCopilotChatRequest,WorkflowCopilotChatSender,WorkflowCopilotStreamMessageType,)from skyvern.forge.sdk.workflow.models.workflow import Workflow-from skyvern.utils.yaml_loader import safe_load_no_datesfrom tests.unit.copilot_test_helpers import (HANDBACK_WORKFLOW_YAML,OWNED_RUN_ID,@@ -63,6 +70,7 @@ from tests.unit.copilot_test_helpers import (scripted_call,scripted_text,settle_cancelled_tools,+ stub_copilot_agent_loop,tool_outputs_for,)@@ -119,7 +127,8 @@ async def test_timeout_finishes_active_call_then_drains_once_in_same_session("message": ("The normal turn budget has expired. headroom is the maximum number of model calls available to finish ""this turn, including this call; the hard time limit may stop it sooner. Your session history and any "- "staged draft remain available. Choose how to use these calls. New build-test runs cannot be dispatched."+ "staged draft remain available. Choose how to use these calls. This step offers no tools that change or "+ "run the workflow."),}assert ctx.budget_expiry_state.drain_attempted is True@@ -127,98 +136,6 @@ async def test_timeout_finishes_active_call_then_drains_once_in_same_session(assert ctx.budget_expiry_state.report_produced is None-async def test_drain_denies_new_run_before_dispatch(monkeypatch: pytest.MonkeyPatch) -> None:- ctx = make_copilot_ctx()- ctx.budget_expiry_state.source = "max_turns"- ctx.budget_expiry_state.drain_active = True- database_lookup = AsyncMock()- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.run_execution.app.DATABASE.workflows.get_workflow_by_permanent_id",- database_lookup,- )-- result = await _run_blocks_and_collect_debug({"block_labels": ["step"]}, ctx)-- assert result == {- "ok": False,- "data": {- "budget_expired": True,- "run_dispatched": False,- "source": "max_turns",- },- }- database_lookup.assert_not_awaited()---async def test_composite_tool_stages_edit_before_drain_denies_run(monkeypatch: pytest.MonkeyPatch) -> None:- from skyvern.forge.sdk.routes.workflow_copilot import _process_workflow_yaml-- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.workflow_yaml.app.WORKFLOW_SERVICE.get_workflow_by_permanent_id",- AsyncMock(return_value=None),- )- workflow_yaml = """-title: Drain draft-workflow_definition:- parameters: []- blocks:- - block_type: navigation- label: staged_step- navigation_goal: Open the staged page.-"""- ctx = make_copilot_ctx()- ctx.budget_expiry_state.source = "deadline"- ctx.budget_expiry_state.drain_active = True-- async def fake_update_workflow(payload: dict[str, Any], copilot_ctx: Any, **_kwargs: Any) -> dict[str, Any]:- copilot_ctx.workflow_yaml = payload["workflow_yaml"]- workflow = await _process_workflow_yaml(- settings_fallback_yaml="",- workflow_id=copilot_ctx.workflow_id,- workflow_permanent_id=copilot_ctx.workflow_permanent_id,- organization_id=copilot_ctx.organization_id,- workflow_yaml=payload["workflow_yaml"],- )- copilot_ctx.staged_workflow = workflow- copilot_ctx.staged_workflow_yaml = payload["workflow_yaml"]- copilot_ctx.has_staged_proposal = True- return {"ok": True, "_workflow": workflow, "data": {"block_count": 1}}-- monkeypatch.setattr(tools_module, "_update_and_run_requires_skipped_run", lambda *_args: False)- monkeypatch.setattr(tools_module, "_authority_tool_error", lambda *_args, **_kwargs: None)- monkeypatch.setattr(tools_module, "_get_prior_workflow_definition", AsyncMock(return_value=None))- monkeypatch.setattr(tools_module, "_update_workflow", fake_update_workflow)- monkeypatch.setattr(tools_module, "_frontier_runtime_page_url", AsyncMock(return_value=None))- monkeypatch.setattr(- tools_module,- "_plan_frontier",- lambda *_args: (["staged_step"], {}, "staged_step", "initial"),- )- monkeypatch.setattr(tools_module, "_verify_and_record_run_blocks_result", AsyncMock(return_value=None))- monkeypatch.setattr(tools_module, "_carry_prior_attempt_change_identity_into_result", lambda *_args: None)- monkeypatch.setattr(tools_module, "record_tool_step_result_for_ctx", lambda *_args: None)- monkeypatch.setattr(tools_module, "finalize_build_test_result", lambda *_args, **_kwargs: None)- monkeypatch.setattr(tools_module, "enqueue_screenshot_from_result", lambda *_args, **_kwargs: None)-- result = await tools_module.update_and_run_blocks_tool.on_invoke_tool(- SimpleNamespace(context=ctx, tool_name="update_and_run_blocks"),- json.dumps({"workflow": safe_load_no_dates(workflow_yaml), "block_labels": ["staged_step"], "parameters": {}}),- )-- assert safe_load_no_dates(ctx.staged_workflow_yaml) == safe_load_no_dates(workflow_yaml)- assert ctx.has_staged_proposal is True- assert json.loads(result) == {- "ok": False,- "data": {- "budget_expired": True,- "run_dispatched": False,- "source": "deadline",- },- }--@pytest.mark.parametrize("resolved", [False, True])def test_prior_expiry_and_question_are_serialized_into_next_model_history(resolved: bool) -> None:interaction = QuestionInteraction(@@ -721,6 +638,203 @@ async def test_unknown_tool_recovery_keeps_the_budget_drain_within_its_headroom(assert len(model.inputs) == 1 + BUDGET_DRAIN_HEADROOM+_REPLACE_WORKFLOW_TEXT = json.dumps(+ {"type": "REPLACE_WORKFLOW", "user_response": "Saved a new draft.", "workflow_yaml": HANDBACK_WORKFLOW_YAML}+)+++_DRAIN_KEPT_NATIVE_TOOLS = {+ "ask_user",+ "cancel_account_group",+ "delete_saved_credentials",+ "discover_workflow_entrypoint",+ "extend_browser_session",+ "fill_credential_field",+ "get_account_group_status",+ "get_organization_usage_quota",+ "get_run_results",+ "inspect_locator_matches",+ "inspect_page_for_composition",+ "list_credentials",+ "list_integrations",+ "read_google_sheet",+ "reply",+ "request_credential",+ "run_browser_code",+ "search_web",+ "set_work_plan",+ "solve_page_challenge",+ "start_fresh_browser",+ "upload_attached_file",+}+++ ("staged_before_expiry", "drain_script"),+ [+ (False, [scripted_call("update_workflow", {"workflow_yaml": "x"}, "call_stage"), REPLY_ARGUMENTS]),+ (True, [scripted_call("update_workflow", {"workflow_yaml": "x"}, "call_stage"), REPLY_ARGUMENTS]),+ (False, [scripted_text(_REPLACE_WORKFLOW_TEXT)]),+ (True, [scripted_text(_REPLACE_WORKFLOW_TEXT)]),+ ],+)+async def test_budget_drain_offers_no_tool_that_changes_or_runs_the_workflow(+ staged_before_expiry: bool, drain_script: list[Any]+) -> None:+ staged_writes: list[str] = []+ dispatched: list[list[str]] = []++ @function_tool+ async def probe() -> str:+ return json.dumps({"ok": True})++ @function_tool(name_override="update_workflow", strict_mode=False)+ async def stage_draft(ctx: RunContextWrapper[CopilotContext], workflow_yaml: str) -> str:+ staged_writes.append(workflow_yaml)+ ctx.context.staged_workflow = SimpleNamespace(workflow_id="wf_after_expiry")+ return json.dumps({"ok": True})++ @function_tool(name_override="run_blocks_and_collect_debug", strict_mode=False)+ async def dispatch_run(block_labels: list[str]) -> str:+ dispatched.append(block_labels)+ return json.dumps({"ok": True})++ stubbed = {stage_draft.name, dispatch_run.name, REPLY_TOOL_NAME}+ native_tools = [tool for tool in tools_module.NATIVE_TOOLS if tool.name not in stubbed]+ ctx = make_copilot_ctx(+ organization_id="org_test",+ workflow_permanent_id="wpid_test",+ workflow_id=None,+ workflow_yaml=None,+ stream=None,+ api_key="sk-copilot-org",+ )+ staged = SimpleNamespace(workflow_id="wf_draft", workflow_definition=SimpleNamespace(blocks=[]))+ if staged_before_expiry:+ ctx.staged_workflow = staged+ script = [+ [scripted_call("probe", {}, "call_probe")],+ *[[scripted_call("reply", item, "call_reply")] if item is REPLY_ARGUMENTS else [item] for item in drain_script],+ ]+ model = ScriptedModel(script)++ result, ctx, _stream = await run_scripted_turn(+ model,+ tools=[probe, stage_draft, dispatch_run, *native_tools],+ max_turns=1,+ ctx=ctx,+ )+ translated = await _translate_to_agent_result(result, ctx, None, _chat_request(), "org-1")++ first_call, *drain_calls = model.served_tool_names_by_call+ assert first_call - drain_calls[0] == first_call & BUDGET_DRAIN_EXCLUDED_TOOLS+ assert all(served == _DRAIN_KEPT_NATIVE_TOOLS | {"probe"} for served in drain_calls)+ assert CODE_WRITE_TOOL_NAMES <= BUDGET_DRAIN_EXCLUDED_TOOLS+ assert len(model.inputs) == len(script)+ for call_index, step in enumerate(drain_script[:-1], start=2):+ (not_run,) = tool_outputs_for(model.inputs[call_index], step.call_id)+ assert json.loads(not_run)["error"].startswith(f"No tool named {step.name} exists")+ assert staged_writes == []+ assert dispatched == []+ assert ctx.staged_workflow is (staged if staged_before_expiry else None)+ assert ctx.budget_expiry_state.staged_draft_id == ("wf_draft" if staged_before_expiry else None)+ assert translated.response_type == "REPLY"+ assert translated.turn_outcome is not None+ assert translated.turn_outcome.budget_expiry_report_produced is True+ assert translated.turn_outcome.budget_expiry_staged_draft_id == ("wf_draft" if staged_before_expiry else None)+ if drain_script[-1] is REPLY_ARGUMENTS:+ assert translated.user_response == REPLY_ARGUMENTS["user_response"]+ else:+ assert translated.updated_workflow is None+ assert translated.user_response.startswith("Saved a new draft.")+ assert FINAL_REPLY_UNAPPLIED_CHANGE_NOTE in translated.user_response+++ ("provider_error", "fallback_llm_key", "untested_retry"),+ [+ pytest.param(+ LiteLLMRateLimitError(message="rate limit", llm_provider="openai", model="m"),+ "SECONDARY",+ False,+ id="fallback_model",+ ),+ pytest.param(+ LiteLLMNotFoundError(message="missing model", llm_provider="azure", model="m"),+ None,+ True,+ id="model_lookup_retry",+ ),+ ],+)+async def test_provider_error_inside_the_budget_drain_never_reruns_with_write_tools(+ monkeypatch: pytest.MonkeyPatch,+ provider_error: Exception,+ fallback_llm_key: str | None,+ untested_retry: bool,+) -> None:+ offered: list[tuple[bool, set[str]]] = []+ contexts: list[CopilotContext] = []+ models: list[str] = []++ def run_streamed(agent: Agent, **kwargs: Any) -> MagicMock:+ ctx = kwargs["context"]+ contexts.append(ctx)+ models.append(str(agent.model))+ state = ctx.budget_expiry_state+ offered.append((state.drain_attempted, {tool.name for tool in agent.tools}))+ if state.drain_active:+ raise provider_error+ if not state.drain_attempted:+ raise MaxTurnsExceeded("cap")+ return _result(json.dumps({"type": "REPLY", "user_response": "Nothing was saved or tested this turn."}))++ surface_loop = agent_module._run_agent_loop_with_surface++ async def loop_with_untested_retry(**kwargs: Any) -> Any:+ return await surface_loop(**{**kwargs, "allow_untested_retry": untested_retry})++ stub_copilot_agent_loop(monkeypatch, run_with_enforcement)+ monkeypatch.setattr(agent_module, "_run_agent_loop_with_surface", loop_with_untested_retry)+ monkeypatch.setattr("skyvern.forge.sdk.copilot.enforcement.Runner.run_streamed", run_streamed)+ monkeypatch.setattr("skyvern.forge.sdk.copilot.streaming_adapter.stream_to_sse", AsyncMock(return_value=None))++ result = await agent_module.run_copilot_agent(+ stream=MagicMock(send=AsyncMock(return_value=True)),+ organization_id="org-1",+ chat_request=WorkflowCopilotChatRequest(+ message="build it",+ workflow_id="wf-1",+ workflow_permanent_id="wfp-1",+ workflow_copilot_chat_id="chat-1",+ workflow_yaml="",+ ),+ chat_history=[],+ global_llm_context=None,+ llm_api_handler=SimpleNamespace(llm_key="PRIMARY"),+ raw_secret_safety_handler=AsyncMock(+ return_value={"version": "1", "state": "clean", "handling": "none", "citations": []}+ ),+ api_key="sk-test",+ config=CopilotConfig(fallback_llm_key=fallback_llm_key),+ )++ (before_drain, *after_drain) = offered+ assert before_drain[0] is False and "update_workflow" in before_drain[1]+ assert after_drain and all(attempted for attempted, _ in after_drain)+ assert all(not served & {"update_workflow", "run_blocks_and_collect_debug"} for _, served in after_drain)+ assert after_drain[-1][1] == {REPLY_TOOL_NAME}+ assert (models[-1] != models[0]) is (fallback_llm_key is not None), models+ assert contexts[-1].budget_expiry_state.staged_draft_id is None+ assert result.response_type == "REPLY"+ assert result.user_response == "Nothing was saved or tested this turn."+ assert result.turn_outcome is not None+ assert result.turn_outcome.budget_expired is True+ assert result.turn_outcome.budget_expiry_staged_draft_id is None++async def _build_test_turn(monkeypatch: pytest.MonkeyPatch,*,diff --git a/tests/unit/test_copilot_build_test_outcome.py b/tests/unit/test_copilot_build_test_outcome.pyindex acfac57ab..f2ab0eb3e 100644--- a/tests/unit/test_copilot_build_test_outcome.py+++ b/tests/unit/test_copilot_build_test_outcome.py@@ -6372,49 +6372,6 @@ async def test_a_foreign_occupier_stops_the_dispatch_before_any_workflow_run_is_assert result["data"]["executed_block_labels"] == []-async def test_budget_denied_dispatch_preserves_prior_run_through_public_tool(- monkeypatch: pytest.MonkeyPatch,-) -> None:- ctx = make_copilot_ctx()- ctx.budget_expiry_state.source = "deadline"- ctx.budget_expiry_state.drain_active = True- ctx.last_run_blocks_workflow_run_id = "wr_prior"- ctx.last_successful_run_blocks_workflow_run_id = "wr_prior"- ctx.last_run_blocks_browser_session_id = "pbs_prior"- ctx.last_test_ok = True- ctx.last_full_workflow_test_ok = True- ctx.last_run_outcome = RecordedRunOutcome(- verdict="verified", reason_code="execution_completed", workflow_run_id="wr_prior", run_completed=True- )- previous_outcome = ctx.last_run_outcome- previous_history = list(ctx.run_outcome_trace)- previous_diagnosis = ctx.latest_diagnosis_repair_contract- monkeypatch.setattr(tools_module, "_authority_tool_error", lambda *_args: None)- monkeypatch.setattr(tools_module, "_get_prior_workflow_definition", AsyncMock(return_value=None))- monkeypatch.setattr(tools_module, "_plan_frontier", lambda *_args: (["step"], {}, "step", "initial"))- dispatch = AsyncMock(side_effect=AssertionError("denied dispatch must not reach the database"))- monkeypatch.setattr(run_execution_module.app.DATABASE.workflows, "get_workflow_by_permanent_id", dispatch)-- result = json.loads(- await tools_module.run_blocks_tool.on_invoke_tool(- SimpleNamespace(context=ctx, tool_name="run_blocks_and_collect_debug"),- json.dumps({"block_labels": ["step"], "parameters": {}}),- )- )-- assert ctx.last_run_blocks_workflow_run_id == "wr_prior"- assert ctx.last_successful_run_blocks_workflow_run_id == "wr_prior"- assert ctx.last_run_blocks_browser_session_id == "pbs_prior"- assert ctx.last_test_ok is True- assert ctx.last_full_workflow_test_ok is True- assert ctx.last_run_outcome is previous_outcome- assert ctx.run_outcome_trace == previous_history- assert ctx.latest_diagnosis_repair_contract is previous_diagnosis- assert result == {"ok": False, "data": {"budget_expired": True, "run_dispatched": False, "source": "deadline"}}- dispatch.assert_not_awaited()--def _two_block_search_failure_result() -> dict[str, Any]:return {"ok": False,@@ -7663,7 +7620,8 @@ async def test_a_failed_ai_fallback_reports_the_code_error_beside_the_recovery_easync def test_a_secret_shaped_token_in_the_code_error_never_reaches_the_run_results(monkeypatch: pytest.MonkeyPatch,) -> None:- token = "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0LXVzZXIifQ.c2lnbmF0dXJlLXZhbHVlLXg"+ # Joined at runtime so no source line holds a whole JWT for secret scanners to flag.+ token = ".".join(("eyJhbGciOiJIUzI1NiJ9", "eyJzdWIiOiJ0ZXN0LXVzZXIifQ", "c2lnbmF0dXJlLXZhbHVlLXg"))ctx = _install_rescued_run(monkeypatch,[_ai_fallback_episode(HealStatus.fired_completed, failing_line=5)],diff --git a/tests/unit/test_copilot_composition_evidence.py b/tests/unit/test_copilot_composition_evidence.pyindex 05023cf9f..06b513452 100644--- a/tests/unit/test_copilot_composition_evidence.py+++ b/tests/unit/test_copilot_composition_evidence.py@@ -72,12 +72,12 @@ from skyvern.forge.sdk.copilot.output_extraction_plan import (candidate_page_context,value_designation_probe_expression,)-from skyvern.forge.sdk.copilot.output_utils import _RECENT_TOOL_OUTPUT_CHAR_CAP+from skyvern.forge.sdk.copilot.output_utils import _RECENT_TOOL_OUTPUT_CHAR_CAP, sanitize_tool_result_for_llmfrom skyvern.forge.sdk.copilot.page_identity import page_location_fingerprintfrom skyvern.forge.sdk.copilot.runtime_authoring_repair import _runtime_form_summariesfrom skyvern.forge.sdk.copilot.tools import run_execution as run_execution_modulefrom skyvern.forge.sdk.copilot.tools.blockers import _artifact_challenge_flag_from_result-from skyvern.forge.sdk.copilot.tools.composition_capture import _model_facing_inspect_result+from skyvern.forge.sdk.copilot.tools.composition_capture import _PAGE_STATE_RESERVE_CHARS, _model_facing_inspect_resultfrom skyvern.forge.sdk.copilot.verification_evidence import WorkflowVerificationEvidence@@ -4371,6 +4371,7 @@ async def test_candidate_counts_reach_an_author_only_where_a_selector_is_not_uni"source": "class","match_count": 3,"visible_match_count": 1,+ "first_match_visible": False,}assert candidates["#q"] == {"selector": "#q", "source": "id"}assert [control["pointer_at_center"] for control in authoring] == ["reaches_control", "reaches_control"]@@ -4391,6 +4392,31 @@ async def test_candidate_counts_reach_an_author_only_where_a_selector_is_not_uni"pointer_at_center" not in control for control in controls(model_visible_composition_evidence(structured)))+ visible_first_raw, _ = await _capture_live_dom(+ url,+ (+ "<html><head><style>.legacy{display:none}</style></head><body>"+ '<form id="search"><input id="q" name="q" type="text">'+ '<button type="submit" class="go">Search</button>'+ '<button type="submit" class="go legacy">Search</button></form></body></html>'+ ),+ "#q",+ )+ visible_first = parse_composition_structured(json.loads(visible_first_raw), inspected_url=url, current_url=url)+ assert visible_first is not None+ visible_first_candidates = {+ candidate["selector"]: candidate+ for control in controls(model_visible_composition_evidence(visible_first, authoring_facts=True))+ for candidate in control["selector_candidates"]+ }+ assert visible_first_candidates["button.go"] == {+ "selector": "button.go",+ "source": "class",+ "match_count": 2,+ "visible_match_count": 1,+ "first_match_visible": True,+ }+_VISUALLY_HIDDEN = "position:absolute;width:1px;height:1px;overflow:hidden;clip:rect(0 0 0 0)"_HIT_TEST_EDGE_PAGE = f"""<!doctype html><html><head><style>html {{ overflow-y: scroll; }} body {{ margin: 0; }}</style><script>@@ -5126,7 +5152,9 @@ async def test_capture_over_its_bound_drops_pointer_states_and_visible_counts_be"return boundedStructuredEvidence(structuredEvidence);","return JSON.stringify(structuredEvidence);",)- html = _full_forms_under_a_page_wide_layer_html(placeholder_chars=placeholder_chars)+ html = _full_forms_under_a_page_wide_layer_html(placeholder_chars=placeholder_chars).replace(+ "<nav>", '<button type="submit" class="submit-btn" hidden>Submit</button><nav>'+ )async with async_playwright() as playwright:browser = await playwright.chromium.launch(@@ -5148,6 +5176,11 @@ async def test_capture_over_its_bound_drops_pointer_states_and_visible_counts_beassert all("visible_match_count" in candidate for control in controls for candidate in control["selector_candidates"])+ assert any(+ candidate.get("first_match_visible") is False+ for control in controls+ for candidate in control["selector_candidates"]+ )over_the_bound = len(json.dumps(unbounded, separators=(",", ":"))) > COMPOSITION_STRUCTURED_EVIDENCE_MAX_CHARSassert bounded == (without_them if over_the_bound else unbounded)@@ -5190,6 +5223,94 @@ async def test_inspect_result_over_the_model_bound_is_the_packet_without_authoriassert data[key] and json.dumps({key: data[key]})[1:-1] in head[email protected]("reader", ["inspect_result", "post_run_result"])+ ("relation_first_match_visible", "fitting", "expected"),+ [+ (False, "projected", "projected"),+ (False, "read_counts", "read_counts"),+ (False, "without_counts", "without_counts"),+ (False, "nothing", "read_counts"),+ (True, "read_counts", "without_counts"),+ (True, "nothing", "without_counts"),+ ],+)+def test_size_fallback_keeps_counts_only_on_a_value_relation_whose_first_match_is_hidden(+ reader: str, relation_first_match_visible: bool, fitting: str, expected: str+) -> None:+ relation_candidate = {+ "selector": "span.stat",+ "source": "class",+ "match_count": 4,+ "visible_match_count": 2,+ "first_match_visible": relation_first_match_visible,+ }+ # A hidden-first candidate unrelated to the read, such as a responsive nav copy, must not keep counts.+ field_candidate = {+ "selector": "input.nav",+ "source": "class",+ "match_count": 2,+ "visible_match_count": 1,+ "first_match_visible": False,+ }+ evidence = {+ "page_title": "Stats",+ "forms": [+ {+ "fields": [+ {+ "id": f"f{index}",+ "type": "text",+ "selector_candidates": [field_candidate],+ "pointer_at_center": "reaches_other_element",+ "element_at_center": {"tag": "div", "label_context": "Promo banner"},+ }+ for index in range(100)+ ]+ }+ ],+ "key_value_relations": [{"value_text": "23.2k", "selector_candidates": [relation_candidate] * 40}],+ }+ bare_fields = [+ {"id": f"f{index}", "type": "text", "selector_candidates": [{"selector": "input.nav", "source": "class"}]}+ for index in range(100)+ ]+ tiers = {+ "projected": evidence,+ "read_counts": {+ "page_title": "Stats",+ "forms": [{"fields": bare_fields}],+ "key_value_relations": [{"value_text": "23.2k", "selector_candidates": [relation_candidate] * 40}],+ },+ "without_counts": {+ "page_title": "Stats",+ "forms": [{"fields": bare_fields}],+ "key_value_relations": [+ {"value_text": "23.2k", "selector_candidates": [{"selector": "span.stat", "source": "class"}] * 40}+ ],+ },+ }++ def result(packet: dict[str, Any], padding: int) -> dict[str, Any]:+ padded = {**packet, "visible_text_excerpt": "x" * padding}+ if reader == "inspect_result":+ return {"ok": True, "data": padded}+ return {"ok": True, "data": {"workflow_run_id": "wr_1", "blocks": [], "post_run_page_evidence": padded}}++ bound = _RECENT_TOOL_OUTPUT_CHAR_CAP - (_PAGE_STATE_RESERVE_CHARS if reader == "inspect_result" else 0)+ padding = bound if fitting == "nothing" else bound - len(json.dumps(result(tiers[fitting], 0)))+ captured = result(evidence, padding)++ shaped = (+ _model_facing_inspect_result(captured)+ if reader == "inspect_result"+ else sanitize_tool_result_for_llm("run_blocks_and_collect_debug", captured)+ )++ packet = shaped["data"] if reader == "inspect_result" else shaped["data"]["post_run_page_evidence"]+ assert {key: value for key, value in packet.items() if key != "visible_text_excerpt"} == tiers[expected]++def test_inspect_result_with_authoring_facts_fits_the_largest_page_state_stamp() -> None:largest_stamp = {"read": "ok",@@ -7550,59 +7671,68 @@ _TWO_PANEL_TABLE = ()+async def _emitted_label_anchors(page: Page, html: str, total: str) -> dict[str, list[str]]:+ """The label-anchored candidates each emitter produces for the Visitors total: page capture, HTML parse, probe."""+ await page.set_content(html)+ raw = await page.evaluate(composition_structured_evidence_expression(("Visitors",)))+ live = parse_composition_structured(json.loads(raw), inspected_url="about:blank", current_url="about:blank")+ probe = await page.evaluate(value_designation_probe_expression(total, "Visitors"))+ parsed = parse_composition_html(+ html, inspected_url="about:blank", current_url="about:blank", requested_targets=("Visitors",)+ )+ assert live is not None+ return {+ "live": [+ candidate["selector"]+ for relation in live["key_value_relations"]+ if relation["key_text"] == "Visitors"+ for candidate in relation["selector_candidates"]+ if ":text-is(" in candidate["selector"]+ ],+ "parsed": [+ candidate["selector"]+ for relation in parsed["key_value_relations"]+ if relation["key_text"] == "Visitors"+ for candidate in relation["selector_candidates"]+ if ":text-is(" in candidate["selector"]+ ],+ "probe": [+ candidate["selector"] for candidate in probe["selector_candidates"] if ":text-is(" in candidate["selector"]+ ],+ }++@_skip_no_browser@pytest.mark.asyncio@pytest.mark.parametrize(- ("panels", "total", "anchored"),+ ("panels", "total", "label_shape"),[- (_TWO_PANEL_CARD.format(total="10.2K") + _TWO_PANEL_TABLE.format(header="Visitors"), "10.2K", True),- (_TWO_PANEL_TABLE.format(header="Visitors") + _TWO_PANEL_CARD.format(total="11.3K"), "11.3K", True),- (_TWO_PANEL_TABLE.format(header="Visitors") + _TWO_PANEL_ROW_CARD.format(total="11.3K"), "11.3K", True),+ (_TWO_PANEL_CARD.format(total="10.2K") + _TWO_PANEL_TABLE.format(header="Visitors"), "10.2K", "span"),+ (_TWO_PANEL_TABLE.format(header="Visitors") + _TWO_PANEL_CARD.format(total="11.3K"), "11.3K", "span"),+ (_TWO_PANEL_TABLE.format(header="Visitors") + _TWO_PANEL_ROW_CARD.format(total="11.3K"), "11.3K", "span"),(_TWO_PANEL_CARD.format(total="10.2K") + _TWO_PANEL_TABLE.format(header='<span class="lbl">Visitors</span>'),"10.2K",- False,+ None,+ ),+ (+ _TWO_PANEL_CARD.format(total="10.2K")+ + _TWO_PANEL_TABLE.format(header='<span class="legend">Visitors</span>'),+ "10.2K",+ "span.lbl",),],- ids=["original", "reordered", "row_with_delta", "header_shares_label_shape"],+ ids=["original", "reordered", "row_with_delta", "header_shares_label_shape", "header_shares_label_tag"],)async def test_label_anchor_names_the_panel_whose_label_a_same_class_panel_repeats(- panels: str, total: str, anchored: bool+ panels: str, total: str, label_shape: str | None) -> None:html = f"<html><head><style>th{{text-transform:uppercase}}</style></head><body><main>{panels}</main></body></html>"async with async_playwright() as playwright:browser = await playwright.chromium.launch(headless=True)try:page = await browser.new_page()- await page.set_content(html)- raw = await page.evaluate(composition_structured_evidence_expression(("Visitors",)))- live = parse_composition_structured(json.loads(raw), inspected_url="about:blank", current_url="about:blank")- probe = await page.evaluate(value_designation_probe_expression(total, "Visitors"))- parsed = parse_composition_html(- html, inspected_url="about:blank", current_url="about:blank", requested_targets=("Visitors",)- )- assert live is not None- emitted = {- "live": [- candidate["selector"]- for relation in live["key_value_relations"]- if relation["key_text"] == "Visitors"- for candidate in relation["selector_candidates"]- if ":text-is(" in candidate["selector"]- ],- "parsed": [- candidate["selector"]- for relation in parsed["key_value_relations"]- if relation["key_text"] == "Visitors"- for candidate in relation["selector_candidates"]- if ":text-is(" in candidate["selector"]- ],- "probe": [- candidate["selector"]- for candidate in probe["selector_candidates"]- if ":text-is(" in candidate["selector"]- ],- }+ emitted = await _emitted_label_anchors(page, html, total)resolved = {selector: (await page.locator(selector).count(), await page.locator(selector).first.inner_text())for selectors in emitted.values()@@ -7614,8 +7744,42 @@ async def test_label_anchor_names_the_panel_whose_label_a_same_class_panel_repeafor selector, (count, text) in resolved.items():assert count == 1, f"{selector} resolves to {count} elements"assert total in text and "6,621" not in text, f"{selector} resolves to {text!r}"- if anchored:+ if label_shape:assert len(emitted["live"]) == 1 and emitted["parsed"] == emitted["live"], emitted+ assert emitted["live"][0].startswith(f'div.panel:has({label_shape}:text-is("Visitors"))'), emittedassert len(emitted["probe"]) == 1, emittedelse:assert emitted["live"] == [] and emitted["parsed"] == [], emitted+++_DRIFT_CARD = (+ '<div class="card"><div class="{classes}">Visitors</div><div class="text-2xl font-bold">{total}</div></div>'+)+_DRIFT_TABLE = '<div class="card"><table><thead><tr><th>Region</th><th>Visitors</th></tr></thead></table></div>'+++@_skip_no_browser+async def test_label_anchor_resolves_after_the_label_class_set_changes() -> None:+ def dashboard(classes: str) -> str:+ cards = _DRIFT_CARD.format(classes=classes, total="10.2K") + _DRIFT_TABLE+ return f"<html><body><main>{cards}</main></body></html>"++ async with async_playwright() as playwright:+ browser = await playwright.chromium.launch(headless=True)+ try:+ page = await browser.new_page()+ emitted = await _emitted_label_anchors(page, dashboard("text-sm font-medium"), "10.2K")+ await page.set_content(dashboard("text-sm font-semibold"))+ resolved = {+ selector: (await page.locator(selector).count(), await page.locator(selector).first.inner_text())+ for selectors in emitted.values()+ for selector in selectors+ }+ finally:+ await browser.close()++ assert len(emitted["live"]) == 1 and emitted["parsed"] == emitted["live"], emitted+ assert len(emitted["probe"]) == 1, emitted+ for selector, (count, text) in resolved.items():+ assert count == 1 and "10.2K" in text, f"{selector} resolves to {count} elements reading {text!r}"diff --git a/tests/unit/test_copilot_credential_pause.py b/tests/unit/test_copilot_credential_pause.pyindex baa2cc520..5d95f251b 100644--- a/tests/unit/test_copilot_credential_pause.py+++ b/tests/unit/test_copilot_credential_pause.py@@ -98,7 +98,10 @@ from skyvern.forge.sdk.copilot.request_policy import (_seed_prior_approved_credentials,_seed_proposed_credential,)-from skyvern.forge.sdk.copilot.runtime import CredentialOriginRecovery, CredentialOriginRecoveryState+from skyvern.forge.sdk.copilot.runtime import (+ CredentialOriginRecovery,+ CredentialOriginRecoveryState,+)from skyvern.forge.sdk.copilot.tools import credential_fill as credential_fill_modulefrom skyvern.forge.sdk.copilot.tools._shared import TOTAL_TIMEOUT_SECONDS, _copilot_seconds_remainingfrom skyvern.forge.sdk.copilot.tools.credential_fill import _request_credential@@ -1706,7 +1709,7 @@ async def _ask(ctx: CopilotContext, login_page_url: str = "https://portal.exampldef test_connected_resume_stamps_the_single_ask_origin() -> None:policy = _ask_origin_policy("https://portal.example.com/login")- credential_pause_module._apply_connected_credential_to_policy(make_copilot_context(), policy, _make_credential())+ credential_pause_module.apply_connected_credential_to_policy(make_copilot_context(), policy, _make_credential())assert policy.live_page_admitted_urls == {"cred_1": "https://portal.example.com/login"}@@ -1714,7 +1717,7 @@ def test_connected_resume_stamps_the_single_ask_origin() -> None:def test_connected_resume_stamps_nothing_when_the_ask_spans_several_origins() -> None:policy = _ask_origin_policy("https://portal.example.com/login", "https://other.example.com/login")- credential_pause_module._apply_connected_credential_to_policy(make_copilot_context(), policy, _make_credential())+ credential_pause_module.apply_connected_credential_to_policy(make_copilot_context(), policy, _make_credential())assert policy.live_page_admitted_urls == {}@@ -1728,7 +1731,7 @@ async def test_generated_credential_connected_from_a_multi_origin_ask_gets_no_fictx.block_authoring_policy = BlockAuthoringPolicy.CODE_ONLY_BROWSERctx.org_credentials_for_turn = []- credential_pause_module._apply_connected_credential_to_policy(ctx, ctx.request_policy, _make_credential("cred_new"))+ credential_pause_module.apply_connected_credential_to_policy(ctx, ctx.request_policy, _make_credential("cred_new"))grant, error = await tools_module.credential_fill._credential_fill_origin_grant(ctx, "cred_new")assert ctx.request_policy.live_page_admitted_urls == {}@@ -1741,7 +1744,7 @@ def test_connected_resume_names_the_picked_credential_superseding_an_earlier_menpolicy = _ask_origin_policy("https://portal.example.com/login")policy.current_turn_named_credential_ids = {"cred_earlier"}- credential_pause_module._apply_connected_credential_to_policy(make_copilot_context(), policy, _make_credential())+ credential_pause_module.apply_connected_credential_to_policy(make_copilot_context(), policy, _make_credential())assert policy.current_turn_named_credential_ids == {"cred_1"}@@ -1751,7 +1754,7 @@ def test_connected_resume_does_not_duplicate_an_already_resolved_credential() ->credential = _make_credential()policy.resolved_credentials = [credential]- credential_pause_module._apply_connected_credential_to_policy(make_copilot_context(), policy, credential)+ credential_pause_module.apply_connected_credential_to_policy(make_copilot_context(), policy, credential)assert [resolved.credential_id for resolved in policy.resolved_credentials] == ["cred_1"]@@ -1765,7 +1768,7 @@ def test_a_card_connected_credential_keeps_its_durable_cross_turn_approval(loginpolicy = _ask_origin_policy(*login_urls)ctx.request_policy = policy- credential_pause_module._apply_connected_credential_to_policy(ctx, policy, _make_credential())+ credential_pause_module.apply_connected_credential_to_policy(ctx, policy, _make_credential())carried = record_approved_credentials_in_global_llm_context(ctx, None)assert [record.credential_id for record in StructuredContext.from_json_str(carried).approved_credentials] == [@@ -1791,7 +1794,7 @@ async def test_card_selection_keeps_its_origin_on_the_next_turn(previous = StructuredContext()if previous_origin is not None:previous.approved_credentials = [ApprovedCredential(credential_id="cred_1", admitted_url=previous_origin)]- credential_pause_module._apply_connected_credential_to_policy(ctx, ctx.request_policy, _make_credential())+ credential_pause_module.apply_connected_credential_to_policy(ctx, ctx.request_policy, _make_credential())carried = record_approved_credentials_in_global_llm_context(ctx, previous.to_json_str())assert carried is not Nonerecords = StructuredContext.from_json_str(carried).approved_credentials@@ -1831,7 +1834,7 @@ async def test_invalid_card_origin_cannot_create_cross_turn_approval(# Retained state must also fail closed if it predates validation at the card boundary.ctx.request_policy = _ask_origin_policy(login_url)- credential_pause_module._apply_connected_credential_to_policy(ctx, ctx.request_policy, _make_credential())+ credential_pause_module.apply_connected_credential_to_policy(ctx, ctx.request_policy, _make_credential())carried = record_approved_credentials_in_global_llm_context(ctx, None)assert StructuredContext.from_json_str(carried).approved_credentials == []@@ -1860,7 +1863,7 @@ async def test_saved_workflow_resolution_requires_user_selection_for_chat_approvasync def test_card_approval_preserves_explicit_port_zero(monkeypatch: pytest.MonkeyPatch) -> None:ctx = make_copilot_context()ctx.request_policy = _ask_origin_policy("https://portal.example.com:0/login?code=synthetic-code")- credential_pause_module._apply_connected_credential_to_policy(ctx, ctx.request_policy, _make_credential())+ credential_pause_module.apply_connected_credential_to_policy(ctx, ctx.request_policy, _make_credential())carried = record_approved_credentials_in_global_llm_context(ctx, None)next_policy = RequestPolicy()_stub_credential_lookup(monkeypatch, _make_credential())diff --git a/tests/unit/test_copilot_demonstration.py b/tests/unit/test_copilot_demonstration.pynew file mode 100644index 000000000..923350eaa--- /dev/null+++ b/tests/unit/test_copilot_demonstration.py@@ -0,0 +1,1775 @@+from __future__ import annotations++import asyncio+import json+from datetime import UTC, datetime, timedelta+from types import SimpleNamespace+from typing import Any, ClassVar+from unittest.mock import AsyncMock, MagicMock++import pytest++from skyvern.exceptions import SkyvernHTTPException+from skyvern.forge import app+from skyvern.forge.sdk.cache.base import NoopLock+from skyvern.forge.sdk.copilot import demonstration_recorder as recorder_module+from skyvern.forge.sdk.copilot import demonstration_replay as replay_module+from skyvern.forge.sdk.copilot.demonstration_recorder import (+ DemonstrationClaim,+ DemonstrationRecord,+ DemonstrationRecorder,+ DemonstrationRejection,+ active_demonstration,+ append_demonstration_control,+ claim_demonstration_recording,+ decode_demonstration_record,+ demonstration_browser_key,+ demonstration_controls_key,+ demonstration_event_key,+ demonstration_record_key,+ encode_demonstration_record,+ load_demonstration_record,+ validate_demonstration_for_replay,+)+from skyvern.forge.sdk.copilot.demonstration_replay import replay_demonstration+from skyvern.forge.sdk.copilot.demonstration_tap import TapEvent+from skyvern.forge.sdk.copilot.mcp_adapter import SkyvernOverlayMCPServer+from skyvern.forge.sdk.copilot.request_policy import RequestPolicy+from skyvern.forge.sdk.copilot.secret_redaction import redact_raw_secrets_for_prompt+from skyvern.forge.sdk.copilot.secret_scrub import register_secret_scrub_value+from skyvern.forge.sdk.copilot.tools import _build_skyvern_mcp_overlays, get_skyvern_mcp_alias_map+from skyvern.forge.sdk.schemas.credentials import Credential, CredentialType, CredentialVaultType+from skyvern.forge.sdk.schemas.workflow_copilot import WorkflowCopilotDemonstrationControlRequest+from tests.unit.copilot_test_helpers import make_copilot_ctx+++class _SharedCache:+ is_shared = True++ def __init__(self) -> None:+ self.store: dict[str, Any] = {}+ self.expirations: dict[str, Any] = {}+ self.lock_names: list[str] = []++ async def get(self, key: str) -> Any:+ return self.store.get(key)++ async def set(self, key: str, value: Any, ex: Any = None) -> None:+ self.store[key] = value+ self.expirations[key] = ex++ def get_lock(self, lock_name: str, blocking_timeout: int = 5, timeout: int = 10) -> NoopLock:+ self.lock_names.append(lock_name)+ return NoopLock(lock_name, blocking_timeout, timeout)+++def _signal_refresh_after(recorder: DemonstrationRecorder, started: asyncio.Event) -> asyncio.Event:+ refreshed = asyncio.Event()+ refresh_owner = recorder._refresh_owner++ async def refresh_and_signal() -> None:+ await refresh_owner()+ if started.is_set():+ refreshed.set()++ recorder._refresh_owner = refresh_and_signal # type: ignore[method-assign]+ return refreshed+++class _Stream:+ def __init__(self, *, disconnected: bool = False) -> None:+ self.frames: list[Any] = []+ self.disconnected = disconnected++ async def send(self, data: Any) -> bool:+ self.frames.append(data)+ return True++ async def is_disconnected(self) -> bool:+ return self.disconnected++ async def close(self) -> None:+ return None+++class _NoDispatchClient:+ async def call_tool(self, name: str, args: dict[str, Any], raise_on_error: bool = False) -> Any:+ raise AssertionError(f"recorded call dispatched to MCP: {name}")+++class _FakeTap:+ events: ClassVar[list[TapEvent]] = []+ flush_events: ClassVar[list[TapEvent]] = []++ def __init__(self, context: Any, *, max_events: int = recorder_module.DEMONSTRATION_MAX_EVENTS) -> None:+ events = list(type(self).events)+ self.max_events = max_events+ self.event_limit_reached = len(events) > max_events+ self.pending = events[:max_events]++ async def start(self, pages: Any) -> None:+ return None++ def drain_events(self) -> list[TapEvent]:+ events, self.pending = self.pending, []+ return events++ async def flush(self) -> None:+ self.pending.extend(type(self).flush_events)+ if len(self.pending) > self.max_events:+ self.event_limit_reached = True+ self.pending = self.pending[: self.max_events]++ async def stop(self) -> None:+ return None++ async def set_intercept_file_chooser_dialog(self, enabled: bool) -> None:+ return None++ async def frame_path(self, event: TapEvent) -> list[dict[str, str]] | None:+ return [] if event.is_main_frame else [{"name": event.frame_name}] if event.frame_name else None++ def begin_control_navigation(self) -> None:+ return None++ def end_control_navigation(self) -> None:+ return None+++def _credential() -> Credential:+ now = datetime(2026, 1, 1, tzinfo=UTC)+ return Credential(+ credential_id="cred-1",+ organization_id="org-1",+ name="Saved login",+ vault_type=CredentialVaultType.SKYVERN,+ item_id="item-1",+ credential_type=CredentialType.PASSWORD,+ username="[email protected]",+ card_last4=None,+ card_brand=None,+ created_at=now,+ modified_at=now,+ )+++async def test_replay_contract_never_touches_a_browser_and_consumes_once(monkeypatch: pytest.MonkeyPatch) -> None:+ cache = _SharedCache()+ monkeypatch.setattr(app, "CACHE", cache)+ credential = _credential()+ monkeypatch.setattr(+ app,+ "DATABASE",+ SimpleNamespace(+ credentials=SimpleNamespace(get_credentials_by_ids=AsyncMock(return_value=[credential])),+ ),+ )+ token = "sk-proj-" + "a" * 48+ assert redact_raw_secrets_for_prompt(token) == "[REDACTED_SECRET]"++ async def safety_handler(*, prompt: str, prompt_name: str) -> dict[str, Any]:+ if "swordfish" in prompt:+ return {"version": "1", "state": "detected", "citations": ["swordfish"]}+ return {"version": "1", "state": "clean", "citations": []}++ monkeypatch.setattr(+ replay_module,+ "resolve_raw_secret_safety_handler",+ AsyncMock(return_value=safety_handler),+ )+ events = [+ {+ "kind": "click",+ "selector": "#company",+ "source_url": "https://example.test/form",+ "frame": {"is_main": True, "name": ""},+ "packet": {+ "role_name": {"role": "button", "accessible_name": "Company"},+ "selector_candidates": [{"selector": "#company", "source": "id", "match_count": 1}],+ "selector_match_count": 1,+ },+ },+ {+ "kind": "gap",+ "reason": "file_chooser",+ "source_url": f"https://example.test/form?token={token}",+ "frame": {"is_main": True, "name": ""},+ },+ {+ "kind": "type",+ "selector": "#name",+ "value": "Acme Corp",+ "source_url": "https://example.test/form",+ "frame": {"is_main": True, "name": ""},+ "packet": {"selector_match_count": 1},+ },+ {+ "kind": "type",+ "selector": "#reference",+ "value": token,+ "source_url": "https://example.test/form",+ "frame": {"is_main": True, "name": ""},+ "packet": {"selector_match_count": 1},+ },+ {+ "kind": "type",+ "selector": "#unlabelled",+ "value": "swordfish",+ "label": recorder_module._event_label({"kind": "type", "selector": "#unlabelled", "value": "swordfish"}),+ "source_url": "https://example.test/form",+ "frame": {"is_main": True, "name": ""},+ "packet": {"selector_match_count": 1},+ },+ {+ "kind": "select",+ "selector": "#tier",+ "value": "Premium",+ "source_url": "https://example.test/form",+ "frame": {"is_main": True, "name": ""},+ },+ {+ "kind": "key",+ "selector": "#tier",+ "key": "Enter",+ "source_url": "https://example.test/form",+ "frame": {"is_main": True, "name": ""},+ },+ {+ "kind": "navigate",+ "url": f"https://example.test/done?token={token}",+ "source_url": "https://example.test/form",+ "frame": {"is_main": True, "name": ""},+ },+ {+ "kind": "secret",+ "selector": "#password",+ "field": "password",+ "source_url": "https://example.test/done",+ "frame": {"is_main": True, "name": ""},+ },+ {+ "kind": "secret",+ "selector": "#nearby-heading",+ "field": "other",+ "field_label": 'Reference "code"',+ "source_url": "https://example.test/done",+ "frame": {"is_main": True, "name": ""},+ },+ {+ "kind": "credential_fill",+ "selector": "#password",+ "credential_id": "cred-1",+ "credential_name": "Saved login",+ "field": "password",+ "source_url": "https://example.test/done",+ "frame": {"is_main": True, "name": ""},+ "result": {"ok": True, "data": {"field": "password"}},+ "scout_fact": {+ "tool_name": "fill_credential_field",+ "selector": "#password",+ "source_url": "https://example.test/done",+ "credential_id": "cred-1",+ "credential_field": "password",+ },+ },+ ]+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="stopped",+ event_count=len(events),+ owner=None,+ owner_seen_at=None,+ final_url="https://example.test/done",+ stop_reason="disconnected",+ insertion={"after_label": "Saved step", "before_label": None},+ )+ )+ for seq, event in enumerate(events, 1):+ cache.store[demonstration_event_key("org-1", "demo-1", seq)] = json.dumps({"seq": seq, **event})++ ctx = make_copilot_ctx(+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ turn_id="turn-1",+ )+ ctx.demonstration_id = "demo-1"+ ctx.demonstration_insertion = {"after_label": "Ignored request step", "before_label": None}+ ctx.request_policy = RequestPolicy()+ aliases = get_skyvern_mcp_alias_map()+ overlays = _build_skyvern_mcp_overlays()+ names = ("click", "type_text", "select_option", "press_key", "navigate_browser")+ server = SkyvernOverlayMCPServer(+ transport=MagicMock(),+ overlays={name: overlays[name] for name in names},+ alias_map={name: aliases[name] for name in names},+ allowlist=frozenset(aliases[name] for name in names),+ context_provider=lambda: ctx,+ )+ server._client = _NoDispatchClient()+ ctx.discovery_mcp_server = server++ cache.store[demonstration_controls_key("org-1", "demo-1")] = "[]"+ history = await replay_demonstration(ctx, server)++ calls = [item for item in history if item.get("type") == "function_call"]+ assert [item["name"] for item in calls] == [+ "click",+ "type_text",+ "type_text",+ "type_text",+ "select_option",+ "press_key",+ "navigate_browser",+ "type_text",+ "type_text",+ "fill_credential_field",+ ]+ serialized = json.dumps(history)+ assert "Acme Corp" in serialized+ assert token not in serialized+ assert "swordfish" not in serialized+ assert "<value not recorded>" in serialized+ assert any("file chooser" in str(item.get("content")) for item in history if item.get("role") == "user")+ assert any("disconnected" in str(item.get("content")) for item in history if item.get("role") == "user")+ gap_index = next(index for index, item in enumerate(history) if "file chooser" in str(item.get("content")))+ first_type_index = next(index for index, item in enumerate(history) if item.get("name") == "type_text")+ assert gap_index < first_type_index+ navigate_index = next(index for index, item in enumerate(history) if item.get("name") == "navigate_browser")+ assert "?token=" not in json.dumps(history[navigate_index : navigate_index + 2])+ secret_call = calls[7]+ assert json.loads(secret_call["arguments"])["text"] == "<value not recorded>"+ outputs = {+ item["call_id"]: json.loads(item["output"]).get("data")+ for item in history+ if item.get("type") == "function_call_output"+ }+ assert outputs[secret_call["call_id"]]["sensitive_field"] == "password"+ assert outputs[calls[8]["call_id"]] == {"value_recorded": False, "field_label": "Reference code"}+ assert json.loads(calls[4]["arguments"])["value"] == "Premium"+ assert ctx.pending_screenshots == []+ assert credential in ctx.request_policy.resolved_credentials+ assert all(item.get("actor") == "user" for item in ctx.scout_trajectory)+ assert ctx.scout_trajectory[0]["selector_candidates"][0]["selector"] == "#company"+ assert ctx.scout_trajectory[0]["result_url"] == "https://example.test/form"+ assert ctx.scout_trajectory[0]["observed_effects"]["url_changed"] is False+ assert "swordfish" not in json.dumps(ctx.demonstration_items)+ assert any("#unlabelled" in label for label in ctx.demonstration_items)+ assert "Saved step" in serialized+ assert "Ignored request step" not in serialized+ record = decode_demonstration_record(cache.store[demonstration_record_key("org-1", "demo-1")])+ assert record is not None and record.status == "consumed"+ payload_keys = [demonstration_event_key("org-1", "demo-1", seq) for seq in range(1, record.event_count + 1)]+ payload_keys.append(demonstration_record_key("org-1", "demo-1"))+ payload_keys.append(demonstration_controls_key("org-1", "demo-1"))+ assert {cache.expirations[key] for key in payload_keys} == {recorder_module.DEMONSTRATION_CANCELLED_TTL}+ with pytest.raises(DemonstrationRejection) as exc_info:+ await replay_demonstration(ctx, server)+ assert exc_info.value.status_code == 409++[email protected]("safety_handler", [None, AsyncMock(side_effect=RuntimeError("screen failed"))])+async def test_replay_values_fail_closed_when_secret_screen_is_unavailable(+ monkeypatch: pytest.MonkeyPatch, safety_handler: Any+) -> None:+ cache = _SharedCache()+ monkeypatch.setattr(app, "CACHE", cache)+ monkeypatch.setattr(+ replay_module,+ "resolve_raw_secret_safety_handler",+ AsyncMock(return_value=safety_handler),+ )+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="stopped",+ event_count=1,+ owner=None,+ owner_seen_at=None,+ final_url="https://example.test/form",+ )+ )+ cache.store[demonstration_event_key("org-1", "demo-1", 1)] = json.dumps(+ {+ "seq": 1,+ "kind": "type",+ "selector": "#plain",+ "value": "ordinary words",+ "source_url": "https://example.test/form",+ "frame": {"is_main": True, "name": ""},+ }+ )+ ctx = make_copilot_ctx(organization_id="org-1", workflow_permanent_id="wfp-1")+ ctx.demonstration_id = "demo-1"+ server = SimpleNamespace(call_tool=AsyncMock(return_value={"ok": True}))++ history = await replay_demonstration(ctx, server) # type: ignore[arg-type]++ assert "ordinary words" not in json.dumps(history)+ assert "<value not recorded>" in json.dumps(history)+ outage_facts = [item for item in history if item.get("role") == "user" and "unavailable" in item.get("content", "")]+ assert len(outage_facts) == 1+++async def test_replay_reports_nested_frame_chain_as_a_gap(monkeypatch: pytest.MonkeyPatch) -> None:+ cache = _SharedCache()+ monkeypatch.setattr(app, "CACHE", cache)+ monkeypatch.setattr(replay_module, "resolve_raw_secret_safety_handler", AsyncMock(return_value=None))+ events = [+ {+ "kind": "click",+ "selector": "#inside",+ "source_url": "https://example.test/frame",+ "frame": {+ "is_main": False,+ "path": [{"name": "outer"}, {"owner_selector": "#inner"}],+ },+ },+ {+ "kind": "click",+ "selector": "#outside",+ "source_url": "https://example.test/",+ "frame": {"is_main": True, "path": []},+ },+ ]+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="stopped",+ event_count=len(events),+ owner=None,+ owner_seen_at=None,+ final_url="https://example.test/",+ )+ )+ for seq, event in enumerate(events, 1):+ cache.store[demonstration_event_key("org-1", "demo-1", seq)] = json.dumps({"seq": seq, **event})+ ctx = make_copilot_ctx(organization_id="org-1", workflow_permanent_id="wfp-1")+ ctx.demonstration_id = "demo-1"+ call_tool = AsyncMock(return_value={"ok": True})++ history = await replay_demonstration(ctx, SimpleNamespace(call_tool=call_tool)) # type: ignore[arg-type]++ call_tool.assert_awaited_once_with("click", {"selector": "#outside"})+ assert any("nested frame" in str(item.get("content")) for item in history)+++async def test_recorder_persists_tap_actions_controls_and_stops(monkeypatch: pytest.MonkeyPatch) -> None:+ cache = _SharedCache()+ demonstration_id = "demo-1"+ owner = "owner-1"+ cache.store[demonstration_record_key("org-1", demonstration_id)] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="recording",+ event_count=0,+ owner=owner,+ owner_seen_at=datetime.now(UTC),+ final_url=None,+ )+ )+ cache.store[demonstration_controls_key("org-1", demonstration_id)] = json.dumps(+ [+ {"seq": 1, "action": "navigate", "url": "http://127.0.0.1/private"},+ {"seq": 2, "action": "navigate", "url": "https://example.test/next"},+ ]+ )+ await cache.set(demonstration_browser_key("org-1", "pbs-1"), demonstration_id, ex=timedelta(minutes=1))+ page = SimpleNamespace(url="https://example.test/start", goto=AsyncMock(), is_closed=lambda: False)+ click = TapEvent(+ page=page,+ session_key="tap-1",+ frame_id="main",+ is_main_frame=True,+ frame_name="",+ payload={+ "kind": "click",+ "ts": 1,+ "url": page.url,+ "selector": "#continue",+ "packet": {"role_name": {"role": "button", "accessible_name": "Continue"}},+ },+ )+ password = TapEvent(+ page=page,+ session_key="tap-1",+ frame_id="main",+ is_main_frame=True,+ frame_name="",+ payload={+ "kind": "secret",+ "ts": 2,+ "url": "https://example.test/start?token=secret#password",+ "selector": "#password",+ "field": "password",+ },+ )+ password_focus = TapEvent(+ page=page,+ session_key="tap-1",+ frame_id="main",+ is_main_frame=True,+ frame_name="",+ payload={+ "kind": "secret_focus",+ "ts": 1.5,+ "url": "https://example.test/start?token=secret#password",+ "selector": "#password",+ "field": "password",+ },+ )+ file_click = TapEvent(+ page=page,+ session_key="tap-1",+ frame_id="main",+ is_main_frame=True,+ frame_name="",+ payload={"kind": "file_click", "ts": 3, "url": page.url, "selector": "#upload"},+ )++ browser_context = SimpleNamespace(pages=[page])+ browser_state = SimpleNamespace(+ browser_context=browser_context,+ get_working_page=AsyncMock(return_value=page),+ )+ _FakeTap.events = [click, password_focus, password, file_click]+ _FakeTap.flush_events = []+ monkeypatch.setattr(recorder_module, "PageTap", _FakeTap)+ monkeypatch.setattr(recorder_module, "resolve_persistent_browser_state", AsyncMock(return_value=browser_state))++ def validate(url: str) -> str:+ if "127.0.0.1" in url:+ raise SkyvernHTTPException(message="private", status_code=400)+ return url++ monkeypatch.setattr(recorder_module, "validate_fetch_url", validate)+ stream = _Stream()+ stream.is_disconnected = AsyncMock(side_effect=[False, True]) # type: ignore[method-assign]+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim(demonstration_id, owner, False, 0),+ stream=stream,+ )++ await recorder.run()++ assert stream.frames[0]["type"] == "demonstration_started"+ assert stream.frames[0]["can_fill_credentials"] is True+ first = json.loads(cache.store[demonstration_event_key("org-1", demonstration_id, 1)])+ second = json.loads(cache.store[demonstration_event_key("org-1", demonstration_id, 2)])+ third = json.loads(cache.store[demonstration_event_key("org-1", demonstration_id, 3)])+ fourth = json.loads(cache.store[demonstration_event_key("org-1", demonstration_id, 4)])+ assert [first["kind"], second["kind"], third["kind"], fourth["kind"]] == [+ "click",+ "secret",+ "gap",+ "navigate",+ ]+ assert third["reason"] == "file_chooser"+ assert page.goto.await_args.args == ("https://example.test/next",)+ assert any(frame.get("type") == "demonstration_notice" and frame.get("level") == "error" for frame in stream.frames)+ action_frames = [frame for frame in stream.frames if frame.get("type") == "demonstration_action"]+ assert [(frame["seq"], frame["tool_name"]) for frame in action_frames] == [+ (1, "click"),+ (3, "gap"),+ (4, "navigate_browser"),+ ]+ sensitive_frame = next(frame for frame in stream.frames if frame.get("type") == "demonstration_sensitive_field")+ assert sensitive_frame == {+ "type": "demonstration_sensitive_field",+ "field": "password",+ "selector": "#password",+ "url": "https://example.test/start",+ }+ stopped = decode_demonstration_record(cache.store[demonstration_record_key("org-1", demonstration_id)])+ assert stopped is not None and stopped.status == "stopped" and stopped.event_count == 4+ assert stopped.stop_reason == "disconnected"+ assert stream.frames[-1] == {+ "type": "demonstration_stopped",+ "demonstration_id": "demo-1",+ "reason": "disconnected",+ "action_count": 4,+ }+ assert cache.expirations[demonstration_browser_key("org-1", "pbs-1")] == recorder_module.DEMONSTRATION_TTL+++async def test_live_owner_blocks_a_second_recording_on_the_browser() -> None:+ cache = _SharedCache()+ cache.store[demonstration_browser_key("org-1", "pbs-1")] = "demo-live"+ cache.store[demonstration_record_key("org-1", "demo-live")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="recording",+ event_count=0,+ owner="owner-live",+ owner_seen_at=datetime.now(UTC),+ final_url=None,+ )+ )++ with pytest.raises(DemonstrationRejection) as exc_info:+ await claim_demonstration_recording(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ demonstration_id=None,+ )++ assert exc_info.value.status_code == 409+++async def test_long_control_keeps_the_owner_live(monkeypatch: pytest.MonkeyPatch) -> None:+ cache = _SharedCache()+ demonstration_id = "demo-long-control"+ owner = "owner-1"+ cache.store[demonstration_browser_key("org-1", "pbs-1")] = demonstration_id+ cache.store[demonstration_record_key("org-1", demonstration_id)] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="recording",+ event_count=0,+ owner=owner,+ owner_seen_at=datetime.now(UTC) - timedelta(seconds=recorder_module.DEMONSTRATION_OWNER_LIVE_SECONDS + 1),+ final_url=None,+ )+ )+ cache.store[demonstration_controls_key("org-1", demonstration_id)] = json.dumps(+ [{"seq": 1, "action": "navigate", "url": "https://example.test/slow"}]+ )+ navigation_started = asyncio.Event()+ finish_navigation = asyncio.Event()++ async def slow_goto(url: str) -> None:+ assert url == "https://example.test/slow"+ navigation_started.set()+ await finish_navigation.wait()++ monkeypatch.setattr(recorder_module, "DEMONSTRATION_OWNER_REFRESH_SECONDS", 0.01)+ monkeypatch.setattr(recorder_module, "validate_fetch_url", lambda url: url)+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim(demonstration_id, owner, False, 0),+ stream=_Stream(),+ )+ recorder.page = SimpleNamespace(url="https://example.test/start", goto=slow_goto)+ refreshed = _signal_refresh_after(recorder, navigation_started)++ control = asyncio.create_task(recorder._run_controls())+ await asyncio.wait_for(refreshed.wait(), 5)+ try:+ with pytest.raises(DemonstrationRejection) as exc_info:+ await claim_demonstration_recording(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ demonstration_id=None,+ )+ assert exc_info.value.status_code == 409+ finally:+ finish_navigation.set()+ await control+++async def test_stop_flush_keeps_the_owner_live(monkeypatch: pytest.MonkeyPatch) -> None:+ cache = _SharedCache()+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ "wfp-1",+ "pbs-1",+ "recording",+ 0,+ "owner-1",+ datetime.now(UTC) - timedelta(seconds=recorder_module.DEMONSTRATION_OWNER_LIVE_SECONDS + 1),+ None,+ )+ )+ flush_started = asyncio.Event()+ finish_flush = asyncio.Event()++ class SlowTap:+ event_limit_reached = False++ async def flush(self) -> None:+ flush_started.set()+ await finish_flush.wait()++ def drain_events(self) -> list[TapEvent]:+ return []++ async def stop(self) -> None:+ return None++ async def set_intercept_file_chooser_dialog(self, enabled: bool) -> None:+ return None++ monkeypatch.setattr(recorder_module, "DEMONSTRATION_OWNER_REFRESH_SECONDS", 0.01)+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim("demo-1", "owner-1", False, 0),+ stream=_Stream(),+ )+ recorder.tap = SlowTap() # type: ignore[assignment]+ recorder.page = SimpleNamespace(url="https://example.test/final", is_closed=lambda: False)++ refreshed = _signal_refresh_after(recorder, flush_started)+ stopping = asyncio.create_task(recorder._stop("stopped"))+ await asyncio.wait_for(refreshed.wait(), 5)+ try:+ with pytest.raises(DemonstrationRejection) as exc_info:+ await validate_demonstration_for_replay(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ demonstration_id="demo-1",+ )+ assert exc_info.value.status_code == 409+ finally:+ finish_flush.set()+ await stopping++ record = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert record is not None and record.status == "stopped"+++async def test_record_claim_never_locks_the_browser_index_key_and_keeps_orgs_isolated() -> None:+ cache = _SharedCache()+ claim = await claim_demonstration_recording(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ demonstration_id=None,+ insertion={"after_label": "First", "before_label": None},+ )++ browser_key = demonstration_browser_key("org-1", "pbs-1")+ assert browser_key not in cache.lock_names+ assert f"{browser_key}:lock" in cache.lock_names+ record = await load_demonstration_record(cache, "org-1", claim.demonstration_id) # type: ignore[arg-type]+ assert record is not None and record.insertion == {"after_label": "First", "before_label": None}+ record.status = "stopped"+ record.owner = None+ record.owner_seen_at = None+ record.stop_reason = "disconnected"+ record.final_url = "https://example.test/old-final"+ cache.store[demonstration_record_key("org-1", claim.demonstration_id)] = encode_demonstration_record(record)+ await claim_demonstration_recording(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ demonstration_id=claim.demonstration_id,+ insertion={"after_label": "Overwritten", "before_label": "Last"},+ )+ resumed = await load_demonstration_record(cache, "org-1", claim.demonstration_id) # type: ignore[arg-type]+ assert resumed is not None and resumed.insertion == {"after_label": "First", "before_label": None}+ assert resumed.stop_reason is None+ assert resumed.final_url is None+ assert await load_demonstration_record(cache, "org-2", claim.demonstration_id) is None # type: ignore[arg-type]+ with pytest.raises(DemonstrationRejection) as exc_info:+ await validate_demonstration_for_replay(+ cache, # type: ignore[arg-type]+ organization_id="org-2",+ workflow_permanent_id="wfp-1",+ demonstration_id=claim.demonstration_id,+ )+ assert exc_info.value.status_code == 404+++async def test_stop_control_flushes_focused_typing_before_recorder_commits_stopped() -> None:+ cache = _SharedCache()+ record = DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="recording",+ event_count=0,+ owner="owner-1",+ owner_seen_at=datetime.now(UTC),+ final_url=None,+ )+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(record)+ cache.store[demonstration_controls_key("org-1", "demo-1")] = "[]"++ await append_demonstration_control(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ request=WorkflowCopilotDemonstrationControlRequest(demonstration_id="demo-1", action="stop"),+ )++ stopping = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert stopping is not None and stopping.status == "stopping" and stopping.stop_reason == "stopped"+ page = SimpleNamespace(url="https://example.test/form", is_closed=lambda: False)+ typed = TapEvent(+ page=page,+ session_key="tap-1",+ frame_id="main",+ is_main_frame=True,+ frame_name="",+ payload={+ "kind": "type",+ "ts": 1,+ "url": page.url,+ "selector": "#final",+ "value": "last field value",+ },+ )+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim("demo-1", "owner-1", False, 0),+ stream=_Stream(),+ )+ _FakeTap.events = []+ _FakeTap.flush_events = [typed]+ recorder.tap = _FakeTap(SimpleNamespace()) # type: ignore[assignment]+ recorder.page = page++ await recorder._stop("disconnected")++ stopped = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert stopped is not None and stopped.status == "stopped" and stopped.stop_reason == "stopped"+ event_key = demonstration_event_key("org-1", "demo-1", 1)+ assert json.loads(cache.store[event_key])["value"] == "last field value"+ assert cache.expirations[event_key] == recorder_module.DEMONSTRATION_TTL+++async def test_discard_after_resume_commits_cancelled_even_when_the_stream_drops() -> None:+ cache = _SharedCache()+ cache.store[demonstration_browser_key("org-1", "pbs-1")] = "demo-1"+ cache.store[demonstration_controls_key("org-1", "demo-1")] = "[]"+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ "wfp-1",+ "pbs-1",+ "stopped",+ 2,+ None,+ None,+ "https://example.test/old-final",+ stop_reason="disconnected",+ )+ )+ for seq in (1, 2):+ await cache.set(+ demonstration_event_key("org-1", "demo-1", seq),+ json.dumps({"kind": "type", "value": f"typed {seq}"}),+ ex=recorder_module.DEMONSTRATION_TTL,+ )+ claim = await claim_demonstration_recording(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ demonstration_id="demo-1",+ )+ await append_demonstration_control(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ request=WorkflowCopilotDemonstrationControlRequest(demonstration_id="demo-1", action="cancel"),+ )+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=claim,+ stream=_Stream(),+ )++ await recorder._stop("disconnected")++ record = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert record is not None and record.status == "cancelled" and record.stop_reason == "cancelled"+ assert {+ cache.expirations[key]+ for key in [+ *(demonstration_event_key("org-1", "demo-1", seq) for seq in (1, 2)),+ demonstration_controls_key("org-1", "demo-1"),+ demonstration_record_key("org-1", "demo-1"),+ ]+ } == {recorder_module.DEMONSTRATION_CANCELLED_TTL}+ active = await active_demonstration(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ )+ assert active is None+++async def test_recorder_runs_controls_queued_before_stop(monkeypatch: pytest.MonkeyPatch) -> None:+ cache = _SharedCache()+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord("wfp-1", "pbs-1", "recording", 0, "owner-1", datetime.now(UTC), None)+ )+ cache.store[demonstration_controls_key("org-1", "demo-1")] = json.dumps(+ [{"seq": 1, "action": "navigate", "url": "https://example.test/next"}]+ )+ await append_demonstration_control(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ request=WorkflowCopilotDemonstrationControlRequest(demonstration_id="demo-1", action="stop"),+ )+ page = SimpleNamespace(url="https://example.test/start", goto=AsyncMock(), is_closed=lambda: False)+ browser_state = SimpleNamespace(+ browser_context=SimpleNamespace(pages=[page]),+ get_working_page=AsyncMock(return_value=page),+ )+ _FakeTap.events = []+ _FakeTap.flush_events = []+ monkeypatch.setattr(recorder_module, "PageTap", _FakeTap)+ monkeypatch.setattr(recorder_module, "validate_fetch_url", lambda url: url)+ monkeypatch.setattr(recorder_module, "resolve_persistent_browser_state", AsyncMock(return_value=browser_state))+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim("demo-1", "owner-1", False, 0),+ stream=_Stream(),+ )++ await recorder.run()++ page.goto.assert_awaited_once_with("https://example.test/next")+ record = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert record is not None and record.status == "stopped" and record.event_count == 1+ event = json.loads(cache.store[demonstration_event_key("org-1", "demo-1", 1)])+ assert event["kind"] == "navigate"+++async def test_replay_failure_restores_stopped_record(monkeypatch: pytest.MonkeyPatch) -> None:+ cache = _SharedCache()+ monkeypatch.setattr(app, "CACHE", cache)+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="stopped",+ event_count=1,+ owner=None,+ owner_seen_at=None,+ final_url=None,+ )+ )+ cache.store[demonstration_event_key("org-1", "demo-1", 1)] = json.dumps(+ {+ "seq": 1,+ "kind": "click",+ "selector": "#submit",+ "source_url": "https://example.test/form",+ "frame": {"is_main": True, "name": ""},+ }+ )+ monkeypatch.setattr(replay_module, "resolve_raw_secret_safety_handler", AsyncMock(return_value=None))+ ctx = make_copilot_ctx(organization_id="org-1", workflow_permanent_id="wfp-1")+ ctx.demonstration_id = "demo-1"+ call_tool = AsyncMock(side_effect=RuntimeError("replay failed"))++ with pytest.raises(RuntimeError, match="replay failed"):+ await replay_demonstration(ctx, SimpleNamespace(call_tool=call_tool)) # type: ignore[arg-type]++ record = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert record is not None and record.status == "stopped"+ call_tool.assert_awaited_once_with("click", {"selector": "#submit"})+++async def test_expired_replay_cannot_finalize_a_newer_lease() -> None:+ cache = _SharedCache()+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord("wfp-1", "pbs-1", "stopped", 0, None, None, None)+ )+ ctx = make_copilot_ctx(organization_id="org-1", workflow_permanent_id="wfp-1")++ first = await replay_module._claim_replay(cache, ctx, "demo-1") # type: ignore[arg-type]+ assert first.replay_token is not None+ stale = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert stale is not None+ stale.replay_started_at = datetime.now(UTC) - recorder_module.DEMONSTRATION_REPLAY_LEASE - timedelta(seconds=1)+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(stale)++ second = await replay_module._claim_replay(cache, ctx, "demo-1") # type: ignore[arg-type]+ assert second.replay_token is not None and second.replay_token != first.replay_token+ await replay_module._finish_replay(+ cache, # type: ignore[arg-type]+ ctx,+ "demo-1",+ first.replay_token,+ consumed=True,+ )++ record = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert record is not None and record.status == "replaying" and record.replay_token == second.replay_token++ await replay_module._finish_replay(+ cache, # type: ignore[arg-type]+ ctx,+ "demo-1",+ second.replay_token,+ consumed=True,+ )+ record = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert record is not None and record.status == "consumed" and record.replay_token is None+++ ("disconnected", "page_closed", "max_seconds", "event_count", "typed_value", "expected_reason"),+ [+ (True, False, recorder_module.DEMONSTRATION_MAX_SECONDS, 0, None, "disconnected"),+ (False, True, recorder_module.DEMONSTRATION_MAX_SECONDS, 0, None, "browser_gone"),+ (False, False, 0, 0, None, "time_limit"),+ (False, False, recorder_module.DEMONSTRATION_MAX_SECONDS, 2, None, "event_limit"),+ (False, False, recorder_module.DEMONSTRATION_MAX_SECONDS, 0, "long notes", "event_limit"),+ ],+)+async def test_recorder_stops_for_terminal_reason(+ monkeypatch: pytest.MonkeyPatch,+ disconnected: bool,+ page_closed: bool,+ max_seconds: int,+ event_count: int,+ typed_value: str | None,+ expected_reason: str,+) -> None:+ cache = _SharedCache()+ demonstration_id = f"demo-{expected_reason}"+ owner = "owner-1"+ cache.store[demonstration_record_key("org-1", demonstration_id)] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="recording",+ event_count=0,+ owner=owner,+ owner_seen_at=datetime.now(UTC),+ final_url=None,+ recording_started_at=datetime.now(UTC),+ )+ )+ cache.store[demonstration_controls_key("org-1", demonstration_id)] = "[]"+ page = SimpleNamespace(url="https://example.test/start", is_closed=lambda: page_closed)++ browser_state = SimpleNamespace(+ browser_context=SimpleNamespace(pages=[page]),+ get_working_page=AsyncMock(return_value=page),+ )+ monkeypatch.setattr(recorder_module, "DEMONSTRATION_MAX_SECONDS", max_seconds)+ monkeypatch.setattr(recorder_module, "DEMONSTRATION_MAX_EVENTS", 1 if typed_value is None else 10)+ _FakeTap.events = [+ TapEvent(+ page,+ "tap-1",+ "main",+ True,+ "",+ {"kind": "click", "ts": seq, "url": page.url, "selector": f"#action-{seq}"},+ )+ for seq in range(event_count)+ ]+ if typed_value is not None:+ monkeypatch.setattr(recorder_module, "DEMONSTRATION_MAX_VALUE_CHARS", len(typed_value))+ _FakeTap.events = [+ TapEvent(+ page,+ "tap-1",+ "main",+ True,+ "",+ {"kind": "type", "ts": seq, "url": page.url, "selector": "#notes", "value": typed_value},+ )+ for seq in range(2)+ ]+ _FakeTap.flush_events = []+ monkeypatch.setattr(recorder_module, "PageTap", _FakeTap)+ monkeypatch.setattr(recorder_module, "resolve_persistent_browser_state", AsyncMock(return_value=browser_state))+ stream = _Stream(disconnected=disconnected)+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim(demonstration_id, owner, False, 0),+ stream=stream,+ )++ await recorder.run()++ record = await load_demonstration_record(cache, "org-1", demonstration_id) # type: ignore[arg-type]+ assert record is not None and record.status == "stopped" and record.stop_reason == expected_reason+ assert stream.frames[-1]["reason"] == expected_reason+ if typed_value is not None:+ assert record.event_count == 1+ stored = json.loads(cache.store[demonstration_event_key("org-1", demonstration_id, 1)])+ assert stored["value"] == typed_value+ with pytest.raises(DemonstrationRejection, match="event limit"):+ await claim_demonstration_recording(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ demonstration_id=demonstration_id,+ )+ if expected_reason == "event_limit":+ monkeypatch.setattr(app, "CACHE", cache)+ monkeypatch.setattr(replay_module, "resolve_raw_secret_safety_handler", AsyncMock(return_value=None))+ ctx = make_copilot_ctx(organization_id="org-1", workflow_permanent_id="wfp-1")+ ctx.demonstration_id = demonstration_id+ history = await replay_demonstration(ctx, SimpleNamespace(call_tool=AsyncMock(return_value={"ok": True}))) # type: ignore[arg-type]+ assert any("event limit" in str(item.get("content")) for item in history)+++async def test_resumed_recording_clears_prior_stop_reason_before_replay(monkeypatch: pytest.MonkeyPatch) -> None:+ cache = _SharedCache()+ monkeypatch.setattr(app, "CACHE", cache)+ monkeypatch.setattr(replay_module, "resolve_raw_secret_safety_handler", AsyncMock(return_value=None))+ cache.store[demonstration_browser_key("org-1", "pbs-1")] = "demo-1"+ cache.store[demonstration_controls_key("org-1", "demo-1")] = "[]"+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ "wfp-1",+ "pbs-1",+ "recording",+ 0,+ "owner-1",+ datetime.now(UTC),+ None,+ )+ )+ page = SimpleNamespace(url="https://example.test/start", is_closed=lambda: False)+ browser_state = SimpleNamespace(+ browser_context=SimpleNamespace(pages=[page]),+ get_working_page=AsyncMock(return_value=page),+ )+ _FakeTap.events = []+ _FakeTap.flush_events = []+ monkeypatch.setattr(recorder_module, "PageTap", _FakeTap)+ monkeypatch.setattr(recorder_module, "resolve_persistent_browser_state", AsyncMock(return_value=browser_state))+ first_recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim("demo-1", "owner-1", False, 0),+ stream=_Stream(disconnected=True),+ )+ await first_recorder.run()+ disconnected = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert disconnected is not None and disconnected.stop_reason == "disconnected"++ claim = await claim_demonstration_recording(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ demonstration_id="demo-1",+ )+ await append_demonstration_control(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ request=WorkflowCopilotDemonstrationControlRequest(demonstration_id="demo-1", action="stop"),+ )+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=claim,+ stream=_Stream(),+ )+ await recorder._stop("stopped")+ ctx = make_copilot_ctx(organization_id="org-1", workflow_permanent_id="wfp-1")+ ctx.demonstration_id = "demo-1"++ history = await replay_demonstration(ctx, SimpleNamespace(call_tool=AsyncMock())) # type: ignore[arg-type]++ assert not any("ended because" in str(item.get("content")) for item in history)+++async def test_recorder_switches_to_remaining_page_when_popup_closes(monkeypatch: pytest.MonkeyPatch) -> None:+ cache = _SharedCache()+ cache.store[demonstration_record_key("org-1", "demo-popup")] = encode_demonstration_record(+ DemonstrationRecord(+ "wfp-1", "pbs-1", "recording", 0, "owner-1", datetime.now(UTC), None, recording_started_at=datetime.now(UTC)+ )+ )+ cache.store[demonstration_controls_key("org-1", "demo-popup")] = "[]"+ popup = SimpleNamespace(url="https://auth.example.test", is_closed=lambda: True)+ opener = SimpleNamespace(url="https://example.test/after-auth", is_closed=lambda: False)+ browser_state = SimpleNamespace(+ browser_context=SimpleNamespace(pages=[opener]),+ get_working_page=AsyncMock(return_value=popup),+ )+ _FakeTap.events = []+ _FakeTap.flush_events = []+ monkeypatch.setattr(recorder_module, "PageTap", _FakeTap)+ monkeypatch.setattr(recorder_module, "DEMONSTRATION_MAX_SECONDS", 0)+ monkeypatch.setattr(recorder_module, "resolve_persistent_browser_state", AsyncMock(return_value=browser_state))+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim("demo-popup", "owner-1", False, 0),+ stream=_Stream(),+ )++ await recorder.run()++ record = await load_demonstration_record(cache, "org-1", "demo-popup") # type: ignore[arg-type]+ assert record is not None and record.stop_reason == "time_limit"+ assert record.final_url == opener.url+++async def test_credential_fill_control_never_persists_secret(+ monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture+) -> None:+ sentinel = "credential-secret-sentinel"+ cache = _SharedCache()+ demonstration_id = "demo-credential"+ owner = "owner-1"+ cache.store[demonstration_record_key("org-1", demonstration_id)] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="recording",+ event_count=0,+ owner=owner,+ owner_seen_at=datetime.now(UTC),+ final_url=None,+ )+ )+ credential = _credential()+ monkeypatch.setattr(+ app,+ "DATABASE",+ SimpleNamespace(credentials=SimpleNamespace(get_credentials_by_ids=AsyncMock(return_value=[credential]))),+ )+ page = SimpleNamespace(url="https://example.test/login")++ async def fill(ctx: Any, selector: str, credential_id: str, field: str) -> dict[str, Any]:+ policy = ctx.request_policy+ assert policy.credential_ask_login_page_urls == ["https://example.test/login"]+ assert credential in policy.resolved_credentials+ register_secret_scrub_value(ctx, sentinel)+ ctx.scout_trajectory.append(+ {+ "tool_name": "fill_credential_field",+ "selector": selector,+ "credential_id": credential_id,+ "credential_field": field,+ }+ )+ await recorder._record_tap_event(+ TapEvent(+ page=page,+ session_key="tap-1",+ frame_id="main",+ is_main_frame=True,+ frame_name="",+ payload={"kind": "secret", "selector": selector, "field": field, "url": page.url},+ )+ )+ return {"ok": True, "data": {"readback": sentinel}}++ monkeypatch.setattr(recorder_module, "fill_credential_field", fill)+ stream = _Stream()+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim(demonstration_id, owner, False, 0),+ stream=stream,+ )+ recorder.page = page++ await recorder._fill_credential(+ {+ "action": "fill_credential",+ "credential_id": credential.credential_id,+ "selector": "#password",+ "field": "password",+ }+ )++ event = cache.store[demonstration_event_key("org-1", demonstration_id, 1)]+ assert sentinel not in event+ assert sentinel not in json.dumps(stream.frames)+ assert sentinel not in caplog.text+ stored = json.loads(event)+ assert stored["result"]["data"]["readback"] == "[REDACTED_SECRET]"+ assert stored["scout_fact"]["actor"] == "user"++ await recorder._record_tap_event(+ TapEvent(+ page=page,+ session_key="tap-1",+ frame_id="main",+ is_main_frame=True,+ frame_name="",+ payload={"kind": "secret", "selector": "#password", "field": "password", "url": page.url},+ )+ )+ corrected = await load_demonstration_record(cache, "org-1", demonstration_id) # type: ignore[arg-type]+ assert corrected is not None and corrected.event_count == 2+ assert json.loads(cache.store[demonstration_event_key("org-1", demonstration_id, 2)])["kind"] == "secret"+++async def test_credential_fill_control_refuses_a_credential_bound_to_another_origin(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ cache = _SharedCache()+ cache.store[demonstration_record_key("org-1", "demo-origin")] = encode_demonstration_record(+ DemonstrationRecord("wfp-1", "pbs-1", "recording", 0, "owner-1", datetime.now(UTC), None)+ )+ credential = _credential().model_copy(update={"tested_url": "https://site-a.example/login"})+ monkeypatch.setattr(+ app,+ "DATABASE",+ SimpleNamespace(credentials=SimpleNamespace(get_credentials_by_ids=AsyncMock(return_value=[credential]))),+ )++ async def fill(ctx: Any, selector: str, credential_id: str, field: str) -> dict[str, Any]:+ assert ctx.request_policy.live_page_admitted_urls[credential_id] == credential.tested_url+ return {"ok": False, "error": "origin mismatch"}++ monkeypatch.setattr(recorder_module, "fill_credential_field", fill)+ stream = _Stream()+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim("demo-origin", "owner-1", False, 0),+ stream=stream,+ )+ recorder.page = SimpleNamespace(url="https://site-b.example/login", is_closed=lambda: False)++ await recorder._fill_credential(+ {"credential_id": credential.credential_id, "selector": "#password", "field": "password"}+ )++ record = await load_demonstration_record(cache, "org-1", "demo-origin") # type: ignore[arg-type]+ assert record is not None and record.event_count == 0+ assert any(frame.get("type") == "demonstration_notice" and frame.get("level") == "error" for frame in stream.frames)+++ ("status", "replay_started_at", "owner", "stop_reason"),+ [+ (+ "replaying",+ datetime.now(UTC) - recorder_module.DEMONSTRATION_REPLAY_LEASE - timedelta(seconds=1),+ None,+ None,+ ),+ ("stopping", None, None, "disconnected"),+ ("recording", None, "dead-owner", "disconnected"),+ ],+)+async def test_abandoned_replay_or_stop_is_retryable_and_active(+ status: str, replay_started_at: datetime | None, owner: str | None, stop_reason: str | None+) -> None:+ cache = _SharedCache()+ cache.store[demonstration_browser_key("org-1", "pbs-1")] = "demo-1"+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status=status, # type: ignore[arg-type]+ event_count=0,+ owner=owner,+ owner_seen_at=datetime.now(UTC) - timedelta(minutes=1) if owner else None,+ final_url=None,+ replay_started_at=replay_started_at,+ stop_reason="stopped" if status == "stopping" else None,+ )+ )++ record = await validate_demonstration_for_replay(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ demonstration_id="demo-1",+ )+ active = await active_demonstration(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ )++ assert record.status == "stopped" and record.replay_started_at is None and record.stop_reason == stop_reason+ assert active is not None and active[1].status == "stopped"+++async def test_active_route_returns_live_labels_and_credential_capability(monkeypatch: pytest.MonkeyPatch) -> None:+ from skyvern.forge.sdk.copilot.config import BlockAuthoringPolicy, CopilotConfig+ from skyvern.forge.sdk.routes import workflow_copilot as route_module++ cache = _SharedCache()+ cache.store[demonstration_browser_key("org-1", "pbs-1")] = "demo-1"+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="stopping",+ event_count=2,+ owner="owner-1",+ owner_seen_at=datetime.now(UTC),+ final_url=None,+ )+ )+ cache.store[demonstration_event_key("org-1", "demo-1", 1)] = json.dumps(+ {+ "seq": 1,+ "kind": "type",+ "selector": "#name",+ "value": "their text",+ "label": 'Type "their text"',+ "packet": {"role_name": {"accessible_name": "Full name"}},+ }+ )+ cache.store[demonstration_event_key("org-1", "demo-1", 2)] = json.dumps(+ {"seq": 2, "kind": "secret", "field": "password"}+ )+ monkeypatch.setattr(route_module, "_demonstration_cache", AsyncMock(return_value=cache))+ monkeypatch.setattr(+ app.AGENT_FUNCTION,+ "get_copilot_config_for_request",+ AsyncMock(return_value=CopilotConfig(block_authoring_policy=BlockAuthoringPolicy.CODE_ONLY_BROWSER)),+ )++ result = await route_module.workflow_copilot_demonstration_active(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ organization=SimpleNamespace(organization_id="org-1"), # type: ignore[arg-type]+ )++ assert result is not None+ assert result.status == "stopping"+ assert result.action_labels == ['Type into "Full name"']+ assert result.can_fill_credentials is True+++async def test_recorder_persists_each_capture_gap_kind() -> None:+ cache = _SharedCache()+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="recording",+ event_count=0,+ owner="owner-1",+ owner_seen_at=datetime.now(UTC),+ final_url=None,+ )+ )+ page = SimpleNamespace(url="https://example.test", is_closed=lambda: False)+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim("demo-1", "owner-1", False, 0),+ stream=_Stream(),+ )+ _FakeTap.events = []+ _FakeTap.flush_events = []+ recorder.tap = _FakeTap(SimpleNamespace()) # type: ignore[assignment]+ recorder.tap.frame_path = AsyncMock(+ side_effect=[+ [{"name": "outer"}, {"owner_selector": "#inner"}],+ [{"name": "child"}],+ ]+ )+ events = [+ TapEvent(page, "tap", "nested", False, "", {"kind": "click", "ts": 1, "selector": "#inside"}),+ TapEvent(+ page,+ "tap",+ "child",+ False,+ "child",+ {"kind": "navigate", "ts": 2, "url": "https://example.test/frame"},+ ),+ TapEvent(+ page,+ "tap",+ "main",+ True,+ "",+ {"kind": "unsupported_input", "ts": 3, "url": page.url, "selector": "#range", "value": "7"},+ ),+ ]++ for event in events:+ await recorder._record_tap_event(event)++ stored = [json.loads(cache.store[demonstration_event_key("org-1", "demo-1", seq)]) for seq in range(1, 4)]+ assert [event["reason"] for event in stored] == [+ "nested_frame",+ "frame_navigation",+ "unsupported_event:unsupported_input",+ ]+++async def test_replay_retains_unreadable_stored_event_as_a_gap(monkeypatch: pytest.MonkeyPatch) -> None:+ cache = _SharedCache()+ monkeypatch.setattr(app, "CACHE", cache)+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="stopped",+ event_count=1,+ owner=None,+ owner_seen_at=None,+ final_url=None,+ )+ )+ cache.store[demonstration_event_key("org-1", "demo-1", 1)] = "not-json"+ ctx = make_copilot_ctx(organization_id="org-1", workflow_permanent_id="wfp-1")+ ctx.demonstration_id = "demo-1"++ history = await replay_demonstration(ctx, SimpleNamespace(call_tool=AsyncMock())) # type: ignore[arg-type]++ assert any("recorded event unavailable" in str(item.get("content")) for item in history)+ assert ctx.demonstration_items == ["Step not recorded"]+++async def test_deleted_demonstrated_credential_replays_a_failed_result(monkeypatch: pytest.MonkeyPatch) -> None:+ cache = _SharedCache()+ monkeypatch.setattr(app, "CACHE", cache)+ monkeypatch.setattr(+ app,+ "DATABASE",+ SimpleNamespace(credentials=SimpleNamespace(get_credentials_by_ids=AsyncMock(return_value=[]))),+ )+ event = {+ "seq": 1,+ "kind": "credential_fill",+ "selector": "#password",+ "credential_id": "cred-deleted",+ "field": "password",+ "source_url": "https://example.test/login",+ "frame": {"is_main": True, "name": ""},+ "result": {"ok": True},+ }+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord("wfp-1", "pbs-1", "stopped", 1, None, None, None)+ )+ cache.store[demonstration_event_key("org-1", "demo-1", 1)] = json.dumps(event)+ ctx = make_copilot_ctx(organization_id="org-1", workflow_permanent_id="wfp-1")+ ctx.demonstration_id = "demo-1"++ history = await replay_demonstration(ctx, SimpleNamespace(call_tool=AsyncMock())) # type: ignore[arg-type]++ result = next(item for item in history if item.get("type") == "function_call_output")+ assert json.loads(result["output"])["ok"] is False+ assert ctx.scout_trajectory == []+++async def test_late_recorder_stop_cannot_take_over_a_replaying_record() -> None:+ cache = _SharedCache()+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ "wfp-1",+ "pbs-1",+ "replaying",+ 1,+ "owner-1",+ datetime.now(UTC),+ None,+ stop_reason="stopped",+ replay_started_at=datetime.now(UTC),+ )+ )+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim("demo-1", "owner-1", False, 0),+ stream=_Stream(),+ )++ await recorder._stop("disconnected")++ record = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert record is not None and record.status == "replaying" and record.stop_reason == "stopped"++[email protected]("path", ["discard_stopped", "discard_abandoned", "superseded"])+async def test_every_discard_path_expires_the_record_events_and_controls(path: str) -> None:+ cache = _SharedCache()+ stale = datetime.now(UTC) - timedelta(seconds=recorder_module.DEMONSTRATION_OWNER_LIVE_SECONDS + 1)+ record = DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="stopping" if path == "discard_abandoned" else "stopped",+ event_count=2,+ owner="dead-owner" if path == "discard_abandoned" else None,+ owner_seen_at=stale if path == "discard_abandoned" else None,+ final_url=None,+ stop_reason="cancelled" if path == "discard_abandoned" else "stopped",+ )+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(record)+ cache.store[demonstration_browser_key("org-1", "pbs-1")] = "demo-1"+ cache.store[demonstration_controls_key("org-1", "demo-1")] = "[]"+ for seq in (1, 2):+ cache.store[demonstration_event_key("org-1", "demo-1", seq)] = json.dumps({"kind": "type", "value": "v"})++ if path == "discard_stopped":+ await append_demonstration_control(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ request=WorkflowCopilotDemonstrationControlRequest(demonstration_id="demo-1", action="cancel"),+ )+ elif path == "discard_abandoned":+ active = await active_demonstration(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ )+ assert active is None+ else:+ await claim_demonstration_recording(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ demonstration_id=None,+ )++ keys = [+ demonstration_record_key("org-1", "demo-1"),+ demonstration_controls_key("org-1", "demo-1"),+ *(demonstration_event_key("org-1", "demo-1", seq) for seq in (1, 2)),+ ]+ assert {cache.expirations.get(key) for key in keys} == {recorder_module.DEMONSTRATION_CANCELLED_TTL}+++async def test_resume_after_the_wall_clock_time_limit_is_refused() -> None:+ cache = _SharedCache()+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ status="stopped",+ event_count=1,+ owner=None,+ owner_seen_at=None,+ final_url=None,+ stop_reason="disconnected",+ recording_started_at=datetime.now(UTC) - timedelta(seconds=recorder_module.DEMONSTRATION_MAX_SECONDS + 1),+ )+ )++ with pytest.raises(DemonstrationRejection, match="time limit"):+ await claim_demonstration_recording(+ cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ demonstration_id="demo-1",+ )+++async def test_sensitive_select_is_reported_and_replays_as_a_withheld_selection(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ cache = _SharedCache()+ monkeypatch.setattr(app, "CACHE", cache)+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord("wfp-1", "pbs-1", "recording", 0, "owner-1", datetime.now(UTC), None)+ )+ page = SimpleNamespace(url="https://example.test/form")+ stream = _Stream()+ recorder = DemonstrationRecorder(+ cache=cache, # type: ignore[arg-type]+ organization_id="org-1",+ workflow_permanent_id="wfp-1",+ browser_session_id="pbs-1",+ claim=DemonstrationClaim("demo-1", "owner-1", False, 0),+ stream=stream,+ )++ await recorder._record_tap_event(+ TapEvent(+ page,+ "tap-1",+ "main",+ True,+ "",+ {"kind": "secret_select", "selector": "#plan", "field": "other", "url": page.url},+ )+ )++ assert [frame["label"] for frame in stream.frames if frame["type"] == "demonstration_action"] == [+ "Select a sensitive option"+ ]+ record = await load_demonstration_record(cache, "org-1", "demo-1") # type: ignore[arg-type]+ assert record is not None+ record.status = "stopped"+ record.owner = None+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(record)+ monkeypatch.setattr(replay_module, "resolve_raw_secret_safety_handler", AsyncMock(return_value=None))+ ctx = make_copilot_ctx(organization_id="org-1", workflow_permanent_id="wfp-1")+ ctx.demonstration_id = "demo-1"++ history = await replay_demonstration(ctx, SimpleNamespace(call_tool=AsyncMock())) # type: ignore[arg-type]++ calls = [item for item in history if item.get("type") == "function_call"]+ assert [(item["name"], json.loads(item["arguments"])) for item in calls] == [+ ("select_option", {"selector": "#plan", "value": "<value not recorded>"})+ ]+ assert ctx.demonstration_items == ["Select a sensitive option"]diff --git a/tests/unit/test_copilot_demonstration_tap.py b/tests/unit/test_copilot_demonstration_tap.pynew file mode 100644index 000000000..0f4996b9d--- /dev/null+++ b/tests/unit/test_copilot_demonstration_tap.py@@ -0,0 +1,904 @@+from __future__ import annotations++import asyncio+import time+from types import SimpleNamespace+from typing import Any+from unittest.mock import AsyncMock+from urllib.parse import urlparse++import pytest+from playwright.async_api import async_playwright+from structlog.testing import capture_logs++from skyvern.forge.sdk.copilot import demonstration_tap as tap_module+from skyvern.forge.sdk.copilot.demonstration_tap import PageTap, TapEvent+from tests.unit.copilot_test_helpers import skip_no_browser+from tests.unit.scoped_asyncio import ScopedAsyncio+++def test_page_tap_queue_stops_accepting_events_at_limit() -> None:+ tap = PageTap(SimpleNamespace(), max_events=1) # type: ignore[arg-type]+ first = TapEvent(SimpleNamespace(), "tap-1", "main", True, "", {"kind": "click", "ts": 1})+ second = TapEvent(SimpleNamespace(), "tap-1", "main", True, "", {"kind": "click", "ts": 2})++ assert tap._enqueue_event(first) is True+ assert tap._enqueue_event(second) is False++ assert tap.event_limit_reached is True+ assert tap.drain_events() == [first]+++async def test_page_tap_frame_scan_logs_an_error_and_continues(monkeypatch: pytest.MonkeyPatch) -> None:+ tap = PageTap(SimpleNamespace(), max_events=1) # type: ignore[arg-type]+ scan = AsyncMock(side_effect=[RuntimeError("scan failed"), None])++ async def sleep(_: float) -> None:+ if scan.await_count == 1:+ tap._stopped = True++ monkeypatch.setattr(tap, "_scan_once", scan)+ monkeypatch.setattr(tap_module, "asyncio", ScopedAsyncio(sleep=sleep))++ with capture_logs() as logs:+ await tap.poll_new_frames()++ assert scan.await_count == 2+ assert any(entry["event"] == "copilot_demonstration_frame_scan_failed" for entry in logs)+++async def _wait_for_event(+ tap: PageTap, *, main_frame: bool | None = None, **payload: Any+) -> tuple[TapEvent, list[TapEvent]]:+ seen: list[TapEvent] = []+ for _ in range(300):+ seen.extend(tap.drain_events())+ match = next(+ (+ event+ for event in seen+ if all(event.payload.get(key) == value for key, value in payload.items())+ and (main_frame is None or event.is_main_frame == main_frame)+ ),+ None,+ )+ if match is not None:+ return match, seen+ await asyncio.sleep(0.05)+ raise AssertionError(f"tap event not observed: {[event.payload for event in seen]}")+++@skip_no_browser+async def test_page_tap_captures_trusted_actions_navigation_and_oopif() -> None:+ main_origin = "http://main.test"+ frame_origin = "http://frame.test"+ pages = {+ "/": f"""+ <html><body>+ <input id="autofocus" type="text" autofocus><button id="plain">Press</button>+ <label id="text-label" for="text">Text</label><input id="text" type="text">+ <input id="password" type="password">+ <select id="choice"><option value="a">Alpha</option><option value="b">Beta</option></select><input id="check" type="checkbox">+ <label id="check-label" for="check">Check</label><a id="next" href="/two">Next</a>+ <input id="show-secret" name="login-secret" type="password">+ <button id="show-secret-toggle" onclick="const prior = document.querySelector('#show-secret');+ const replacement = document.createElement('input'); replacement.id = 'shown-password';+ replacement.name = 'visible-value'; replacement.type = 'text'; replacement.value = prior.value;+ prior.replaceWith(replacement)">Show password</button>+ <div id="secret-label-fields">+ <label>Credit or debit card<input id="card" type="text"></label>+ <label>Security code<input id="security-code" type="text"></label>+ <label>API token<input id="api-token" type="text"></label>+ <label>Secret key<input id="secret-key" type="text"></label>+ <label>Auth token<input id="auth-token" type="text"></label>+ <label>Routing number<input id="routing" type="text"></label>+ <label>Account number<input id="account" type="text"></label>+ <label>Social security number<input id="ssn" type="text"></label>+ <label>PIN<input id="pin" type="text"></label>+ <label>Recovery code<input id="recovery" type="text"></label>+ <label>CVV<input id="cvv" type="text"></label>+ <label>CVC<input id="cvc" type="text"></label>+ <label>Card number<input id="card-number" type="text"></label>+ <label>Passcode<input id="passcode" type="text"></label>+ <label>Token<input id="token" type="text"></label>+ <label>Secret<input id="secret" type="text"></label>+ <span>Access key</span><input id="access-key" type="text">+ <label>Refresh token<input id="refresh-token" type="text"></label>+ <label>Card no.<input id="card-no" type="text"></label>+ <label>CVV2<input id="cvv2" type="text"></label>+ <label>Bank account<input id="bank-account" type="text"></label>+ <label>Acct #<input id="acct" type="text"></label>+ <label>Account #<input id="account-hash" type="text"></label>+ <label>Social Security #<input id="social-security" type="text"></label>+ <label>Passport<input id="passport" type="text"></label>+ <label>Tax ID<input id="tax-id" type="text"></label>+ <label>Login code<input id="login-code" type="text"></label>+ <label>Access code<input id="access-code" type="text"></label>+ <label>Security answer<input id="security-answer" type="text"></label>+ <label>Seed phrase<input id="seed-phrase" type="text"></label>+ <label>Recovery phrase<input id="recovery-phrase" type="text"></label>+ <label>IBAN<input id="iban" type="text"></label>+ <label>API key<input id="focus-only-secret" type="text" tabindex="-1"></label>+ </div>+ <iframe id="child" name="child-frame" src="{frame_origin}/frame"></iframe>+ <iframe id="writer" name="writer-frame" src="/writer"></iframe>+ <button id="add-frame" onclick="const frame = document.createElement('iframe'); frame.id = 'late';+ frame.name = 'late-frame'; document.body.append(frame)">Add frame</button>+ <div id="shadow-host"></div>+ <script>+ const root = document.querySelector('#shadow-host').attachShadow({{mode: 'open'}});+ root.innerHTML = '<button id="shadow-button">Shadow button</button>'+ + '<select id="shadow-select"><option value="a">A</option><option value="b">B</option></select>';+ </script>+ </body></html>+ """.encode(),+ "/frame": f'<html><body><button id="frame-button">Frame button</button><iframe id="nested" src="{frame_origin}/nested"></iframe></body></html>'.encode(),+ "/nested": b'<html><body><button id="nested-button">Nested button</button></body></html>',+ "/writer": b'<html><body><button id="before-write">Before</button></body></html>',+ "/late": b'<html><body><button id="late-button">Late frame</button></body></html>',+ "/two": b'<html><body><button id="after">After navigation</button><input id="final"><input id="slider" type="range" min="0" max="10"></body></html>',+ "/submit-form": b'<html><body><form action="/submitted"><input id="submit-value"><button type="submit">Submit</button></form></body></html>',+ "/submitted": b"<html><body>Submitted</body></html>",+ }+ tap: PageTap | None = None+ try:+ async with async_playwright() as playwright:+ browser = await playwright.chromium.launch(+ headless=True,+ args=["--site-per-process", "--use-mock-keychain", "--password-store=basic"],+ )+ context = await browser.new_context()++ async def serve(route: Any) -> None:+ body = pages.get(urlparse(route.request.url).path, b"not found")+ await route.fulfill(+ status=200 if body != b"not found" else 404,+ content_type="text/html",+ body=body,+ )++ await context.route("**/*", serve)+ page = await context.new_page()+ await page.goto(f"{main_origin}/two")+ await page.goto(f"{main_origin}/")+ tap = PageTap(context)+ await tap.start([page])++ assert await page.evaluate("typeof globalThis.__skyvernDemoTap") == "undefined"++ await page.go_back()+ await page.go_forward()+ await page.wait_for_timeout(200)+ assert not any(event.payload.get("kind") == "navigate" for event in tap.drain_events())++ await page.goto(f"{main_origin}/two")+ navigation, _ = await _wait_for_event(tap, kind="navigate")+ assert navigation.payload["url"] == f"{main_origin}/two"+ assert navigation.payload["transition_type"] == "typed"+ await page.go_back()+ await page.wait_for_timeout(200)+ assert not any(event.payload.get("kind") == "navigate" for event in tap.drain_events())++ tap.begin_control_navigation()+ try:+ await page.goto(f"{main_origin}/two")+ finally:+ tap.end_control_navigation()+ await page.wait_for_timeout(200)+ assert not any(event.payload.get("kind") == "navigate" for event in tap.drain_events())+ await page.go_back()++ await page.locator("#autofocus").focus()+ await page.keyboard.type("first")+ await page.keyboard.press("Enter")+ _, autofocus_events = await _wait_for_event(tap, kind="key", key="Enter")+ assert [(event.payload["kind"], event.payload.get("value")) for event in autofocus_events] == [+ ("type", "first"),+ ("key", None),+ ]+ await page.keyboard.type("second")+ await page.keyboard.press("Enter")+ _, restarted_events = await _wait_for_event(tap, kind="key", key="Enter")+ assert [event.payload["kind"] for event in restarted_events] == ["type", "key"]+ assert restarted_events[0].payload["value"] == "firstsecond"++ await page.click("#plain")+ click, _ = await _wait_for_event(tap, kind="click")+ assert click.payload["selector"] == "#plain"+ assert click.payload["packet"]["selector_match_count"] == 1++ await page.click("#text-label")+ await page.keyboard.type("full typed value")+ await page.keyboard.press("Tab")+ typed, typed_events = await _wait_for_event(tap, kind="type")+ if not any(event.payload.get("key") == "Tab" for event in typed_events):+ _, tab_events = await _wait_for_event(tap, kind="key", key="Tab")+ typed_events.extend(tab_events)+ assert typed.payload["value"] == "full typed value"+ assert not any(event.payload.get("kind") == "click" for event in typed_events)+ assert any(+ event.payload.get("kind") == "key" and event.payload.get("key") == "Tab" for event in typed_events+ )++ secret_focus = next((event for event in typed_events if event.payload.get("kind") == "secret_focus"), None)+ if secret_focus is None:+ secret_focus, _ = await _wait_for_event(tap, kind="secret_focus")+ assert secret_focus.payload["selector"] == "#password"+ await page.keyboard.type("do-not-expose")+ await page.keyboard.press("Tab")+ secret, _ = await _wait_for_event(tap, kind="secret", selector="#password")+ assert secret.payload["typed_length"] == len("do-not-expose")++ await page.click("#check-label")+ _, checkbox_events = await _wait_for_event(tap, kind="click", selector="#check-label")+ await page.wait_for_timeout(1100)+ checkbox_events.extend(tap.drain_events())+ assert "do-not-expose" not in repr([event.payload for event in checkbox_events])+ assert not any(event.payload.get("kind") == "secret" for event in checkbox_events)+ assert [+ event.payload.get("selector") for event in checkbox_events if event.payload.get("kind") == "click"+ ] == ["#check-label"]++ tap.drain_events()+ await page.locator("#show-secret").focus()+ await page.keyboard.type("first-secret")+ await page.click("#show-secret-toggle")+ await page.locator("#shown-password").focus()+ await page.keyboard.type("-still-secret")+ await page.keyboard.press("Tab")+ _, toggled_events = await _wait_for_event(tap, kind="secret", selector="#shown-password")+ await page.wait_for_timeout(100)+ toggled_events.extend(tap.drain_events())+ assert "first-secret" not in repr([event.payload for event in toggled_events])+ assert "still-secret" not in repr([event.payload for event in toggled_events])+ assert not any(+ event.payload.get("kind") == "type"+ and event.payload.get("selector") in {"#show-secret", "#shown-password"}+ for event in toggled_events+ )++ secret_selectors = [+ "#card",+ "#security-code",+ "#api-token",+ "#secret-key",+ "#auth-token",+ "#routing",+ "#account",+ "#ssn",+ "#pin",+ "#recovery",+ "#cvv",+ "#cvc",+ "#card-number",+ "#passcode",+ "#token",+ "#secret",+ "#access-key",+ "#refresh-token",+ "#card-no",+ "#cvv2",+ "#bank-account",+ "#acct",+ "#account-hash",+ "#social-security",+ "#passport",+ "#tax-id",+ "#login-code",+ "#access-code",+ "#security-answer",+ "#seed-phrase",+ "#recovery-phrase",+ "#iban",+ ]+ tap.drain_events()+ for selector in secret_selectors:+ await page.locator(selector).focus()+ await page.keyboard.type("hidden-value")+ await page.keyboard.press("Tab")+ _, labelled_secret_events = await _wait_for_event(tap, kind="secret", selector="#passcode")+ await page.wait_for_timeout(100)+ labelled_secret_events.extend(tap.drain_events())+ assert "hidden-value" not in repr([event.payload for event in labelled_secret_events])+ assert set(secret_selectors) <= {+ event.payload.get("selector")+ for event in labelled_secret_events+ if event.payload.get("kind") == "secret"+ }++ tap.drain_events()+ await page.click("#plain")+ await page.click("#focus-only-secret")+ await page.keyboard.press("Tab")+ await tap.flush()+ focus_only_events = tap.drain_events()+ assert any(event.payload.get("kind") == "secret_focus" for event in focus_only_events)+ assert not any(event.payload.get("kind") == "secret" for event in focus_only_events)++ tap.drain_events()+ await page.evaluate(+ """() => {+ document.querySelector('#plain').dispatchEvent(new MouseEvent('click', {bubbles: true}));+ document.querySelector('#choice').dispatchEvent(new Event('change', {bubbles: true}));+ document.dispatchEvent(new KeyboardEvent('keydown', {key: 'Enter', bubbles: true}));+ }"""+ )+ await page.wait_for_timeout(1100)+ assert tap.drain_events() == []++ await page.locator("#shadow-host").locator("#shadow-button").click()+ shadow_click, _ = await _wait_for_event(tap, selector="#shadow-host #shadow-button")+ assert shadow_click.payload["packet"]["selector_match_count"] == 1++ await page.locator("#shadow-host").locator("#shadow-select").focus()+ await page.keyboard.press("b")+ shadow_select, select_events = await _wait_for_event(tap, kind="select")+ await page.wait_for_timeout(1100)+ select_events.extend(tap.drain_events())+ assert shadow_select.payload["value"] == "b"+ assert sum(event.payload.get("kind") == "select" for event in select_events) == 1++ await page.evaluate("document.activeElement.blur()")+ await page.keyboard.press("Escape")+ escape, _ = await _wait_for_event(tap, kind="key", key="Escape")+ assert "selector" not in escape.payload+ assert "packet" not in escape.payload++ frame = next(frame for frame in page.frames if frame.name == "child-frame")+ await frame.click("#frame-button")+ frame_click, _ = await _wait_for_event(tap, main_frame=False, kind="click")+ assert await tap.frame_owner_selector(frame_click) == "#child"+ nested = next(frame for frame in page.frames if frame.url.endswith("/nested"))+ await nested.click("#nested-button")+ nested_click, _ = await _wait_for_event(tap, main_frame=False, selector="#nested-button")+ assert await tap.frame_path(nested_click) == [{"name": "child-frame"}, {"owner_selector": "#nested"}]++ writer = next(frame for frame in page.frames if frame.name == "writer-frame")+ await writer.evaluate(+ """() => {+ document.open();+ document.write('<button id="after-write">After</button>');+ document.close();+ }"""+ )+ await page.wait_for_timeout(1200)+ await writer.click("#after-write")+ await _wait_for_event(tap, selector="#after-write")++ await page.click("#add-frame")+ late = next(frame for frame in page.frames if frame.name == "late-frame")+ for _ in range(40):+ if any(frame is late and url == "about:blank" for frame, url in tap._refused_frames):+ break+ await asyncio.sleep(0.05)+ else:+ raise AssertionError("about:blank frame was not observed as in-process")+ await page.evaluate(f"document.querySelector('#late').src = '{frame_origin}/late'")+ await late.wait_for_url("**/late")+ await page.wait_for_timeout(700)+ await late.click("#late-button")+ await _wait_for_event(tap, main_frame=False, selector="#late-button")++ await page.locator("#next").evaluate(+ "element => element.addEventListener('auxclick', event => event.preventDefault(), {once: true})"+ )+ tap.drain_events()+ await page.locator("#next").click(button="middle")+ await page.locator("#next").click(button="right")+ await page.keyboard.press("Escape")+ await page.wait_for_timeout(700)+ assert not any(+ event.payload.get("kind") == "click" and event.payload.get("selector") == "#next"+ for event in tap.drain_events()+ )++ await page.click("#next")+ await page.wait_for_url("**/two")+ navigating_click, link_events = await _wait_for_event(tap, kind="click", selector="#next")+ await page.wait_for_timeout(200)+ link_events.extend(tap.drain_events())+ assert navigating_click.payload["url"].endswith("/")+ assert (+ sum(+ event.payload.get("kind") == "click" and event.payload.get("selector") == "#next"+ for event in link_events+ )+ == 1+ )+ assert not any(event.payload.get("kind") == "navigate" for event in link_events)+ assert await page.evaluate("typeof globalThis.__skyvernDemoTap") == "undefined"++ await page.click("#after")+ await _wait_for_event(tap, kind="click", selector="#after")+ tap.drain_events()+ await page.click("#slider")+ await page.keyboard.press("ArrowRight")+ await page.keyboard.press("Tab")+ slider, slider_events = await _wait_for_event(tap, kind="unsupported_input", selector="#slider")+ assert int(slider.payload["value"]) > 5+ assert not any(+ event.payload.get("kind") == "click" and event.payload.get("selector") == "#slider"+ for event in slider_events+ )++ tap.begin_control_navigation()+ try:+ await page.goto(f"{main_origin}/submit-form")+ finally:+ tap.end_control_navigation()+ await page.wait_for_timeout(200)+ tap.drain_events()+ await page.click("#submit-value")+ await page.keyboard.type("final answer")+ await page.keyboard.press("Enter")+ await page.wait_for_url("**/submitted*")+ _, submit_events = await _wait_for_event(tap, kind="key", key="Enter")+ assert [event.payload.get("kind") for event in submit_events] == ["type", "key"]+ assert submit_events[0].payload["value"] == "final answer"++ tap.begin_control_navigation()+ try:+ await page.goto(f"{main_origin}/two")+ finally:+ tap.end_control_navigation()+ tap.drain_events()+ await page.click("#final")+ await page.keyboard.type("last field value")+ await tap.stop()+ final_events = tap.drain_events()+ assert [event.payload.get("value") for event in final_events if event.payload.get("kind") == "type"] == [+ "last field value"+ ]+ await page.click("#after")+ await page.wait_for_timeout(1200)+ assert tap.drain_events() == []+ await tap.stop()++ tap = PageTap(context)+ await tap.start([page])+ await page.click("#after")+ await _wait_for_event(tap, kind="click", selector="#after")+ await browser.close()+ finally:+ if tap is not None:+ await tap.stop()+++@skip_no_browser+async def test_page_tap_protects_labelled_secrets_and_captures_combobox_typing() -> None:+ tap: PageTap | None = None+ async with async_playwright() as playwright:+ browser = await playwright.chromium.launch(headless=True)+ context = await browser.new_context()+ page = await context.new_page()+ await page.set_content(+ """+ <label for="api-key">API key</label><input id="api-key">+ <label>Password <input id="wrapped-secret"></label>+ <span id="token-label">Access token</span><input id="aria-secret" aria-labelledby="token-label">+ <span>API key</span><span class="material-icons">visibility</span><svg aria-hidden="true"></svg><input id="icon-field">+ <div><label>Access key</label></div><div><input id="wrapper-field"></div>+ <h4>Secret key</h4><input id="heading-field">+ <dl><dt>API token</dt><dd><input id="dt-field"></dd></dl>+ <section>+ <label>Card number</label>+ <div><input id="card-1"><input id="card-2"><input id="card-3"><input id="card-4"></div>+ </section>+ <form><input type="hidden" name="authenticity_token" value="csrf-page-value"></form>+ <div><h2>Forgot your password?</h2><form><input id="reset-email" type="email"></form></div>+ <div>+ <h1>Reset password</h1>+ <div class="form-floating">+ <input id="floating-email" placeholder="[email protected]"><label for="floating-email">Email address</label>+ </div>+ </div>+ <div><h2>Two-factor authentication</h2><div><input id="tfa-phone" placeholder="Phone number"></div></div>+ <input id="early-toggle" type="password">+ <div><span>API key</span><input id="aria-labelled-email" aria-label="Email"></div>+ <div><span>Access token</span><form><input id="boundary-name" placeholder="Name"></form></div>+ <table><tr><td>Secret key</td><td><input id="table-field"></td></tr></table>+ <input id="password" type="password">+ <div id="page-copied-toggle">+ <input id="copied-source" type="password">+ <input id="copied-plain" type="text" hidden>+ </div>+ <div id="two-input-toggle">+ <input id="toggle-password" type="password">+ <input id="toggle-visible" type="text" hidden>+ </div>+ <section><label>API key</label><button>Copy</button><input id="copy-field"></section>+ <section>+ <div><label>API token</label><input id="api-token-row"></div>+ <div><input id="placeholder-name" placeholder="Full name"></div>+ </section>+ <section>+ <div><label>Card number</label><input id="card-row"></div>+ <div><input id="city" placeholder="City"></div>+ </section>+ <section>+ <p>Questions about your account number or PIN? Call us any time.</p>+ <input id="search-bills" placeholder="Search bills">+ </section>+ <div id="secret-shadow-host"></div>+ <script>+ document.querySelector('#secret-shadow-host').attachShadow({mode: 'open'}).innerHTML =+ '<label>API key <input id="shadow-secret"></label>';+ </script>+ <section id="identifier-first">+ <input id="username-first" type="text">+ <input id="hidden-login-password" type="password" hidden>+ </section>+ <label for="secret-select">API key</label>+ <select id="secret-select"><option value="">Choose</option><option value="select-sensitive-value">Secret</option></select>+ <input id="normal">+ <input id="combo" role="combobox" aria-controls="suggestions">+ <div id="suggestions"><button id="suggestion">John Smith</button></div>+ """+ )+ try:+ tap = PageTap(context)+ await tap.start([page])++ for selector, value in (+ ("#api-key", "api-sensitive-value"),+ ("#wrapped-secret", "wrapped-sensitive-value"),+ ("#aria-secret", "aria-sensitive-value"),+ ("#icon-field", "icon-sensitive-value"),+ ("#wrapper-field", "wrapper-sensitive-value"),+ ("#table-field", "table-sensitive-value"),+ ("#copy-field", "copy-sensitive-value"),+ ("#heading-field", "heading-sensitive-value"),+ ("#dt-field", "dt-sensitive-value"),+ ("#card-2", "box-sensitive-4242"),+ ("#tfa-phone", "tfa-sensitive-5550100"),+ ("#boundary-name", "boundary-sensitive-value"),+ ):+ await page.locator(selector).focus()+ await page.keyboard.type(value)+ await page.keyboard.press("Tab")++ await page.locator("#password").focus()+ await page.keyboard.press("Control+Alt+@")+ await page.keyboard.press("Shift+A")+ await page.keyboard.type("password-sensitive-value")+ await page.keyboard.press("Tab")++ await page.locator("#copied-source").focus()+ await page.keyboard.type("page-copied-value")+ await page.keyboard.press("Tab")+ await page.evaluate(+ """() => {+ const visible = document.querySelector('#copied-plain');+ visible.value = document.querySelector('#copied-source').value;+ visible.hidden = false;+ }"""+ )+ await page.locator("#toggle-password").focus()+ await page.keyboard.type("copied-password-value")+ await page.keyboard.press("Tab")+ await page.evaluate(+ """() => {+ const password = document.querySelector('#toggle-password');+ const visible = document.querySelector('#toggle-visible');+ password.hidden = true;+ visible.hidden = false;+ visible.value = password.value;+ }"""+ )+ await page.locator("#toggle-visible").focus()+ await page.keyboard.type("-corrected")+ await page.keyboard.press("Tab")+ await page.locator("#secret-shadow-host").locator("#shadow-secret").focus()+ await page.keyboard.type("shadow-sensitive-value")+ await page.keyboard.press("Tab")+ await page.evaluate("() => { document.querySelector('#early-toggle').type = 'text'; }")+ await page.locator("#early-toggle").focus()+ await page.keyboard.type("early-toggle-value")+ await page.keyboard.press("Tab")+ await page.evaluate(+ """() => {+ const late = Object.assign(document.createElement('input'), {id: 'late-toggle', type: 'password'});+ document.body.append(late);+ const host = document.createElement('div');+ host.id = 'late-shadow-host';+ document.body.append(host);+ host.attachShadow({mode: 'open'}).innerHTML = '<div><input id="shadow-toggle" type="password"></div>';+ }"""+ )+ await page.evaluate(+ """() => {+ document.querySelector('#late-toggle').type = 'text';+ document.querySelector('#late-shadow-host').shadowRoot.querySelector('#shadow-toggle').type = 'text';+ }"""+ )+ await page.evaluate(+ """() => {+ const host = document.createElement('div');+ host.id = 'lazy-shadow-host';+ document.body.append(host);+ }"""+ )+ await page.evaluate(+ """() => {+ const root = document.querySelector('#lazy-shadow-host').attachShadow({mode: 'open'});+ root.innerHTML = '<input id="lazy-toggle" type="password"><button id="lazy-show">Show</button>';+ root.querySelector('#lazy-show').addEventListener('click', () => {+ root.querySelector('#lazy-toggle').type = 'text';+ });+ }"""+ )+ await page.locator("#lazy-shadow-host").locator("#lazy-show").click()+ for locator, value in (+ (page.locator("#lazy-shadow-host").locator("#lazy-toggle"), "lazy-toggle-value"),+ (page.locator("#late-toggle"), "late-toggle-value"),+ (page.locator("#late-shadow-host").locator("#shadow-toggle"), "shadow-toggle-value"),+ ):+ await locator.focus()+ await page.keyboard.type(value)+ await page.keyboard.press("Tab")+ ordinary_fields = {+ "#reset-email": "[email protected]",+ "#floating-email": "[email protected]",+ "#aria-labelled-email": "[email protected]",+ "#placeholder-name": "Taylor Example",+ "#city": "Springfield",+ "#search-bills": "electric",+ "#username-first": "[email protected]",+ }+ for selector, value in ordinary_fields.items():+ await page.locator(selector).focus()+ await page.keyboard.type(value)+ await page.keyboard.press("Tab")++ await page.locator("#secret-select").focus()+ await page.keyboard.press("s")++ await page.locator("#normal").focus()+ await page.keyboard.press("Enter")++ await page.locator("#combo").focus()+ await page.keyboard.type("John")+ await page.click("#suggestion")+ _, events = await _wait_for_event(tap, kind="click", selector="#suggestion")+ await tap.flush()+ events.extend(tap.drain_events())+ payloads = [event.payload for event in events]++ secret_selectors = {payload.get("selector") for payload in payloads if payload.get("kind") == "secret"}+ assert {+ "#api-key",+ "#wrapped-secret",+ "#aria-secret",+ "#icon-field",+ "#wrapper-field",+ "#table-field",+ "#heading-field",+ "#dt-field",+ "#card-2",+ "#tfa-phone",+ "#boundary-name",+ "#password",+ "#toggle-password",+ "#toggle-visible",+ "#copy-field",+ "#early-toggle",+ "#late-toggle",+ } <= secret_selectors+ assert not secret_selectors & set(ordinary_fields)+ assert {+ (payload.get("selector"), payload.get("value")) for payload in payloads if payload.get("kind") == "type"+ } >= set(ordinary_fields.items())+ await tap.stop()+ assert await page.locator("#api-key").input_value() == ""+ assert await page.locator("#secret-select").input_value() == ""+ assert await page.locator("#secret-shadow-host").locator("#shadow-secret").input_value() == ""+ assert await page.locator("#city").input_value() == "Springfield"+ assert await page.locator("#copied-plain").input_value() == ""+ assert await page.locator("input[name=authenticity_token]").input_value() == "csrf-page-value"+ assert not any(+ payload.get("kind") == "type" and payload.get("selector") in secret_selectors for payload in payloads+ )+ assert not any(+ payload.get("kind") == "key" and payload.get("selector") == "#password" for payload in payloads+ )+ assert not any(payload.get("key") in {"Control+Alt+@", "Shift+A"} for payload in payloads)+ assert any(+ payload.get("kind") == "secret_select" and payload.get("selector") == "#secret-select"+ for payload in payloads+ )+ assert any(payload.get("kind") == "key" and payload.get("key") == "Enter" for payload in payloads)+ assert all(+ value not in repr(payloads)+ for value in (+ "api-sensitive-value",+ "wrapped-sensitive-value",+ "aria-sensitive-value",+ "icon-sensitive-value",+ "wrapper-sensitive-value",+ "copy-sensitive-value",+ "shadow-sensitive-value",+ "table-sensitive-value",+ "heading-sensitive-value",+ "dt-sensitive-value",+ "box-sensitive-4242",+ "tfa-sensitive-5550100",+ "boundary-sensitive-value",+ "early-toggle-value",+ "late-toggle-value",+ "shadow-toggle-value",+ "lazy-toggle-value",+ "password-sensitive-value",+ "copied-password-value",+ "corrected",+ "select-sensitive-value",+ )+ )+ assert [+ (payload["kind"], payload.get("selector"), payload.get("value"))+ for payload in payloads+ if payload.get("selector") in {"#combo", "#suggestion"}+ ] == [("type", "#combo", "John"), ("click", "#suggestion", None)]+ finally:+ if tap is not None:+ await tap.stop()+ await browser.close()+++@skip_no_browser+async def test_page_tap_names_field_kind_only_from_type_and_autocomplete() -> None:+ tap: PageTap | None = None+ async with async_playwright() as playwright:+ browser = await playwright.chromium.launch(headless=True)+ page = await (await browser.new_context()).new_page()+ await page.set_content(+ """+ <div><input id="masked-onetime" type="password" autocomplete="one-time-code"></div>+ <div><input id="masked-csc" type="password" autocomplete="cc-csc"></div>+ <div><input id="masked-four" type="password" name="pin"></div>+ <div><input id="login-box" type="password" autocomplete="current-password"></div>+ <div><input id="autofill-off" name="company" autocomplete="new-password"></div>+ <div><input id="tel-reveal" type="password"></div>+ <div><input id="reused-box" type="password" placeholder="Email"></div>+ <div><input id="confirm-box" type="text"></div>+ <div><h3>Security code</h3><div><input id="w1"></div></div>+ <div><input id="w2" name="securityCode"></div>+ <fieldset><legend>Password</legend><div><input id="w3"></div></fieldset>+ <div><h2>API token</h2><div class="input-group"><input id="w4"><button>Copy</button></div></div>+ <div><h4>PIN</h4><div><input id="w5"></div></div>+ <div><h2>Enter your one-time code</h2><form><input id="w6" name="entry"></form></div>+ <div><span>CVV</span><form><input id="w7"></form></div>+ <div><span>PIN</span><input id="w8" aria-labelledby="missing-node"></div>+ <div><span>PIN</span><input id="w9"><label for="w9"></label></div>+ <div>+ <h2>Enter verification code</h2>+ <div><input id="d1" aria-label="Digit 1"><input id="d2" aria-label="Digit 2"><input id="d3"></div>+ </div>+ <form><h2>One-time code</h2><div><input id="w10" placeholder="Enter here"></div></form>+ <div>+ <h3>Verification code</h3><p>We sent a six-digit number to the phone you added earlier.</p>+ <div><input id="w11"></div>+ </div>+ <div><p>Account number: 1234</p><h3>Change email</h3><div><input id="o1" type="email"></div></div>+ <div><span>Password</span><h2>Contact details</h2><div><input id="o2" placeholder="Full name"></div></div>+ <form><h2>Forgot your password?</h2><input id="o3" type="email"></form>+ <div id="swap-wrap"><input id="swap-box" type="password"></div>+ <div id="rerender"><div><input id="rr-box" type="password"><button>Show</button></div></div>+ <div id="delayed"><input id="delayed-box" type="password"></div>+ <div id="lazy-host"></div><button id="outside-show">Show</button>+ <button id="done-marker">Done</button>+ """+ )+ try:+ tap = PageTap(page.context)+ await tap.start([page])+ await page.evaluate(+ """() => {+ document.querySelector('#reused-box').type = 'email';+ document.querySelector('#tel-reveal').type = 'tel';+ document.querySelector('#swap-box').replaceWith(+ Object.assign(document.createElement('input'), {id: 'swap-shown', type: 'text'}));+ document.querySelector('#rerender').innerHTML =+ '<div><input id="rr-shown" type="text"><button>Hide</button></div>';+ document.querySelector('#delayed-box').remove();+ const root = document.querySelector('#lazy-host').attachShadow({mode: 'open'});+ root.innerHTML = '<input id="lazy-box" type="password">';+ document.querySelector('#outside-show').addEventListener('click', () => {+ root.querySelector('#lazy-box').type = 'text';+ });+ }"""+ )+ await page.evaluate(+ """() => document.querySelector('#delayed').append(+ Object.assign(document.createElement('input'), {id: 'delayed-shown', type: 'text'}))"""+ )+ await page.click("#outside-show")++ secret_fields = {+ "#masked-onetime": "totp",+ "#masked-csc": "other",+ "#masked-four": "other",+ "#login-box": "password",+ "#autofill-off": "other",+ "#tel-reveal": "other",+ "#swap-shown": "other",+ "#rr-shown": "other",+ "#delayed-shown": "other",+ **{f"#w{index}": "other" for index in range(1, 12)},+ "#d2": "other",+ "#d3": "other",+ }+ ordinary_fields = {+ "#reused-box": "[email protected]",+ "#confirm-box": "[email protected]",+ "#o1": "[email protected]",+ "#o2": "Taylor Example",+ "#o3": "[email protected]",+ }+ for selector in secret_fields:+ await page.locator(selector).focus()+ await page.keyboard.type(f"withheld{selector[1:]}")+ await page.keyboard.press("Tab")+ await page.locator("#lazy-host").locator("#lazy-box").focus()+ await page.keyboard.type("withheld-lazy")+ await page.keyboard.press("Tab")+ for selector, value in ordinary_fields.items():+ await page.locator(selector).focus()+ await page.keyboard.type(value)+ await page.keyboard.press("Tab")+ await page.click("#done-marker")+ _, events = await _wait_for_event(tap, kind="click", selector="#done-marker")+ await tap.flush()+ events.extend(tap.drain_events())+ payloads = [event.payload for event in events]++ secrets = {payload["selector"]: payload for payload in payloads if payload.get("kind") == "secret"}+ assert {selector: secrets[selector]["field"] for selector in secret_fields if selector in secrets} == (+ secret_fields+ )+ assert any(selector.endswith("#lazy-box") for selector in secrets)+ assert secrets["#d2"]["field_label"] == "Digit 2"+ assert secrets["#w10"]["field_label"] == "Enter here"+ typed = {+ (payload.get("selector"), payload.get("value")) for payload in payloads if payload.get("kind") == "type"+ }+ assert typed >= set(ordinary_fields.items())+ assert "withheld" not in repr([payload for payload in payloads if payload.get("kind") != "secret"])+ assert "withheld" not in repr(secrets)+ finally:+ if tap is not None:+ await tap.stop()+ await browser.close()+++@skip_no_browser+async def test_page_tap_click_snapshot_is_bounded_on_large_pages() -> None:+ tap: PageTap | None = None+ async with async_playwright() as playwright:+ browser = await playwright.chromium.launch(headless=True)+ context = await browser.new_context()+ page = await context.new_page()+ links = '<a href="#">Same name</a>' * 10_000+ await page.set_content(f'<a id="target" href="#">Same name</a>{links}')+ try:+ tap = PageTap(context)+ await tap.start([page])+ box = await page.locator("#target").bounding_box()+ assert box is not None+ await page.mouse.move(box["x"] + box["width"] / 2, box["y"] + box["height"] / 2)++ started = time.monotonic()+ await page.mouse.down()+ elapsed = time.monotonic() - started+ await page.mouse.up()++ await _wait_for_event(tap, kind="click", selector="#target")+ assert elapsed < 0.5+ finally:+ if tap is not None:+ await tap.stop()+ await browser.close()diff --git a/tests/unit/test_copilot_failure_publication.py b/tests/unit/test_copilot_failure_publication.pyindex 178f30101..1bf9795a1 100644--- a/tests/unit/test_copilot_failure_publication.py+++ b/tests/unit/test_copilot_failure_publication.py@@ -24,7 +24,6 @@ from skyvern.forge.sdk.copilot.tools.run_execution import (_ExecutionResult,_record_run_blocks_result,_recorded_run_block_result,- _run_blocks_and_collect_debug,_RunExecution,finalize_build_test_result,)@@ -430,6 +429,7 @@ async def test_successful_self_heal_never_starts_failure_capture(monkeypatch: pyresult = await block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=RuntimeError("initial failure"),failing_line=1,build_failure_result=AsyncMock(),@@ -506,6 +506,7 @@ async def test_outer_deadline_during_staging_still_publishes_the_healable_failurtask = asyncio.create_task(block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=RuntimeError("initial failure"),failing_line=1,build_failure_result=build_failure_result,@@ -581,6 +582,7 @@ async def test_healable_failure_is_not_held_behind_a_stalled_frame(result = await asyncio.wait_for(block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=RuntimeError("initial failure"),failing_line=1,build_failure_result=AsyncMock(),@@ -642,6 +644,7 @@ async def test_failed_self_heal_is_not_persisted_twice_before_capture(monkeypatcresult = await block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=RuntimeError("initial failure"),failing_line=1,build_failure_result=AsyncMock(),@@ -1666,11 +1669,6 @@ async def test_real_soft_and_hard_deadlines_persist_and_emit_held_production_faiassert packet["failure"]["error_codes"] == ["browser_operation_failed"]assert persisted_row.output == {"failure": "specific runner failure"}- dispatch_lookup = AsyncMock()- monkeypatch.setattr(- "skyvern.forge.sdk.copilot.tools.run_execution.app.DATABASE.workflows.get_workflow_by_permanent_id",- dispatch_lookup,- )first_result = SimpleNamespace(context_wrapper=SimpleNamespace(usage=None), cancel=MagicMock())unfinished_report = SimpleNamespace(context_wrapper=SimpleNamespace(usage=None),@@ -1696,8 +1694,6 @@ async def test_real_soft_and_hard_deadlines_persist_and_emit_held_production_faireturnassert copilot_context.last_run_outcome is outcomeassert workflow_context.has_failure_evidence_capture is True- denial = await _run_blocks_and_collect_debug({"block_labels": ["run_code"]}, copilot_context)- assert denial["data"] == {"budget_expired": True, "run_dispatched": False, "source": "deadline"}report_started.set()try:await asyncio.Event().wait()@@ -1732,7 +1728,6 @@ async def test_real_soft_and_hard_deadlines_persist_and_emit_held_production_faiassert copilot_context.budget_expiry_state.drain_attempted is Trueassert copilot_context.budget_expiry_state.hard_backstop_reached is Trueassert unfinished_report.final_output is None- dispatch_lookup.assert_not_awaited()assert workflow_context.has_failure_evidence_capture is Truehard_result = _build_timeout_exit_result(copilot_context, global_llm_context=None)@@ -1789,7 +1784,6 @@ async def test_real_soft_and_hard_deadlines_persist_and_emit_held_production_faiassert emitted.narrative_payload["turnFacts"]["recordedFailure"] == "specific runner failure"assert emitted.narrative_payload["turnFacts"]["runId"] == "wr_test"assert emitted.narrative_payload["turnFacts"]["evaluationState"] == "not_demonstrated"- dispatch_lookup.assert_not_awaited()assert workflow_context.has_failure_evidence_capture is Truecapture_release.set()diff --git a/tests/unit/test_copilot_fill_credential_field.py b/tests/unit/test_copilot_fill_credential_field.pyindex d2d9d0135..b3aaa9820 100644--- a/tests/unit/test_copilot_fill_credential_field.py+++ b/tests/unit/test_copilot_fill_credential_field.py@@ -469,7 +469,7 @@ async def test_the_readback_comes_from_the_page_not_the_tool_layer(ctx = _ctx()selector = f"#{field}"- result = await tools_module._fill_credential_field_impl(ctx, selector, "cred_123", field)+ result = await tools_module.fill_credential_field(ctx, selector, "cred_123", field)assert result["ok"] is Trueassert result["data"]["readback_outcome"] == "exact_match"@@ -556,7 +556,7 @@ async def test_target_identity_is_captured_before_fill_and_effect_afterward(monkmonkeypatch.setattr(credential_fill_module, "_probe_target_facts", probe)ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert result["ok"] is Trueassert events.index("probe") < events.index("fill") < events.index("post_effect")@@ -589,7 +589,7 @@ async def test_multi_match_selector_still_reaches_an_outcome(monkeypatch: pytest_wire_impl(monkeypatch, page, secret_value="123456")ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "input.otp", "cred_123", "totp")+ result = await tools_module.fill_credential_field(ctx, "input.otp", "cred_123", "totp")assert result["ok"] is Trueassert result["data"]["readback_outcome"] == "different"@@ -611,7 +611,7 @@ async def test_a_field_not_holding_what_was_typed_is_reported_without_failing_or_wire_impl(monkeypatch, page, secret_value=secret_value)ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#totp", "cred_123", "totp")+ result = await tools_module.fill_credential_field(ctx, "#totp", "cred_123", "totp")assert result["ok"] is Trueassert result["data"]["readback_outcome"] == "different"@@ -634,7 +634,7 @@ async def test_unreadable_field_records_the_fill_and_says_so(monkeypatch: pytestctx = _ctx()with capture_logs() as logs:- result = await tools_module._fill_credential_field_impl(ctx, "#totp", "cred_123", "totp")+ result = await tools_module.fill_credential_field(ctx, "#totp", "cred_123", "totp")assert result["ok"] is Trueassert result["data"]["readback_outcome"] == "unavailable"@@ -653,7 +653,7 @@ async def test_happy_path_fills_and_records_value_free(monkeypatch: pytest.Monke_wire_impl(monkeypatch, page)ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert result["ok"] is Trueassert page.fill_calls == [("#passwordInput", _FAKE_PASSWORD)]@@ -692,7 +692,7 @@ async def test_email_otp_scout_failure_returns_only_factual_credential_identity(monkeypatch.setattr(credential_fill_module, "_resolve_credential_fill_value", runtime_only_otp)- result = await tools_module._fill_credential_field_impl(_ctx(), "#otp", "cred_123", "totp")+ result = await tools_module.fill_credential_field(_ctx(), "#otp", "cred_123", "totp")assert result["ok"] is Falseassert result["data"] == {@@ -708,7 +708,7 @@ async def test_fill_error_text_is_scrubbed(monkeypatch: pytest.MonkeyPatch) -> Npage = _FakePage(fill_error=RuntimeError(f"could not type {_FAKE_PASSWORD} into element"))_wire_impl(monkeypatch, page)- result = await tools_module._fill_credential_field_impl(_ctx(), "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(_ctx(), "#passwordInput", "cred_123", "password")assert result["ok"] is Falseassert _FAKE_PASSWORD not in result["error"]@@ -720,7 +720,7 @@ async def test_rejects_unknown_field(monkeypatch: pytest.MonkeyPatch) -> None:page = _FakePage()_wire_impl(monkeypatch, page)- result = await tools_module._fill_credential_field_impl(_ctx(), "#cvv", "cred_123", "cvv")+ result = await tools_module.fill_credential_field(_ctx(), "#cvv", "cred_123", "cvv")assert result["ok"] is Falseassert "username, password, totp" in result["error"]@@ -732,7 +732,7 @@ async def test_rejects_empty_selector(monkeypatch: pytest.MonkeyPatch) -> None:page = _FakePage()_wire_impl(monkeypatch, page)- result = await tools_module._fill_credential_field_impl(_ctx(), " ", "cred_123", "password")+ result = await tools_module.fill_credential_field(_ctx(), " ", "cred_123", "password")assert result["ok"] is Falseassert page.fill_calls == []@@ -745,7 +745,7 @@ async def test_unresolved_credential_never_reaches_vault_or_page(monkeypatch: pyresolver = AsyncMock()monkeypatch.setattr(credential_fill_module, "_resolve_credential_fill_value", resolver)- result = await tools_module._fill_credential_field_impl(_ctx(), "#passwordInput", "cred_999", "password")+ result = await tools_module.fill_credential_field(_ctx(), "#passwordInput", "cred_999", "password")assert result["ok"] is Falseassert "cred_999" in result["error"]@@ -759,7 +759,7 @@ async def test_agent_blocks_only_is_rejected(monkeypatch: pytest.MonkeyPatch) ->_wire_impl(monkeypatch, page)ctx = _ctx(block_authoring_policy=BlockAuthoringPolicy.TASK_V3_PURE)- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert result["ok"] is Falseassert page.fill_calls == []@@ -772,7 +772,7 @@ async def test_readback_failure_surfaces_and_skips_recording(monkeypatch: pytestctx = _ctx()with capture_logs() as logs:- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert result["ok"] is Falseassert "still empty" in result["error"]@@ -800,7 +800,7 @@ async def test_a_field_cleared_by_its_own_submit_is_not_reported_as_a_lost_fill(monkeypatch.setattr(credential_fill_module, "_live_working_page_url", navigated)ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#totpCode", "cred_123", "totp")+ result = await tools_module.fill_credential_field(ctx, "#totpCode", "cred_123", "totp")# The form committed on the last digit and cleared its own field. Calling that a lost fill# sends the model back to re-type a code into a page the sign-in has already left.@@ -829,7 +829,7 @@ async def test_a_navigation_cannot_stand_in_for_a_readback_nobody_took(monkeypatmonkeypatch.setattr(credential_fill_module, "_live_working_page_url", navigated)ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#totpCode", "cred_123", "totp")+ result = await tools_module.fill_credential_field(ctx, "#totpCode", "cred_123", "totp")assert result["ok"] is Trueassert result["data"]["readback_outcome"] == "unavailable"@@ -847,7 +847,7 @@ async def test_a_rejected_code_re_rendering_the_same_page_is_still_a_lost_fill(mmonkeypatch.setattr(credential_fill_module, "_live_working_page_url", same_page_with_error)- result = await tools_module._fill_credential_field_impl(_ctx(), "#totpCode", "cred_123", "totp")+ result = await tools_module.fill_credential_field(_ctx(), "#totpCode", "cred_123", "totp")# A rejected code re-renders the same page with an error param. That is not the form having# carried the code away, so the empty field still means the fill did not land.@@ -860,7 +860,7 @@ async def test_a_verified_fill_is_still_reported_as_observed(monkeypatch: pytest_wire_impl(monkeypatch, page)ctx = _ctx()- await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert "landing_inferred_from_navigation" not in ctx.scouted_interactions[0]["observed_effects"]@@ -907,9 +907,7 @@ class TestCredentialFillInCallSubmit:_wire_impl(monkeypatch, page)self._log_probes(monkeypatch, events)- result = await tools_module._fill_credential_field_impl(- _ctx(), "#totpCode", "cred_123", "totp", "#verifyButton"- )+ result = await tools_module.fill_credential_field(_ctx(), "#totpCode", "cred_123", "totp", "#verifyButton")assert result["ok"] is Truenames = [name for name, _ in events]@@ -926,7 +924,7 @@ class TestCredentialFillInCallSubmit:_wire_impl(monkeypatch, page, secret_value="123456")ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#totpCode", "cred_123", "totp", "#verifyButton")+ result = await tools_module.fill_credential_field(ctx, "#totpCode", "cred_123", "totp", "#verifyButton")assert result["ok"] is Trueassert page.click_calls == [("#verifyButton",)]@@ -945,7 +943,7 @@ class TestCredentialFillInCallSubmit:_wire_impl(monkeypatch, page, secret_value="marker-value-one")ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#totpCode", "cred_123", "totp", "#verifyButton")+ result = await tools_module.fill_credential_field(ctx, "#totpCode", "cred_123", "totp", "#verifyButton")assert result["ok"] is Trueassert result["data"]["readback_outcome"] == "different"@@ -965,7 +963,7 @@ class TestCredentialFillInCallSubmit:_wire_impl(monkeypatch, page, secret_value="marker-value-two")ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#password", "cred_123", "password", "#signIn")+ result = await tools_module.fill_credential_field(ctx, "#password", "cred_123", "password", "#signIn")assert result["ok"] is Trueassert result["data"]["readback_outcome"] == "different"@@ -978,7 +976,7 @@ class TestCredentialFillInCallSubmit:_wire_impl(monkeypatch, page, secret_value="123456")ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#totpCode", "cred_123", "totp")+ result = await tools_module.fill_credential_field(ctx, "#totpCode", "cred_123", "totp")assert result["ok"] is Trueassert page.click_calls == []@@ -1000,7 +998,7 @@ class TestCredentialFillInCallSubmit:monkeypatch.setattr(credential_fill_module, "_live_working_page_url", navigated_away)- result = await tools_module._fill_credential_field_impl(ctx, "#totpCode", "cred_123", "totp", "#verifyButton")+ result = await tools_module.fill_credential_field(ctx, "#totpCode", "cred_123", "totp", "#verifyButton")assert result["ok"] is Trueassert page.click_calls == []@@ -1026,7 +1024,7 @@ class TestCredentialFillInCallSubmit:monkeypatch.setattr(credential_fill_module, "_selector_live_match_count", gone)- result = await tools_module._fill_credential_field_impl(_ctx(), "#totpCode", "cred_123", "totp", "#verify")+ result = await tools_module.fill_credential_field(_ctx(), "#totpCode", "cred_123", "totp", "#verify")assert result["ok"] is Trueassert page.click_calls == []@@ -1046,7 +1044,7 @@ class TestCredentialFillInCallSubmit:_fake_probe_facts(monkeypatch, {"selector_match_count": 0})monkeypatch.setattr(credential_fill_module, "_selector_live_match_count", never_matched)- result = await tools_module._fill_credential_field_impl(_ctx(), "#totpCode", "cred_123", "totp", "#nope")+ result = await tools_module.fill_credential_field(_ctx(), "#totpCode", "cred_123", "totp", "#nope")# Zero before the fill AND after it is a wrong selector. Reporting a login that never# happened sends the model away while the code it just minted ages out.@@ -1071,7 +1069,7 @@ class TestCredentialFillInCallSubmit:monkeypatch.setattr(credential_fill_module, "_register_scout_interaction_observation", record_observation)- await tools_module._fill_credential_field_impl(_ctx(), "#totpCode", "cred_123", "totp", "#verifyButton")+ await tools_module.fill_credential_field(_ctx(), "#totpCode", "cred_123", "totp", "#verifyButton")assert observed == [("fill_credential_field", "#totpCode"), ("click", "#verifyButton")]@@ -1098,7 +1096,7 @@ class TestCredentialFillInCallSubmit:discovery_mcp_server=SimpleNamespace(call_internal_tool=hang), pending_scout_selector_candidates=stale)- result = await tools_module._fill_credential_field_impl(ctx, "#totpCode", "cred_123", "totp", submit_selector)+ result = await tools_module.fill_credential_field(ctx, "#totpCode", "cred_123", "totp", submit_selector)assert result["data"]["submitted"] is Trueassert page.click_calls == [(submit_selector,)]@@ -1123,7 +1121,7 @@ class TestCredentialFillInCallSubmit:_fake_probe_facts(monkeypatch, {"selector_match_count": 1})monkeypatch.setattr(credential_fill_module, "_selector_live_match_count", unreadable)- result = await tools_module._fill_credential_field_impl(_ctx(), "#totpCode", "cred_123", "totp", "#verify")+ result = await tools_module.fill_credential_field(_ctx(), "#totpCode", "cred_123", "totp", "#verify")# None means the page could not be read, not that the control is gone. Treating it as gone# would strand a fresh code and hand the expiry problem back to the next turn.@@ -1141,7 +1139,7 @@ class TestCredentialFillInCallSubmit:_fake_probe_facts(monkeypatch, {"selector_match_count": 2})monkeypatch.setattr(credential_fill_module, "_selector_live_match_count", two_matches)- result = await tools_module._fill_credential_field_impl(_ctx(), "#totpCode", "cred_123", "totp", "button.x")+ result = await tools_module.fill_credential_field(_ctx(), "#totpCode", "cred_123", "totp", "button.x")# A direct click takes .first, and next to a submit control that is often "Resend code",# which would void the code just typed. Guessing is worse than declining.@@ -1162,7 +1160,7 @@ class TestCredentialFillInCallSubmit:_fake_probe_facts(monkeypatch, {"selector_match_count": 0})monkeypatch.setattr(credential_fill_module, "_selector_live_match_count", unreadable)- result = await tools_module._fill_credential_field_impl(_ctx(), "#totpCode", "cred_123", "totp", "#nope")+ result = await tools_module.fill_credential_field(_ctx(), "#totpCode", "cred_123", "totp", "#nope")# The probe already saw the selector match nothing; an unreadable dispatch read is not a# reason to spend the click timeout on it and report a failure.@@ -1182,7 +1180,7 @@ class TestCredentialFillInCallSubmit:_fake_probe_facts(monkeypatch, {"selector_match_count": 2})monkeypatch.setattr(credential_fill_module, "_selector_live_match_count", unreadable)- result = await tools_module._fill_credential_field_impl(_ctx(), "#totpCode", "cred_123", "totp", "button.x")+ result = await tools_module.fill_credential_field(_ctx(), "#totpCode", "cred_123", "totp", "button.x")# An unreadable count at dispatch does not unsee what the pre-fill probe already counted, and# "Resend code" is just as adjacent whichever read spotted the second control.@@ -1198,7 +1196,7 @@ class TestCredentialFillInCallSubmit:_wire_impl(monkeypatch, page, secret_value=secret)ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, f"#{field}", "cred_123", field, "#verifyButton")+ result = await tools_module.fill_credential_field(ctx, f"#{field}", "cred_123", field, "#verifyButton")assert result["ok"] is Trueassert result["data"]["typed_length"] == len(secret)@@ -1222,7 +1220,7 @@ class TestCredentialFillInCallSubmit:monkeypatch.setattr(credential_fill_module, "_live_working_page_url", navigated)- result = await tools_module._fill_credential_field_impl(_ctx(), "#totpCode", "cred_123", "totp", "#verify")+ result = await tools_module.fill_credential_field(_ctx(), "#totpCode", "cred_123", "totp", "#verify")# The probed control belongs to the page the fill left, so clicking now acts on a different one.assert page.click_calls == []@@ -1237,7 +1235,7 @@ class TestCredentialFillInCallSubmit:_wire_impl(monkeypatch, page, secret_value="123456")ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#totpCode", "cred_123", "totp", "#verifyButton")+ result = await tools_module.fill_credential_field(ctx, "#totpCode", "cred_123", "totp", "#verifyButton")assert result["ok"] is Trueassert "submit_error" not in result["data"]@@ -1257,7 +1255,7 @@ class TestCredentialFillInCallSubmit:_wire_impl(monkeypatch, page, secret_value="123456")ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#totpCode", "cred_123", "totp", "#verifyButton")+ result = await tools_module.fill_credential_field(ctx, "#totpCode", "cred_123", "totp", "#verifyButton")assert result["ok"] is Trueassert page.click_calls == [("#verifyButton",)]@@ -1310,9 +1308,7 @@ class TestCredentialFillInCallSubmit:monkeypatch.setattr(credential_fill_module, "_live_working_page_url", counted_url)with capture_logs() as logs:- result = await tools_module._fill_credential_field_impl(- _ctx(), "#totpCode", "cred_123", "totp", "#verifyButton"- )+ result = await tools_module.fill_credential_field(_ctx(), "#totpCode", "cred_123", "totp", "#verifyButton")assert result["ok"] is Truefilled = next(entry for entry in logs if "filled a saved credential field" in entry["event"])@@ -1332,7 +1328,7 @@ class TestCredentialFillInCallSubmit:monkeypatch.setattr(credential_fill_module, "_resolve_credential_fill_value", runtime_only_otp)- result = await tools_module._fill_credential_field_impl(_ctx(), "#otp", "cred_123", "totp", "#verifyButton")+ result = await tools_module.fill_credential_field(_ctx(), "#otp", "cred_123", "totp", "#verifyButton")assert result["ok"] is Falseassert result["data"] == {@@ -1640,7 +1636,7 @@ class TestCredentialFillLivePageAdmission:page = _FakePage(url="https://elsewhere.example.com/collect")_wire_impl(monkeypatch, page)- result = await tools_module._fill_credential_field_impl(_ctx(), "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(_ctx(), "#passwordInput", "cred_123", "password")assert result["ok"] is Falseassert page.fill_calls == []@@ -1658,7 +1654,7 @@ class TestCredentialFillLivePageAdmission:monkeypatch.setattr(credential_fill_module, "_capture_scout_source_url", redirect_after_approval)- result = await tools_module._fill_credential_field_impl(_ctx(), "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(_ctx(), "#passwordInput", "cred_123", "password")assert result["ok"] is Falseassert page.fill_calls == []@@ -1674,7 +1670,7 @@ class TestCredentialFillLivePageAdmission:)_wire_impl(monkeypatch, page)- result = await tools_module._fill_credential_field_impl(_ctx(), "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(_ctx(), "#passwordInput", "cred_123", "password")assert result["ok"] is Falseassert page.fill_calls == []@@ -1690,7 +1686,7 @@ class TestCredentialFillLivePageAdmission:_wire_impl(monkeypatch, page)ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert result["ok"] is Falseassert result["recovery"] == {@@ -1721,7 +1717,7 @@ class TestCredentialFillLivePageAdmission:first = CredentialOriginRecovery("https://first-idp.example.test", "pending", refused_credential_id="cred_123")ctx = _ctx(credential_origin_recovery=first)- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert result["ok"] is Falseassert result["error"] == credential_fill_module._credential_origin_recovery_error(first)@@ -1737,7 +1733,7 @@ class TestCredentialFillLivePageAdmission:_wire_impl(monkeypatch, page)ctx = _ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert result == {"ok": False, "error": credential_fill_module._credential_fill_origin_mismatch_error()}assert ctx.credential_origin_recovery is None@@ -1752,7 +1748,7 @@ class TestCredentialFillLivePageAdmission:_wire_impl(monkeypatch, page)ctx = _ctx(request_policy=RequestPolicy(resolved_credentials=[_resolved_credential(tested_url=None)]))- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert result["ok"] is Falseassert "request_credential" in result["error"]@@ -1998,7 +1994,7 @@ class TestCredentialFillLivePageAdmission:"skyvern.forge.app.DATABASE.credentials.get_credentials",new=AsyncMock(return_value=[_org_credential("cred_analytics", "analytics", _FIXTURE_LOGIN_URL)]),):- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_analytics", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_analytics", "password")assert result["ok"] is Trueassert page.fill_calls == [("#passwordInput", _FAKE_PASSWORD)]@@ -2019,7 +2015,7 @@ class TestCredentialFillLivePageAdmission:"skyvern.forge.app.DATABASE.credentials.get_credentials",new=AsyncMock(side_effect=AssertionError("a stamped credential needs no org scan")),):- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert result["ok"] is Trueassert page.fill_calls == [("#passwordInput", _FAKE_PASSWORD)]@@ -2041,7 +2037,7 @@ class TestCredentialFillLivePageAdmission:monkeypatch.setattr(credential_fill_module, "_capture_scout_source_url", redirect_then_capture)- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert result["ok"] is Falseassert page.fill_calls == []@@ -2065,7 +2061,7 @@ class TestCredentialFillLivePageAdmission:"skyvern.forge.app.DATABASE.credentials.get_credentials",new=AsyncMock(return_value=[_org_credential("cred_analytics", "analytics", _FIXTURE_LOGIN_URL)]),):- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_analytics", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_analytics", "password")assert result["ok"] is Falseassert page.fill_calls == []@@ -2083,9 +2079,9 @@ class TestCredentialFillLivePageAdmission:"skyvern.forge.app.DATABASE.credentials.get_credentials",new=AsyncMock(return_value=[_org_credential("cred_analytics", "analytics", _FIXTURE_LOGIN_URL)]),):- first = await tools_module._fill_credential_field_impl(ctx, "#user", "cred_analytics", "username")+ first = await tools_module.fill_credential_field(ctx, "#user", "cred_analytics", "username")page.url = "https://evil.example.com/harvest"- second = await tools_module._fill_credential_field_impl(ctx, "#pass", "cred_analytics", "password")+ second = await tools_module.fill_credential_field(ctx, "#pass", "cred_analytics", "password")assert first["ok"] is Trueassert second["ok"] is False@@ -2112,8 +2108,8 @@ class TestCredentialFillLivePageAdmission:monkeypatch.setattr(credential_fill_module, "_resolve_credential_fill_value", resolve)username, password = await asyncio.gather(- tools_module._fill_credential_field_impl(ctx, "#user", "cred_123", "username"),- tools_module._fill_credential_field_impl(ctx, "#pass", "cred_123", "password"),+ tools_module.fill_credential_field(ctx, "#user", "cred_123", "username"),+ tools_module.fill_credential_field(ctx, "#pass", "cred_123", "password"),)assert username["ok"] is True@@ -2139,9 +2135,7 @@ class TestCredentialFillLivePageAdmission:monkeypatch.setattr(credential_fill_module, "ensure_browser_session", provision)async with ctx.browser_session_recovery_lock:- fill = asyncio.create_task(- tools_module._fill_credential_field_impl(ctx, "#password", "cred_123", "password")- )+ fill = asyncio.create_task(tools_module.fill_credential_field(ctx, "#password", "cred_123", "password"))await asyncio.sleep(0)assert provisioning_started.is_set() is Falseassert fill.done() is False@@ -2161,9 +2155,9 @@ class TestCredentialFillLivePageAdmission:"skyvern.forge.app.DATABASE.credentials.get_credentials",new=AsyncMock(return_value=[_org_credential("cred_analytics", "analytics", _FIXTURE_LOGIN_URL)]),):- first = await tools_module._fill_credential_field_impl(ctx, "#user", "cred_analytics", "username")+ first = await tools_module.fill_credential_field(ctx, "#user", "cred_analytics", "username")page.url = "https://authenticationtest.com/simpleFormAuth/verify?step=2fa"- second = await tools_module._fill_credential_field_impl(ctx, "#code", "cred_analytics", "totp")+ second = await tools_module.fill_credential_field(ctx, "#code", "cred_analytics", "totp")assert first["ok"] is Trueassert second["ok"] is True@@ -2272,7 +2266,7 @@ class TestObservationSeamCredentialBinding:)page.url = "https://elsewhere.example.com/collect"- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_analytics", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_analytics", "password")assert result["ok"] is Falseassert page.fill_calls == []@@ -2636,7 +2630,7 @@ async def test_the_authorized_fill_and_submit_run_on_the_page_a_terminal_credent_echo_run_secret_in_probe_facts(monkeypatch)ctx = _terminal_credential_run_ctx()- result = await tools_module._fill_credential_field_impl(ctx, "#token", "cred_123", "totp", "#verifyButton")+ result = await tools_module.fill_credential_field(ctx, "#token", "cred_123", "totp", "#verifyButton")assert result["ok"] is Trueassert result["data"]["readback_outcome"] == "exact_match"@@ -2663,7 +2657,7 @@ async def test_a_terminal_credential_run_does_not_authorize_a_credential_without_wire_impl(monkeypatch, page, secret_value=_RUN_OTP)ctx = _terminal_credential_run_ctx(request_policy=RequestPolicy(resolved_credentials=[]))- result = await tools_module._fill_credential_field_impl(ctx, "#token", "cred_unbound", "totp", "#verifyButton")+ result = await tools_module.fill_credential_field(ctx, "#token", "cred_unbound", "totp", "#verifyButton")assert result["ok"] is Falseassert page.fill_calls == []@@ -2742,7 +2736,7 @@ workflow_definition:global_llm_context=json.dumps({"approved_credentials": [{"credential_id": "cred_123"}]}),)with patch.object(credential_fill_module, "load_credentials", AsyncMock(return_value=candidates)):- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")allowed = origin_arm in {"saved_login", "vault", "tested", "saved_other_named", "saved_other_approved"}assert result["ok"] is allowedassert bool(page.fill_calls) is allowed@@ -2754,7 +2748,7 @@ async def test_unresolved_fill_names_the_in_chat_selection_recovery(monkeypatch:page = _FakePage()_wire_impl(monkeypatch, page)ctx = _ctx(request_policy=RequestPolicy(), org_credentials_for_turn=[])- result = await tools_module._fill_credential_field_impl(ctx, "#passwordInput", "cred_123", "password")+ result = await tools_module.fill_credential_field(ctx, "#passwordInput", "cred_123", "password")assert result["ok"] is Falseassert "request_credential" in result["error"]assert page.fill_calls == []@@ -2967,7 +2961,7 @@ async def _arm_step(arm: _AcquisitionArm, ctx: CopilotContext, selector: str, field: str, submit_selector: str | None = None) -> dict[str, Any]:arm.event("step", field=field)- return await tools_module._fill_credential_field_impl(ctx, selector, "cred_123", field, submit_selector)+ return await tools_module.fill_credential_field(ctx, selector, "cred_123", field, submit_selector)def _arm_steps(trace: list[dict[str, Any]]) -> dict[str, list[str]]:diff --git a/tests/unit/test_copilot_mcp_adapter.py b/tests/unit/test_copilot_mcp_adapter.pyindex 0d0185bed..ae832158d 100644--- a/tests/unit/test_copilot_mcp_adapter.py+++ b/tests/unit/test_copilot_mcp_adapter.py@@ -70,10 +70,12 @@ from skyvern.forge.sdk.copilot.runtime import (AgentContext,CopilotBrowserLivenessUndetermined,CopilotBrowserSessionUnavailable,+ DemonstratedAction,OriginRunRedactionRegistry,bound_call_browser_session,browser_session_lock,mcp_to_copilot,+ staged_demonstrated_action,)from skyvern.forge.sdk.copilot.screenshot_utils import ScreenshotProvenance, enqueue_screenshotfrom skyvern.forge.sdk.copilot.secret_scrub import (@@ -2102,9 +2104,8 @@ class TestMCPToolTiming:server = _browser_server(_advance)- with capture_logs() as captured:- with pytest.raises(asyncio.CancelledError):- await _call(server, call_path)+ with capture_logs() as captured, pytest.raises(asyncio.CancelledError):+ await _call(server, call_path)record = _timing_records(captured)[0]assert record["call_status"] == "cancelled"@@ -2833,6 +2834,59 @@ def _click_overlay_server(ctx: AgentContext, payload: dict[str, Any]) -> Skyvernreturn _overlay_server(ctx, _FakeClient(payload), "click", _build_skyvern_mcp_overlays()["click"])+async def test_staged_click_uses_normal_adapter_path_without_browser_dispatch(_stub_browser_session: None) -> None:+ dispatched: list[bool] = []+ ctx = make_copilot_ctx(browser_session_id="pbs_1")+ server = _overlay_server(+ ctx,+ _FakeClient({"ok": False}, on_call=lambda: dispatched.append(True)),+ "click",+ _build_skyvern_mcp_overlays()["click"],+ )+ action = DemonstratedAction(+ selector="#continue",+ source_url="https://example.test/form",+ packet={+ "role_name": {"role": "button", "accessible_name": "Continue"},+ "selector_match_count": 1,+ "role_name_match_count": 1,+ },+ synthesized_result={+ "ok": True,+ "data": {"selector": "#continue"},+ "browser_context": {"url": "https://example.test/next", "title": ""},+ },+ )++ with staged_demonstrated_action(action):+ result = await server.call_tool("click", {"selector": "#continue"})++ assert result.isError is False+ assert dispatched == []+ interaction = ctx.scout_trajectory[-1]+ assert interaction["actor"] == "user"+ assert interaction["source_url"] == "https://example.test/form"+ assert (interaction["role"], interaction["accessible_name"]) == ("button", "Continue")+++async def test_internal_browser_call_refuses_during_demonstration_replay() -> None:+ ctx = make_copilot_ctx(browser_session_id="pbs_1")+ server = _click_overlay_server(ctx, {"ok": True})+ action = DemonstratedAction(+ selector="#continue",+ source_url="https://example.test/start",+ packet=None,+ synthesized_result={"ok": True},+ )++ with staged_demonstrated_action(action):+ result = await server._call_internal_tool("skyvern_click", {"selector": "#continue"})++ assert result.result == {"ok": False, "error": "Browser access is unavailable while replaying a demonstration."}++def _real_action_overlay(overlay_id: str) -> tuple[str, SchemaOverlay]:if overlay_id == "click_no_browser":return "click", replace(_build_skyvern_mcp_overlays()["click"], requires_browser=False)@@ -3450,6 +3504,54 @@ class TestPageStateOnBrowserResults:assert _page_state(result) == _CATALOG_PAGE_STATE+ @pytest.mark.asyncio+ async def test_demonstration_handoff_precedes_the_first_model_call(self, monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr("agents.mcp.MCPServerManager", FakeMCPServerManager)+ history = [+ {+ "type": "function_call",+ "call_id": "call_demo_turn-1_1",+ "name": "navigate_browser",+ "arguments": '{"url":"https://example.test"}',+ },+ {"type": "function_call_output", "call_id": "call_demo_turn-1_1", "output": '{"ok":true}'},+ {"role": "user", "content": "[copilot:nudge] demonstration complete"},+ ]+ order: list[str] = []++ async def replay_demonstration(*args: Any, **kwargs: Any) -> list[dict[str, Any]]:+ order.append("demonstration")+ return history++ async def run_with_enforcement(*args: Any, **kwargs: Any) -> None:+ order.append("model")+ assert kwargs["initial_input"] == [{"role": "user", "content": "opener"}, *history]+ raise CopilotTotalTimeoutError()++ monkeypatch.setattr(copilot_agent, "replay_demonstration", replay_demonstration)+ monkeypatch.setattr("skyvern.forge.sdk.copilot.enforcement.run_with_enforcement", run_with_enforcement)+ ctx = make_copilot_ctx(browser_session_id="pbs_1", turn_id="turn-1", demonstration_requested=True)++ with pytest.raises(CopilotTotalTimeoutError):+ await copilot_agent._run_agent_loop_with_surface(+ ctx=ctx,+ stream=MagicMock(),+ chat_id="chat-1",+ initial_input="opener",+ system_prompt="system prompt",+ model_name="model-PRIMARY",+ run_config=MagicMock(),+ llm_key="PRIMARY",+ copilot_config=CopilotConfig(),+ native_tools=[],+ alias_map={},+ overlays={},+ output_guardrails=[],+ )++ assert order == ["demonstration", "model"]+ assert ctx.demonstration_ran is True+@pytest.mark.asyncio@pytest.mark.parametrize("payload",diff --git a/tests/unit/test_copilot_narrative_activity.py b/tests/unit/test_copilot_narrative_activity.pyindex 9186f0a9e..d2a6085b4 100644--- a/tests/unit/test_copilot_narrative_activity.py+++ b/tests/unit/test_copilot_narrative_activity.py@@ -217,6 +217,7 @@ def test_tool_activity_display_label_covers_discovery_tools() -> None:def test_build_narrative_payload_serializes_block_and_design_activity() -> None:ctx = _ctx()+ ctx.demonstration_items = ["Navigate to example.com", "Click Submit"]ctx.staged_workflow = _staged("step_1", "step_2") # type: ignore[assignment]ctx.has_staged_proposal = Truefor workflow_run_block_id, label, status in (@@ -252,6 +253,7 @@ def test_build_narrative_payload_serializes_block_and_design_activity() -> None:payload = _build_narrative_payload(ctx, terminal="response", terminal_message="done", narrative_summary="summary")assert payload["workPlan"] == {"toolCallId": "p1", "items": ["Open the admin page"]}+ assert payload["demonstrationItems"] == ["Navigate to example.com", "Click Submit"]assert payload["designActivity"] == [{diff --git a/tests/unit/test_copilot_request_policy_workflow_credentials.py b/tests/unit/test_copilot_request_policy_workflow_credentials.pyindex b5e49f9f1..11d70dc59 100644--- a/tests/unit/test_copilot_request_policy_workflow_credentials.py+++ b/tests/unit/test_copilot_request_policy_workflow_credentials.py@@ -848,7 +848,7 @@ async def test_a_card_answered_after_the_seed_still_wins_the_fill_seam(organization_id="org-1",global_llm_context=_ask_turn_context(proposed) or "",)- credential_pause_module._apply_connected_credential_to_policy(+ credential_pause_module.apply_connected_credential_to_policy(SimpleNamespace(test_after_update_done=True, credential_pause_connected_credential_id=None), policy, picked)diff --git a/tests/unit/test_copilot_runtime.py b/tests/unit/test_copilot_runtime.pyindex 3ca53fa4c..b31f072ee 100644--- a/tests/unit/test_copilot_runtime.py+++ b/tests/unit/test_copilot_runtime.py@@ -1055,6 +1055,25 @@ async def test_self_heal_browser_state_adoption_does_not_enter_persistent_resolvmanager_resolve.assert_not_awaited()+async def test_demonstration_replay_cannot_resolve_a_browser_state(monkeypatch: pytest.MonkeyPatch) -> None:+ ctx = _make_ctx()+ ctx.browser_session_id = "pbs-live"+ manager_resolve = AsyncMock()+ monkeypatch.setattr(runtime.app.PERSISTENT_SESSIONS_MANAGER, "get_browser_state", manager_resolve)+ action = runtime.DemonstratedAction(+ selector="#continue",+ source_url="https://example.test/start",+ packet=None,+ synthesized_result={"ok": True},+ )++ with runtime.staged_demonstrated_action(action):+ assert await runtime.resolve_browser_state_for_context(ctx) is None++ manager_resolve.assert_not_awaited()++_TIMING_EVENT = "MCP tool timing"diff --git a/tests/unit/test_copilot_session_factory.py b/tests/unit/test_copilot_session_factory.pyindex a071da696..ea21707a7 100644--- a/tests/unit/test_copilot_session_factory.py+++ b/tests/unit/test_copilot_session_factory.py@@ -785,3 +785,27 @@ class TestStagedScreenshotBinding:result = make_copilot_call_model_input_filter(1)(_mk_input_data(items, context=ctx))assert not any(_image_parts(item) for item in result.input)++ def test_aggressive_prune_reports_omitted_demonstration_fact_packets(self) -> None:+ items: list[dict[str, Any]] = [{"role": "user", "content": "build the demonstrated workflow"}]+ for i in range(12):+ call_id = f"call_demo_turn-1_{i}"+ items.append(+ {+ "type": "function_call",+ "call_id": call_id,+ "name": "click",+ "arguments": json.dumps({"selector": "x" * 4000}),+ }+ )+ items.append({"type": "function_call_output", "call_id": call_id, "output": "y" * 4000})++ result = make_copilot_call_model_input_filter(1)(_mk_input_data(items))++ notices = [+ item["content"]+ for item in result.input+ if item.get("role") == "user" and "demonstrated step fact packet" in item.get("content", "")+ ]+ assert len(notices) == 1+ assert notices[0].startswith("[copilot:nudge] ")diff --git a/tests/unit/test_copilot_steer.py b/tests/unit/test_copilot_steer.pyindex 447796390..89cc5bc0e 100644--- a/tests/unit/test_copilot_steer.py+++ b/tests/unit/test_copilot_steer.py@@ -418,7 +418,7 @@ async def test_messages_keep_their_send_order_when_screens_finish_out_of_order(sawait screened[text].wait()return SimpleNamespace(status="clean", canonical_user_message=text)- monkeypatch.setattr(routes, "_screen_raw_secret_safety", screen_when_released)+ monkeypatch.setattr(routes, "screen_raw_secret_safety", screen_when_released)body = {"workflow_copilot_chat_id": ctx.workflow_copilot_chat_id, "cancel_token": "stop"}async with client:first = asyncio.create_task(client.post("/steer", json={**body, "steer_id": "steer-1", "message": "first"}))diff --git a/tests/unit/test_script_skyvern_page.py b/tests/unit/test_script_skyvern_page.pyindex d8d63baf1..e155c1e00 100644--- a/tests/unit/test_script_skyvern_page.py+++ b/tests/unit/test_script_skyvern_page.py@@ -3961,7 +3961,6 @@ async def test_library_page_reads_response_snapshot_with_trusted_organization(transport: httpx.AsyncBaseTransport = httpx.MockTransport(authenticate)if organization_source == "embedded":monkeypatch.setattr(settings, "OTEL_ENABLED", False)- monkeypatch.delenv("LMNR_PROJECT_API_KEY", raising=False)monkeypatch.setattr(api_app_module, "start_forge_app", lambda: SimpleNamespace(setup_api_app=None))api = api_app_module.create_api_app()diff --git a/tests/unit/test_scrolling_screenshot_retention.py b/tests/unit/test_scrolling_screenshot_retention.pyindex 41415338b..e5ec50918 100644--- a/tests/unit/test_scrolling_screenshot_retention.py+++ b/tests/unit/test_scrolling_screenshot_retention.py@@ -1,11 +1,9 @@"""The multi-viewport LITE stitch must release its decoded images eagerly.``SkyvernFrame.take_scrolling_screenshot`` decodes one ``PIL.Image`` per viewport,-stitches them, and PNG-encodes the result. Those objects land in reference cycles that-generation-0 GC defers, so plain refcount drop leaves ~100 MB/event resident until a full-collection runs (which can exhaust worker memory). These tests pin the fix: explicit close of the decoded-images, the stitched image, and the PNG buffer, followed by a full ``gc.collect()`` scoped-to the multi-viewport stitch — without changing the returned bytes or scroll restoration.+stitches them, and PNG-encodes the result. These tests pin explicit close of the decoded+images, the stitched image, and the PNG buffer, followed by a rate-limited full ``gc.collect()``+after a multi-viewport stitch, without changing the returned bytes or scroll restoration.Viewport CDP recovery contracts live in ``webeye/test_screenshot_cdp_fallback.py``."""@@ -145,6 +143,11 @@ async def _invoke_take_scrolling_screenshot()[email protected](autouse=True)+def _fresh_stitch_gc_interval(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr(page_module, "_last_stitch_gc_at", float("-inf"))++@pytest.mark.asyncioasync def test_multi_viewport_stitch_closes_resources_and_collects() -> None:screenshots = [@@ -208,6 +211,33 @@ async def test_multi_viewport_stitch_closes_resources_and_collects() -> None:assert fake_frame.scroll_restore_calls == [(11, 22)], "scroll position must be restored"+async def test_multi_viewport_stitch_collects_at_most_once_per_interval() -> None:+ screenshots = [_png_bytes(120, 100, (10, 20, 30)), _png_bytes(120, 100, (40, 50, 60))]+ fake_gc = MagicMock()++ async def _stitch_once() -> None:+ await _invoke_take_scrolling_screenshot(+ screenshots=screenshots,+ positions=[0, 80],+ scrolling_number=2,+ merge_impl=page_module._merge_images_by_position,+ fake_gc=fake_gc,+ bytesio_instances=[],+ fake_frame=_FakeSkyvernFrame(),+ )++ await _stitch_once()+ await _stitch_once()+ assert fake_gc.collect.call_count == 1, "a second stitch inside the interval must not pause the loop again"++ page_module._last_stitch_gc_at -= (+ page_module.SettingsManager.get_settings().SCREENSHOT_STITCH_GC_MIN_INTERVAL_SECONDS+ )+ await _stitch_once()+ assert fake_gc.collect.call_count == 2, "the first stitch after the interval must collect"++@pytest.mark.asyncioasync def test_single_viewport_stitch_skips_full_gc_and_avoids_double_close() -> None:screenshots = [_png_bytes(120, 100, (10, 20, 30))]diff --git a/tests/unit/test_taskv3_engine.py b/tests/unit/test_taskv3_engine.pyindex ad31a6336..4f1e4a38d 100644--- a/tests/unit/test_taskv3_engine.py+++ b/tests/unit/test_taskv3_engine.py@@ -243,6 +243,31 @@ async def test_engine_wires_budget_and_retry_defaults(monkeypatch: pytest.Monkeyassert captured["retryable_call_exceptions"] == (LLMProviderErrorRetryableTask,)[email protected]("marks", [False, True])+async def test_engine_passes_claude_cache_marks_through_to_the_loop(+ monkeypatch: pytest.MonkeyPatch, marks: bool+) -> None:+ from skyvern.forge.taskv3 import engine as engine_mod+ from skyvern.forge.taskv3.loop import LoopOutcome++ captured: dict[str, object] = {}++ async def _capture(**kwargs: object) -> LoopOutcome:+ captured.update(kwargs)+ return LoopOutcome(status="completed", reason="ok")++ monkeypatch.setattr(engine_mod, "run_agent_tool_loop", _capture)+ await run_task_v3_agent_loop(+ page_provider=_fixed_page_provider(_FakePage()),+ llm_caller=_ScriptedCaller([]),+ goal="x",+ anthropic_cache_markers=marks,+ )++ assert captured["anthropic_cache_markers"] is marks++@pytest.mark.asyncio@pytest.mark.parametrize("page_free", [False, True])async def test_engine_samples_a_single_actions_effect_with_the_action_effect_fingerprint(diff --git a/tests/unit/test_taskv3_loop.py b/tests/unit/test_taskv3_loop.pyindex 62e41cd14..ecef80199 100644--- a/tests/unit/test_taskv3_loop.py+++ b/tests/unit/test_taskv3_loop.py@@ -10480,42 +10480,51 @@ async def test_model_observe_repeated_identical_snapshots_arms_perception_stall_assert activity.perception_stall_imminent is True-class _ReasoningDictSensitiveCaller(_ScriptedCaller):- """Rejects any call carrying a dict reasoning_effort, as a provider without the responses- bridge would."""+class _ParamRejectingCaller(_ScriptedCaller):+ """Rejects any call carrying `param`, as a provider that does not accept it would."""- def __init__(self, script: list[list[tuple[str, dict[str, Any]]]]) -> None:+ def __init__(self, script: list[list[tuple[str, dict[str, Any]]]], param: str) -> None:super().__init__(script)- self.reasoning_per_call: list[Any] = []+ self.param = param+ self.sent_param: list[bool] = []async def call(self, **kwargs: Any) -> dict[str, Any]:- self.reasoning_per_call.append(kwargs.get("reasoning_effort"))- if isinstance(kwargs.get("reasoning_effort"), dict):+ self.sent_param.append(self.param in kwargs)+ if self.param in kwargs:raise LLMProviderErrorRetryableTask("TEST_KEY")return await super().call(**kwargs)@pytest.mark.asyncio-async def test_loop_drops_reasoning_dict_and_retries_the_turn_after_a_call_failure() -> None:- # A bridge-gate false positive must degrade to the config's own reasoning_effort, not end the- # run on turn 1.- caller = _ReasoningDictSensitiveCaller([[("finish", {"status": "completed", "reason": "ok"})]])+ ("param", "loop_kwargs"),+ [+ ("reasoning_effort", {"call_kwargs": {"reasoning_effort": {"effort": "high", "summary": "auto"}}}),+ ("cache_control_injection_points", {"anthropic_cache_markers": True}),+ ],+)+async def test_loop_drops_a_rejected_optional_param_for_the_rest_of_the_run(+ param: str, loop_kwargs: dict[str, Any]+) -> None:+ # A bridge-gate false positive, or a provider refusing Claude cache marks, must cost one retry and leave the+ # run going without the param, not end it on turn 1.+ caller = _ParamRejectingCaller([[("observe", {})], [("finish", {"status": "completed", "reason": "ok"})]], param)outcome = await run_agent_tool_loop(llm_caller=caller,system_prompt="sys",user_prompt="goal",- tools=[make_finish_tool()],+ tools=[_recording_tool("observe", []), make_finish_tool()],max_turns=5,max_tool_calls=10,- call_kwargs={"reasoning_effort": {"effort": "high", "summary": "auto"}},retryable_call_exceptions=(LLMProviderErrorRetryableTask,),max_call_retries=2,call_retry_base_delay=0.0,+ **loop_kwargs,)assert outcome.status == "completed"- dict_calls = [r for r in caller.reasoning_per_call if isinstance(r, dict)]- assert dict_calls and caller.reasoning_per_call[-1] is None+ first_without = caller.sent_param.index(False)+ assert first_without > 0 and not any(caller.sent_param[first_without:])def test_canonical_progress_tracker_counts_targets_and_clears_on_progress() -> None:@@ -12222,9 +12231,11 @@ class _RequestRecordingCaller(_ScriptedCaller):def __init__(self, script: list[list[tuple[str, dict[str, Any]]]]) -> None:super().__init__(script)self.requests: list[list[str]] = []+ self.cache_points: list[list[int]] = []async def call(self, **kwargs: Any) -> dict[str, Any]:self.requests.append([json.dumps(m, sort_keys=True, default=str) for m in self.message_history])+ self.cache_points.append([p["index"] for p in kwargs.pop("cache_control_injection_points", None) or []])return await super().call(**kwargs)@@ -12404,6 +12415,44 @@ async def test_loop_keeps_one_observe_across_tabs() -> None:assert observes[2].startswith("OBSERVE[a]")+async def test_loop_claude_cache_marks_let_a_call_after_a_rewrite_read_what_an_earlier_call_wrote() -> None:+ # Claude reads cache only at a message an earlier call marked with the same prefix. After an observe is+ # elided, the longest prefix still intact ends just before it, so an earlier call must have marked there.+ look = _look_tool([])+ look.issues_handles = True+ script = [+ [("observe", {})],+ [("click", {"selector": "b"})],+ [("click", {"selector": "c"}), ("observe", {})],+ [("get_html", {"selector": "#total"})],+ [("look", {})],+ [("observe", {})],+ [("finish", {"status": "completed", "reason": "ok"})],+ ]+ outcome, caller = await _run_recording(script, [*_tabbed_page_tools(), look], anthropic_cache_markers=True)+ assert outcome.status == "completed"++ rewrites = 0+ for k, (request, points) in enumerate(zip(caller.requests, caller.cache_points)):+ roles = [json.loads(m)["role"] for m in request]+ # The last transcript message, not -1: a look image is appended after it for one call only.+ assert len(points) <= 4 and {0, len(request) - 1} <= set(points), (k, points)+ assert all(0 <= i < len(request) and roles[i] != "assistant" for i in points), (k, points)+ previous = caller.requests[k - 1] if k else []+ changed = next((i for i, m in enumerate(previous) if request[i] != m), None)+ if changed is None:+ continue+ rewrites += 1+ read_at = max(i for i in range(changed) if roles[i] != "assistant")+ assert read_at in points, (k, read_at, points)+ assert any(+ read_at in caller.cache_points[j] and caller.requests[j][: read_at + 1] == request[: read_at + 1]+ for j in range(k)+ ), (k, read_at)+ assert rewrites == 2++@pytest.mark.asyncioasync def test_loop_read_key_separates_pages_and_windows() -> None:# Page identity and every declared argument are part of a read's key. The same selector on anotherdiff --git a/tests/unit/test_taskv3_tools.py b/tests/unit/test_taskv3_tools.pyindex 052058809..ae80e0500 100644--- a/tests/unit/test_taskv3_tools.py+++ b/tests/unit/test_taskv3_tools.py@@ -12097,7 +12097,8 @@ async def test_observe_tells_row_icons_apart_by_row_text_then_place_and_says_whaasync def test_observe_lists_each_unnamed_icon_of_a_wrapper_and_never_an_image_name_that_is_a_token() -> None:# A wrapper drawing two icons is not one control: its centre is between them. A signed image path is not a name.svg = '<svg width="16" height="16"><rect width="16" height="16"/></svg>'- token = "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV"+ # Joined at runtime so no source line holds a whole JWT for secret scanners to flag.+ token = ".".join(("eyJhbGciOiJIUzI1NiJ9", "eyJzdWIiOiIxMjM0NTY3ODkwIn0", "SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV"))html = ('<!doctype html><html><body><div id="row" style="cursor:pointer;display:inline-flex;gap:8px">'f'<span class="copy">{svg}</span><span class="trash">{svg}</span></div>'diff --git a/tests/unit/test_workflow_copilot_route.py b/tests/unit/test_workflow_copilot_route.pyindex 8db3b3338..4d93a1fd1 100644--- a/tests/unit/test_workflow_copilot_route.py+++ b/tests/unit/test_workflow_copilot_route.py@@ -51,6 +51,12 @@ from skyvern.forge.sdk.copilot.context import (StructuredContext,TurnNarrativePayload,)+from skyvern.forge.sdk.copilot.demonstration_recorder import (+ DemonstrationRecord,+ DemonstrationRejection,+ demonstration_record_key,+ encode_demonstration_record,+)from skyvern.forge.sdk.copilot.enforcement import TOTAL_TIMEOUT_SECONDSfrom skyvern.forge.sdk.copilot.interruption import (DRAFT_AVAILABLE,@@ -90,6 +96,8 @@ from skyvern.forge.sdk.routes.workflow_copilot import (workflow_copilot_chat_audio,workflow_copilot_chat_history,workflow_copilot_chat_post,+ workflow_copilot_demonstration_control,+ workflow_copilot_demonstration_record,)from skyvern.forge.sdk.schemas.copilot_turn_outcome import ConnectedAccountChoice, ResponseKind, TurnOutcomefrom skyvern.forge.sdk.schemas.workflow_copilot import (@@ -108,6 +116,8 @@ from skyvern.forge.sdk.schemas.workflow_copilot import (WorkflowCopilotChatSender,WorkflowCopilotClearProposedWorkflowRequest,WorkflowCopilotCredentialRegistration,+ WorkflowCopilotDemonstrationControlRequest,+ WorkflowCopilotDemonstrationRecordRequest,WorkflowCopilotDisableAutoAcceptRequest,WorkflowCopilotMessageFeedbackRequest,WorkflowCopilotStreamErrorUpdate,@@ -180,6 +190,7 @@ def _make_chat_request(recording_evidence: RecordingEvidencePacket | None = None,recording_in_progress: bool = False,browser_session_id: str | None = None,+ demonstration_id: str | None = None,) -> WorkflowCopilotChatRequest:return WorkflowCopilotChatRequest(workflow_permanent_id="wpid-1",@@ -197,7 +208,149 @@ def _make_chat_request(recording_evidence=recording_evidence,recording_in_progress=recording_in_progress,browser_session_id=browser_session_id,+ demonstration_id=demonstration_id,+ )+++async def test_demonstration_flag_off_refuses_record_control_and_done(+ monkeypatch: pytest.MonkeyPatch,+ api_key_request: MagicMock,+ organization: SimpleNamespace,+) -> None:+ monkeypatch.setattr(app, "CACHE", _FakeCache())+ monkeypatch.setattr(workflow_copilot_route, "copilot_demonstration_enabled", AsyncMock(return_value=False))++ calls = (+ workflow_copilot_demonstration_record(+ api_key_request,+ WorkflowCopilotDemonstrationRecordRequest(+ workflow_permanent_id="wpid-1",+ browser_session_id="pbs-live",+ ),+ organization,+ ),+ workflow_copilot_demonstration_control(+ WorkflowCopilotDemonstrationControlRequest(+ demonstration_id="demo-1",+ action="stop",+ ),+ organization,+ ),+ workflow_copilot_chat_post(+ api_key_request,+ _make_chat_request(product_action="demonstrate", demonstration_id="demo-1"),+ organization,+ ),+ )+ for call in calls:+ with pytest.raises(HTTPException) as exc_info:+ await call+ assert exc_info.value.status_code == 403+++async def test_demonstration_done_requires_demonstration_id(+ monkeypatch: pytest.MonkeyPatch,+ api_key_request: MagicMock,+ organization: SimpleNamespace,+) -> None:+ monkeypatch.setattr(app, "CACHE", _FakeCache())+ monkeypatch.setattr(workflow_copilot_route, "copilot_demonstration_enabled", AsyncMock(return_value=True))++ with pytest.raises(HTTPException) as exc_info:+ await workflow_copilot_chat_post(+ api_key_request,+ _make_chat_request(product_action="demonstrate"),+ organization,+ )++ assert exc_info.value.status_code == 400++[email protected]("failure_seam", ["api_key", "config", "browser_session", "workflow", None])+async def test_demonstration_record_resolves_fallible_setup_before_claim(+ monkeypatch: pytest.MonkeyPatch,+ api_key_request: MagicMock,+ organization: SimpleNamespace,+ failure_seam: str | None,+) -> None:+ monkeypatch.setattr(workflow_copilot_route, "_demonstration_cache", AsyncMock(return_value=_FakeCache()))+ resolve_api_key = AsyncMock(return_value="sk-test")+ get_config = AsyncMock(return_value=None)+ if failure_seam == "api_key":+ resolve_api_key.side_effect = RuntimeError("api key failed")+ elif failure_seam == "config":+ get_config.side_effect = RuntimeError("config failed")+ monkeypatch.setattr(+ app.DATABASE.browser_sessions,+ "get_persistent_browser_session",+ AsyncMock(return_value=None if failure_seam == "browser_session" else SimpleNamespace()),+ )+ monkeypatch.setattr(+ app.DATABASE.workflows,+ "get_workflow_by_permanent_id",+ AsyncMock(return_value=None if failure_seam == "workflow" else SimpleNamespace()),+ )+ claim = AsyncMock()+ monkeypatch.setattr(workflow_copilot_route, "_resolve_copilot_api_key", resolve_api_key)+ monkeypatch.setattr(app.AGENT_FUNCTION, "get_copilot_config_for_request", get_config)+ monkeypatch.setattr(workflow_copilot_route, "claim_demonstration_recording", claim)++ record_request = WorkflowCopilotDemonstrationRecordRequest(+ workflow_permanent_id="wpid-1",+ browser_session_id="pbs-live",)+ if failure_seam is None:+ await workflow_copilot_demonstration_record(api_key_request, record_request, organization)+ claim.assert_awaited_once()+ return++ with pytest.raises((RuntimeError, HTTPException)) as exc_info:+ await workflow_copilot_demonstration_record(api_key_request, record_request, organization)++ if isinstance(exc_info.value, HTTPException):+ assert failure_seam in {"browser_session", "workflow"} and exc_info.value.status_code == 404+ claim.assert_not_awaited()+++ ("workflow_permanent_id", "record_status", "expected_status"),+ [("wpid-1", "recording", 409), ("wpid-1", "consumed", 409), ("wpid-other", "stopped", 404)],+)+async def test_demonstration_done_rejects_unfinished_consumed_or_other_workflow(+ monkeypatch: pytest.MonkeyPatch,+ api_key_request: MagicMock,+ organization: SimpleNamespace,+ workflow_permanent_id: str,+ record_status: str,+ expected_status: int,+) -> None:+ cache = _FakeCache()+ monkeypatch.setattr(app, "CACHE", cache)+ monkeypatch.setattr(workflow_copilot_route, "copilot_demonstration_enabled", AsyncMock(return_value=True))+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id=workflow_permanent_id,+ browser_session_id="pbs-live",+ status=record_status, # type: ignore[arg-type]+ event_count=1,+ owner="owner-1" if record_status == "recording" else None,+ owner_seen_at=datetime.now(UTC) if record_status == "recording" else None,+ final_url="https://example.test/done",+ )+ )++ with pytest.raises(HTTPException) as exc_info:+ await workflow_copilot_chat_post(+ api_key_request,+ _make_chat_request(product_action="demonstrate", demonstration_id="demo-1"),+ organization,+ )++ assert exc_info.value.status_code == expected_status@pytest.mark.asyncio@@ -6962,6 +7115,85 @@ async def test_marker_survives_a_finalizer_that_raises_with_no_assistant_row(app.DATABASE.workflow_params.clear_pending_copilot_turn.assert_not_awaited()+async def test_concurrent_demonstration_replay_claim_finalizes_the_losing_turn(+ monkeypatch: pytest.MonkeyPatch, api_key_request: MagicMock, copilot_stream: MagicMock+) -> None:+ captured = install_fake_create(monkeypatch)+ cache = _FakeCache()+ monkeypatch.setattr(app, "CACHE", cache)+ monkeypatch.setattr(workflow_copilot_route, "copilot_demonstration_enabled", AsyncMock(return_value=True))+ cache.store[demonstration_record_key("org-1", "demo-1")] = encode_demonstration_record(+ DemonstrationRecord(+ workflow_permanent_id="wpid-1",+ browser_session_id="pbs-live",+ status="stopped",+ event_count=0,+ owner=None,+ owner_seen_at=None,+ final_url="https://example.test/done",+ )+ )+ chat = SimpleNamespace(+ workflow_copilot_chat_id="chat-1",+ workflow_permanent_id="wpid-1",+ organization_id="org-1",+ proposed_workflow=None,+ auto_accept=False,+ )+ original_workflow = SimpleNamespace(+ workflow_id="wf-canonical",+ title="Original",+ description="Original description",+ workflow_definition=None,+ )+ agent_result = AgentResult(+ user_response="",+ updated_workflow=None,+ global_llm_context=None,+ workflow_yaml=None,+ workflow_was_persisted=False,+ clear_proposed_workflow=False,+ authoring_barred=False,+ resolved_model=None,+ turn_outcome=None,+ cancelled=False,+ has_staged_proposal=False,+ )+ setup_new_copilot_mocks(monkeypatch, chat, original_workflow, agent_result)+ monkeypatch.setattr(+ agent_module,+ "_run_copilot_turn_impl",+ AsyncMock(side_effect=DemonstrationRejection(409, "Replay already claimed")),+ )+ monkeypatch.setattr(workflow_copilot_route, "run_copilot_agent", agent_module.run_copilot_agent)++ await workflow_copilot_chat_post(+ api_key_request,+ _make_chat_request(product_action="demonstrate", demonstration_id="demo-1"),+ SimpleNamespace(organization_id="org-1"),+ )+ handler = captured["handler"]+ assert callable(handler)+ await handler(copilot_stream)++ assistant_messages = [+ call.kwargs+ for call in app.DATABASE.workflow_params.create_workflow_copilot_chat_message.await_args_list+ if call.kwargs.get("sender") == WorkflowCopilotChatSender.AI+ ]+ assert len(assistant_messages) == 1+ turn_outcome = assistant_messages[0]["turn_outcome"]+ assert turn_outcome is not None+ assert turn_outcome.response_kind == "recover"+ assert turn_outcome.terminal_reason == workflow_copilot_route.COPILOT_RECOVERABLE_FAILURE_TERMINAL_REASON+ app.DATABASE.workflow_params.claim_pending_copilot_turn_for_finalisation.assert_awaited_once()+ app.DATABASE.workflow_params.clear_pending_copilot_turn.assert_awaited_once()+ frames = [call.args[0] for call in copilot_stream.send.await_args_list if call.args]+ assert any(isinstance(frame, WorkflowCopilotStreamResponseUpdate) for frame in frames)+ assert not any(isinstance(frame, WorkflowCopilotStreamErrorUpdate) for frame in frames)++@pytest.mark.asyncio@pytest.mark.parametrize("ending", ["finished", "cancelled", "errored"])async def test_a_turn_reconcile_claimed_leaves_canonical_and_its_row_alone(@@ -8265,6 +8497,20 @@ async def test_refine_recording_forces_code_only_mode_when_the_composer_toggle_iassert workflow_params.start_copilot_turn.await_args.kwargs["pending_turn"].copilot_effective_mode == "code"+async def test_demonstration_turn_follows_the_authoring_dial(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ get_config = AsyncMock(return_value=None)+ monkeypatch.setattr(app.AGENT_FUNCTION, "get_copilot_config_for_request", get_config)++ await workflow_copilot_route._resolve_copilot_request_config(+ "org-1", _make_chat_request(product_action="demonstrate", code_block=None)+ )++ get_config.assert_awaited_once_with("org-1", code_block_mode=None)++@pytest.mark.asyncioasync def test_refine_recording_seeds_the_receipt_before_a_pre_callback_exception(monkeypatch: pytest.MonkeyPatch,diff --git a/tests/unit/test_workflow_copilot_route_helpers.py b/tests/unit/test_workflow_copilot_route_helpers.pyindex 682b4ebdf..f68c9310a 100644--- a/tests/unit/test_workflow_copilot_route_helpers.py+++ b/tests/unit/test_workflow_copilot_route_helpers.py@@ -47,6 +47,7 @@ from skyvern.forge.sdk.routes.workflow_copilot import (_should_restore_persisted_workflow,_turn_attachment_ids,_workflow_copilot_ingress_log_fields,+ convert_to_history_messages,)from skyvern.forge.sdk.schemas.workflow_copilot import (CopilotAttachedFile,@@ -660,6 +661,21 @@ def _persisted_message(narrative_payload: dict[str, Any]) -> WorkflowCopilotChat)+def test_chat_history_keeps_demonstration_items_in_the_narrative_payload_only() -> None:+ message = _persisted_message(+ {+ **_non_error_narrative_payload(),+ "demonstrationItems": ["Navigate to example.com", "Click Submit"],+ }+ )++ history = convert_to_history_messages([message])++ assert history[0].narrative_payload is not None+ assert history[0].narrative_payload["demonstrationItems"] == ["Navigate to example.com", "Click Submit"]+ assert "demonstration_items" not in history[0].model_dump()++def _non_error_narrative_payload() -> dict[str, Any]:"""The shape the acting path builds — every key it actually supplies, and nothing else."""return {diff --git a/tests/unit/webeye/test_action_deadline.py b/tests/unit/webeye/test_action_deadline.pyindex a165ad2e7..f039d378d 100644--- a/tests/unit/webeye/test_action_deadline.py+++ b/tests/unit/webeye/test_action_deadline.py@@ -23,6 +23,8 @@ async def test_a_caller_cancelled_before_the_call_takes_its_first_step_still_stainstrumentation = pytest.importorskip("opentelemetry.instrumentation.asyncio")instrumentor = instrumentation.AsyncioInstrumentor()+ # Finalized here rather than inside the recording window: garbage an earlier test left behind.+ gc.collect()instrumentor.instrument()try:with warnings.catch_warnings(record=True) as caught:diff --git a/tests/unit/workflow/test_code_block_escalation.py b/tests/unit/workflow/test_code_block_escalation.pyindex 5d0d52750..24c70bec2 100644--- a/tests/unit/workflow/test_code_block_escalation.py+++ b/tests/unit/workflow/test_code_block_escalation.py@@ -16,6 +16,7 @@ from typing import Anyfrom unittest.mock import AsyncMock, MagicMockimport pytest+from playwright.async_api import Error as PlaywrightErrorfrom playwright.async_api import TimeoutError as PlaywrightTimeoutErrorfrom structlog.testing import capture_logs@@ -40,7 +41,7 @@ from skyvern.forge.sdk.core.skyvern_context import SkyvernContextfrom skyvern.forge.sdk.db.repositories.workflow_runs import WorkflowRunsRepositoryfrom skyvern.forge.sdk.schemas.files import FileInfofrom skyvern.forge.sdk.schemas.tasks import Task, TaskStatus-from skyvern.forge.sdk.workflow.context_manager import WorkflowRunContext+from skyvern.forge.sdk.workflow.context_manager import WorkflowContextManager, WorkflowRunContextfrom skyvern.forge.sdk.workflow.models.block import (_OPERATION_ATTRIBUTION_ALLOWLIST,BlockResult,@@ -328,6 +329,7 @@ async def test_user_defined_failure_records_dedicated_skip_without_healing(monkeresult = await block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=ErrorCode("typed", "reason"),failing_line=1,build_failure_result=build_failure,@@ -363,6 +365,7 @@ async def test_chokepoint_gate_follows_the_org_flag(result = await block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=PlaywrightTimeoutError("timeout"),failing_line=1,build_failure_result=build_failure,@@ -469,6 +472,7 @@ async def test_ai_fallback_never_rescues_an_authoring_run(result = await block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=exception,failing_line=2,build_failure_result=_build_failure,@@ -1009,6 +1013,7 @@ async def _heal(browser_state=browser_state if browser_state is not None else _browser_state(),page=page if page is not None else MagicMock(),authored_code=authored_code,+ code_failure_reason=None,)@@ -3692,6 +3697,7 @@ async def test_self_heal_passes_pre_resolved_browser_state_by_identity(with skyvern_context.scoped(SkyvernContext(organization_id="o_test", workflow_run_id="wr_test")):result = await block._attempt_self_heal(authored_code=None,+ code_failure_reason=None,exception=exc,failing_line=1,recording_page=_recording_page(exc),@@ -4466,6 +4472,7 @@ async def _resolve_failed_heal(with skyvern_context.scoped(SkyvernContext(organization_id="o_test", workflow_run_id="wr_test")):result = await block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=exception,failing_line=1,build_failure_result=_build_failure,@@ -4563,6 +4570,7 @@ async def test_the_escalation_hop_before_the_recovery_runs_decides_what_the_healwith skyvern_context.scoped(SkyvernContext(organization_id="o_test", workflow_run_id="wr_test")):result = await block._attempt_self_heal(authored_code=None,+ code_failure_reason=None,exception=exc,failing_line=1,recording_page=_recording_page(exc),@@ -4737,10 +4745,134 @@ async def test_executed_block_keeps_its_screened_driver_nav_code_through_a_faileassert result.success is Falseassert state["heal_episodes"][0]["status"] == HealStatus.fired_failed- assert result.failure_reason == "agent gave up"+ assert result.failure_reason is not None and result.failure_reason.endswith("\nAI fallback: agent gave up")assert [code for code in result.error_codes if code != "user_code_error"] == expected_codes+_SIGNED_OUT_FAILURE_PAGE = (+ f"Final URL: https://example.com/notauthenticated?session={SECRET_VALUE}\nPage title: Not Authenticated"+)+_RECOVERY_RESOLVED_CODE = "918273"+++async def _execute_signed_out_block(+ monkeypatch: pytest.MonkeyPatch, *, engine: str, data_schema: dict[str, Any] | None+) -> BlockResult:+ block = _make_code_block(+ code=(+ "await page.goto('https://example.com/members')\n"+ "await page.wait_for_selector('text=Signed in as', timeout=5000)\n"+ ),+ steps=[CodeBlockStep(description="Open the members page", line_start=1, line_end=1)],+ data_schema=data_schema,+ )+ fake_browser_state = SimpleNamespace(+ get_working_page=AsyncMock(return_value=MagicMock()),+ browser_artifacts=BrowserArtifacts(),+ navigate_to_url=AsyncMock(return_value=None),+ )+ _patch_execute_chokepoint_environment(+ monkeypatch,+ context=_make_context(with_secret=True),+ fake_browser_state=fake_browser_state,+ use_codeblock_runner=engine == "secure",+ )+ monkeypatch.setattr(+ app.AGENT_FUNCTION,+ "execute_code_block_override",+ AsyncMock(+ return_value=CodeBlockEngineResult(+ block_result=None,+ failure=CodeBlockEngineFailure(+ error_code="browser_operation_failed",+ safe_message=None,+ failure_reason=f"Locator.wait_for: Timeout 5000ms exceeded.\n{_SIGNED_OUT_FAILURE_PAGE}",+ exception_class="playwright._impl._errors.TimeoutError",+ failing_line=2,+ healability_hint=True,+ ),+ )+ if engine == "secure"+ else None+ ),+ )+ monkeypatch.setattr(CodeBlock, "_is_healable_page_failure", MagicMock(return_value=True))+ monkeypatch.setattr(CodeBlock, "record_output_parameter_value", AsyncMock(return_value=None))+ FakeRecorder.reset()+ page_error = PlaywrightError(f"Page.wait_for_selector: Target closed\n{_SIGNED_OUT_FAILURE_PAGE}")++ class _SignedOutRecorder(FakeRecorder):+ def __init__(self, **kwargs: object) -> None:+ super().__init__(**kwargs)+ self.recording_page.goto = AsyncMock(return_value=None)+ self.recording_page.wait_for_selector = AsyncMock(side_effect=page_error)++ monkeypatch.setattr("skyvern.forge.sdk.workflow.models.block.CodeBlockActionRecording", _SignedOutRecorder)+ run_context = SkyvernContext(+ organization_id="o_test", workflow_run_id="wr_test", runtime_secret_values={_RECOVERY_RESOLVED_CODE}+ )+ with skyvern_context.scoped(run_context):+ return await block.execute(+ workflow_run_id="wr_test",+ workflow_run_block_id="wrb_test",+ organization_id="o_test",+ browser_session_id="pbs_test",+ )++[email protected]("engine", ["inline", "secure"])+ ("final_status", "extracted_information", "data_schema"),+ [+ pytest.param(TaskStatus.failed, _UNSET, None, id="recovery_failed"),+ pytest.param(TaskStatus.running, _UNSET, None, id="no_final_status"),+ pytest.param(+ TaskStatus.completed,+ {},+ {"type": "object", "properties": {"report": {"type": "string"}}},+ id="declared_values_missing",+ ),+ ],+)+async def test_failed_ai_fallback_reports_the_code_failure_first(+ monkeypatch: pytest.MonkeyPatch,+ ai_fallback_flag: Callable[[str | None], None],+ engine: str,+ final_status: TaskStatus,+ extracted_information: ExtractedInformation | _Unset,+ data_schema: dict[str, Any] | None,+) -> None:+ monkeypatch.setattr("skyvern.config.settings.ENABLE_SECRET_ARTIFACT_REDACTION", True)+ monkeypatch.setattr(app, "WORKFLOW_CONTEXT_MANAGER", WorkflowContextManager())+ ai_fallback_flag(None)+ _install_db_fakes(monkeypatch, final_status=final_status, extracted_information=extracted_information)+ unhealed = await _execute_signed_out_block(monkeypatch, engine=engine, data_schema=data_schema)++ ai_fallback_flag("o_test")+ state = _install_db_fakes(monkeypatch, final_status=final_status, extracted_information=extracted_information)+ if final_status is TaskStatus.failed:+ state["updated_task"].failure_reason = f"The verification code {_RECOVERY_RESOLVED_CODE} was rejected"+ result = await _execute_signed_out_block(monkeypatch, engine=engine, data_schema=data_schema)++ assert unhealed.failure_reason is not None and "Not Authenticated" in unhealed.failure_reason+ code_reason = unhealed.failure_reason.replace(SECRET_VALUE, "*****")+ fallback_sentence = state["heal_episodes"][0]["failure_message"]+ assert fallback_sentence and "Not Authenticated" not in fallback_sentence+ assert _RECOVERY_RESOLVED_CODE not in fallback_sentence+ assert result.success is False+ assert result.failure_reason == f"{code_reason}\nAI fallback: {fallback_sentence}"+ assert SECRET_VALUE not in result.failure_reason+ parent_reasons = {+ update["failure_reason"]+ for update in state["workflow_run_block_updates"]+ if update.get("workflow_run_block_id") == "wrb_test" and update.get("failure_reason")+ }+ assert parent_reasons == {result.failure_reason}+ child_reasons = {u["failure_reason"] for u in state["recovery_block_updates"] if u.get("failure_reason")}+ assert child_reasons == (set() if final_status is TaskStatus.running else {fallback_sentence})++@pytest.mark.asyncio@pytest.mark.parametrize("engine", ["inline", "secure"])@pytest.mark.parametrize(@@ -4921,6 +5053,7 @@ async def test_recovery_task_is_built_from_the_block_fields(with skyvern_context.scoped(SkyvernContext(organization_id="o_test", workflow_run_id="wr_test")):result = await block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=exception,failing_line=1,build_failure_result=_build_failure,@@ -4974,6 +5107,7 @@ async def test_block_without_prompt_skips_recovery_without_an_llm_call(with capture_logs() as logs:result = await block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=exception,failing_line=2,build_failure_result=_build_failure,@@ -5037,6 +5171,7 @@ async def test_failed_recovery_outcome_carries_task_id_and_action_count(with capture_logs() as logs:result = await block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=exception,failing_line=2,build_failure_result=_build_failure,@@ -5092,6 +5227,7 @@ async def test_heal_episode_persistence_failure_does_not_change_heal_outcome(with capture_logs() as logs:result = await block._resolve_failure_with_heal(authored_code=None,+ code_failure_reason=None,exception=exception,failing_line=2,build_failure_result=_build_failure,diff --git a/uv.lock b/uv.lockindex 12efdf9e4..eeb5ff5da 100644--- a/uv.lock+++ b/uv.lock@@ -6897,7 +6897,7 @@ wheels = [[[package]]name = "skyvern"-version = "1.0.55"+version = "1.0.56"source = { editable = "." }dependencies = [{ name = "cachetools" },