Skip to content

Automate browser based workflows with AI

SKY-13579 Copilot: recognize a saved credential by its name, however the message is phrased (#8107)

Andrew Neilson62bca008a2b5
diff --git a/docs/api-reference/openapi.json b/docs/api-reference/openapi.json
index 858ee32d0..c9d108ddd 100644
--- a/docs/api-reference/openapi.json
+++ b/docs/api-reference/openapi.json
@@ -10185,6 +10185,22 @@
"description": "Whether the browser session supports VNC streaming",
"default": false
},
+ "stream_transport": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "title": "Stream Transport",
+ "description": "Live-view transport for this session: \"vnc\" or \"cdp\". Resolved on the single-session fetch only; null elsewhere.",
+ "examples": [
+ "vnc",
+ "cdp"
+ ]
+ },
"download_path": {
"anyOf": [
{
diff --git a/docs/plans/2026-08-06-sky-13291-vendor-live-view-cdp-screencast.md b/docs/plans/2026-08-06-sky-13291-vendor-live-view-cdp-screencast.md
new file mode 100644
index 000000000..7ff4c66de
--- /dev/null
+++ b/docs/plans/2026-08-06-sky-13291-vendor-live-view-cdp-screencast.md
@@ -0,0 +1,865 @@
+# Vendor Live View via Per-Session CDP Screencast (SKY-13291 slice 3) Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+> **Amended after review — the run surfaces are out of scope, and the viewer connects differently.**
+> Tasks 1–3 below plan a per-session gate on the task and workflow-run stream endpoints, backed by
+> a fallback that resolves a run's browser through its session. Review established that both parts
+> were wrong, and what shipped differs:
+>
+> - **Run surfaces keep the deployment-global gate.** A run's worker already publishes frames for a
+> remote-CDP browser (`CDPFramePublisher` → `StorageBase.get_streaming_file`), which is how those
+> endpoints already serve an externally hosted session. Routing them per session bought nothing —
+> and missed anyway, because a task inside a workflow leases its session under the *run's* id, so
+> resolving by task id never matched. `verify.browser_session_id_for_runnable` was dropped.
+> - **The viewer no longer reuses `get_browser_state`.** For an externally hosted session that adopts
+> the browser through the Fetch-download factory, which would arm a second download interceptor on
+> pages a run is driving and leave the connection in a cache a run later reads. Watching now goes
+> through `get_observer_browser_state` / `release_observer_browser_state`: one uncached,
+> interception-free connection per websocket, given back when the viewer leaves.
+>
+> The per-session gate on the browser-session endpoint, the `stream_transport` field, the frontend
+> selection, and the capability flip are as planned.
+
+**Goal:** Make live-stream transport a per-session decision instead of the deployment-global `BROWSER_STREAMING_MODE`, so external-provider ("vendor") browser sessions stream via CDP `Page.startScreencast` through the session router while first-party sessions keep VNC.
+
+**Architecture:** A new `AgentFunction.resolve_stream_transport` hook answers `"vnc"` or `"cdp"` per session (OSS base returns the global setting; the cloud override answers `"cdp"` for vendor-held rows from `browser_session_infra`). The four backend serving-path gates that read `settings.BROWSER_STREAMING_MODE` switch to the hook, the CDP screencast path learns to resolve a task/workflow-run's browser state through `PERSISTENT_SESSIONS_MANAGER` (remote `connect_over_cdp` on the routed address) when the in-process `BROWSER_MANAGER` has none, `BrowserSessionResponse` grows a `stream_transport` field, and the frontend's 7 stream-selection call sites consult the session's transport with the global mode as fallback. The final, separately-revertable step flips `live_view=True` in the vendor capability table, which simultaneously permits routing `needs_live_view` sessions to vendors and (via the existing `supports_live_view` override, which already delegates to the capability table) makes `vnc_streaming_supported` true for vendor sessions.
+
+**Tech Stack:** Python/FastAPI (skyvern OSS + cloud overlay), Playwright CDP sessions, React/TypeScript + react-query (skyvern-frontend), pytest (`@pytest.mark.asyncio`), vitest.
+
+**Context (verified 2026-08-06):** Both vendors are spike-proven for `Page.startScreencast` on the shared driver connection (41–53 fps, zero driver interference, second concurrent CDP connection accepted). See the SKY-13291 Linear comments for numbers. The viewer stack (screencast loop, CDP input channel, `InteractiveStreamView`) already exists from OSS local mode.
+
+## Global Constraints
+
+- **OSS boundary:** never import `cloud.*` from `skyvern/`. All cloud-specific logic goes through the `AgentFunction` override in `cloud/agent_functions.py`.
+- **OSS sync / non-disclosure:** `skyvern/`, `tests/unit*/` and `docs/` are all synced to the public repo (see `.github/sync.yml` for the full list — this document is inside that surface). No provider name may appear anywhere in it — say "external provider". Provider-specific logic and tests live in `cloud/` and `tests/cloud/`.
+- Line length 120; `structlog` (`LOG = structlog.get_logger()`), imports at top of file.
+- Comments: only for non-obvious constraints; no change-log or task-reference comments.
+- Tests: extend existing `test_X` files for module X; check `tests/unit/conftest.py` fixtures before writing fakes; no exact-prompt or mock-call-count assertions.
+- Branch naming: `{username}/sky-13291-{short-description}`. Commits use conventional messages. No migrations are needed (no schema changes — `stream_transport` is computed, not stored).
+- PR bodies use `.github/pull_request_template.md` (Problem / Solution / How Has This Been Tested?); validate with `python3 .github/scripts/validate_pr_body.py .github/pull_request_template.md <body-file>`.
+- Validation floor per modified Python file: `uv run python -m py_compile <file>`; frontend: `cd skyvern-frontend && npx tsc --noEmit` and `npx eslint <files>`.
+- Suggested PR split: Tasks 1–5 (backend), Tasks 6–7 (frontend), Task 8 (capability flip — its own PR, the rollout switch).
+
+---
+
+### Task 1: `resolve_stream_transport` AgentFunction + resolution helpers
+
+**Files:**
+- Modify: `skyvern/forge/agent_functions.py` (next to `supports_live_view`, ~line 917)
+- Modify: `skyvern/forge/sdk/routes/streaming/verify.py` (module level, after `verify_browser_session`)
+- Test: `tests/unit/forge/sdk/routes/streaming/test_stream_transport.py` (new)
+
+**Interfaces:**
+- Consumes: `settings.BROWSER_STREAMING_MODE` (str, `"vnc"` default), `app.AGENT_FUNCTION`, `app.PERSISTENT_SESSIONS_MANAGER.get_session_by_runnable_id(organization_id=..., runnable_id=...)`.
+- Produces (later tasks rely on these exact names):
+ - `AgentFunction.resolve_stream_transport(*, browser_session_id: str | None, organization_id: str | None) -> str`
+ - `verify.stream_transport(browser_session_id: str | None, organization_id: str) -> str` (never raises)
+ - `verify.browser_session_id_for_runnable(runnable_id: str, organization_id: str) -> str | None` (never raises)
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `tests/unit/forge/sdk/routes/streaming/test_stream_transport.py`:
+
+```python
+from unittest.mock import AsyncMock
+
+import pytest
+
+from skyvern.forge import app
+from skyvern.forge.sdk.routes.streaming import verify
+
+
+async def test_stream_transport_delegates_to_agent_function(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(app, "AGENT_FUNCTION", AsyncMock(resolve_stream_transport=AsyncMock(return_value="cdp")))
+
+ assert await verify.stream_transport("pbs_1", "org_1") == "cdp"
+ app.AGENT_FUNCTION.resolve_stream_transport.assert_awaited_once_with(
+ browser_session_id="pbs_1", organization_id="org_1"
+ )
+
+
+async def test_stream_transport_falls_back_to_setting_on_error(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(
+ app, "AGENT_FUNCTION", AsyncMock(resolve_stream_transport=AsyncMock(side_effect=RuntimeError("db down")))
+ )
+ monkeypatch.setattr(verify.settings, "BROWSER_STREAMING_MODE", "vnc")
+
+ assert await verify.stream_transport("pbs_1", "org_1") == "vnc"
+
+
+async def test_base_agent_function_returns_deployment_setting(monkeypatch: pytest.MonkeyPatch) -> None:
+ from skyvern.forge.agent_functions import AgentFunction
+
+ monkeypatch.setattr(verify.settings, "BROWSER_STREAMING_MODE", "cdp")
+
+ assert await AgentFunction().resolve_stream_transport(browser_session_id=None, organization_id=None) == "cdp"
+
+
+async def test_browser_session_id_for_runnable_resolves_and_swallows(monkeypatch: pytest.MonkeyPatch) -> None:
+ session = AsyncMock()
+ session.persistent_browser_session_id = "pbs_9"
+ manager = AsyncMock(get_session_by_runnable_id=AsyncMock(return_value=session))
+ monkeypatch.setattr(app, "PERSISTENT_SESSIONS_MANAGER", manager)
+
+ assert await verify.browser_session_id_for_runnable("wr_1", "org_1") == "pbs_9"
+
+ manager.get_session_by_runnable_id.side_effect = RuntimeError("db down")
+ assert await verify.browser_session_id_for_runnable("wr_1", "org_1") is None
+```
+
+If this directory's existing tests use a different async marker or app-patching fixture (see `test_verify_browser_session.py`), match that style instead.
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `uv run pytest tests/unit/forge/sdk/routes/streaming/test_stream_transport.py -v`
+Expected: FAIL — `AttributeError` / `ImportError` (no `resolve_stream_transport`, no `verify.stream_transport`).
+
+- [ ] **Step 3: Implement the AgentFunction base**
+
+In `skyvern/forge/agent_functions.py`, directly below `supports_live_view` (the file already imports `settings`):
+
+```python
+ async def resolve_stream_transport(
+ self, *, browser_session_id: str | None, organization_id: str | None
+ ) -> str:
+ """Which live-view transport serves this session: "vnc" or "cdp".
+
+ A self-hosted deployment streams every browser the same way, so the
+ deployment-wide setting decides.
+ """
+ return settings.BROWSER_STREAMING_MODE
+```
+
+- [ ] **Step 4: Implement the helpers in verify.py**
+
+In `skyvern/forge/sdk/routes/streaming/verify.py` (module already imports `app`, `settings`, `LOG`):
+
+```python
+async def stream_transport(browser_session_id: str | None, organization_id: str) -> str:
+ try:
+ return await app.AGENT_FUNCTION.resolve_stream_transport(
+ browser_session_id=browser_session_id, organization_id=organization_id
+ )
+ except Exception:
+ LOG.warning(
+ "Could not resolve the stream transport; using the deployment default",
+ browser_session_id=browser_session_id,
+ organization_id=organization_id,
+ exc_info=True,
+ )
+ return settings.BROWSER_STREAMING_MODE
+
+
+async def browser_session_id_for_runnable(runnable_id: str, organization_id: str) -> str | None:
+ try:
+ session = await app.PERSISTENT_SESSIONS_MANAGER.get_session_by_runnable_id(
+ organization_id=organization_id, runnable_id=runnable_id
+ )
+ except Exception:
+ LOG.warning(
+ "Could not resolve a browser session for this runnable",
+ runnable_id=runnable_id,
+ organization_id=organization_id,
+ exc_info=True,
+ )
+ return None
+ return session.persistent_browser_session_id if session else None
+```
+
+- [ ] **Step 5: Run tests to verify they pass**
+
+Run: `uv run pytest tests/unit/forge/sdk/routes/streaming/test_stream_transport.py -v`
+Expected: PASS (4 tests).
+
+- [ ] **Step 6: Compile check and commit**
+
+```bash
+uv run python -m py_compile skyvern/forge/agent_functions.py skyvern/forge/sdk/routes/streaming/verify.py
+git add skyvern/forge/agent_functions.py skyvern/forge/sdk/routes/streaming/verify.py \
+ tests/unit/forge/sdk/routes/streaming/test_stream_transport.py
+git commit -m "feat(SKY-13291): add per-session stream-transport resolution hook"
+```
+
+---
+
+### Task 2: Per-session transport at the four serving-path gates
+
+**Files:**
+- Modify: `skyvern/forge/sdk/routes/streaming/screenshot.py:68` (task stream), `:191` (workflow-run stream), `:339` (browser-session stream)
+- Modify: `skyvern/forge/sdk/routes/streaming/verify.py:94` (address-readiness bypass inside `verify_browser_session`)
+- Test: `tests/unit/forge/sdk/routes/streaming/test_verify_browser_session.py` (extend)
+
+**Interfaces:**
+- Consumes: `verify.stream_transport(...)` and `verify.browser_session_id_for_runnable(...)` from Task 1; existing `_local_screencast_for_task/_workflow_run/_browser_session` helpers in `screenshot.py`.
+- Produces: no new symbols — behavior change only. After this task, a deployment with `BROWSER_STREAMING_MODE=vnc` serves the CDP screencast to any session whose `resolve_stream_transport` answers `"cdp"`, and vice versa.
+
+- [ ] **Step 1: Write the failing test**
+
+Append to `tests/unit/forge/sdk/routes/streaming/test_verify_browser_session.py` (match its existing fixtures for building a ready session; the essential assertions):
+
+```python
+async def test_verify_browser_session_cdp_transport_bypasses_missing_address(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """A session whose transport is cdp verifies without a browser address; a vnc session still requires one."""
+ session = _make_session(is_browser_ready=False, browser_address=None) # reuse this file's session factory
+ manager = AsyncMock(get_session=AsyncMock(return_value=session))
+ monkeypatch.setattr(app, "PERSISTENT_SESSIONS_MANAGER", manager)
+ monkeypatch.setattr(app, "AGENT_FUNCTION", AsyncMock(resolve_stream_transport=AsyncMock(return_value="cdp")))
+ monkeypatch.setattr(verify.settings, "BROWSER_STREAMING_MODE", "vnc")
+
+ result = await verify.verify_browser_session("pbs_1", "org_1")
+ assert result is not None
+ assert result.browser_address == ""
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `uv run pytest tests/unit/forge/sdk/routes/streaming/test_verify_browser_session.py -v -k cdp_transport`
+Expected: FAIL — with the global set to `vnc`, the current code takes the address-lookup branch and returns `None`.
+
+- [ ] **Step 3: Swap the verify.py bypass**
+
+In `verify_browser_session` (`verify.py:94`), replace:
+
+```python
+ if not browser_address:
+ if settings.BROWSER_STREAMING_MODE == "cdp":
+ browser_address = ""
+```
+
+with:
+
+```python
+ if not browser_address:
+ if await stream_transport(browser_session_id, organization_id) == "cdp":
+ browser_address = ""
+```
+
+- [ ] **Step 4: Swap the three screenshot.py gates**
+
+`screenshot.py` already imports from `verify`; extend that import with `stream_transport` and `browser_session_id_for_runnable`.
+
+Task stream (line 68):
+
+```python
+ task_session_id = await browser_session_id_for_runnable(task_id, organization_id)
+ if await stream_transport(task_session_id, organization_id) == "cdp":
+ await _local_screencast_for_task(websocket, task_id, organization_id)
+ return
+```
+
+Workflow-run stream (line 191):
+
+```python
+ run_session_id = await browser_session_id_for_runnable(workflow_run_id, organization_id)
+ if await stream_transport(run_session_id, organization_id) == "cdp":
+ await _local_screencast_for_workflow_run(websocket, workflow_run_id, organization_id)
+ return
+```
+
+Browser-session stream (line 339):
+
+```python
+ if await stream_transport(browser_session_id, organization_id) == "cdp":
+ await _local_screencast_for_browser_session(websocket, browser_session_id, organization_id)
+ return
+```
+
+Leave `settings.BROWSER_STREAMING_MODE` reads outside the serving path untouched (`runtime_config.py`, `forge_app.py`, `default_persistent_sessions_manager.py`, CLI).
+
+- [ ] **Step 5: Run the streaming suites**
+
+Run: `uv run pytest tests/unit/forge/sdk/routes/streaming/ tests/unit_tests/test_streaming_screencast.py -v`
+Expected: PASS (new test green, no regressions — existing tests exercising the old gates keep passing because the base hook returns the same setting).
+
+- [ ] **Step 6: Compile check and commit**
+
+```bash
+uv run python -m py_compile skyvern/forge/sdk/routes/streaming/screenshot.py skyvern/forge/sdk/routes/streaming/verify.py
+git add skyvern/forge/sdk/routes/streaming/screenshot.py skyvern/forge/sdk/routes/streaming/verify.py \
+ tests/unit/forge/sdk/routes/streaming/test_verify_browser_session.py
+git commit -m "feat(SKY-13291): gate stream endpoints on per-session transport"
+```
+
+---
+
+### Task 3: Screencast browser-state fallback through the persistent-session manager
+
+**Files:**
+- Modify: `skyvern/forge/sdk/routes/streaming/screencast.py:53-65` (`_resolve_browser_state`)
+- Test: `tests/unit_tests/test_streaming_screencast.py` (extend — this is the module's existing test file)
+
+**Interfaces:**
+- Consumes: `app.BROWSER_MANAGER.get_for_workflow_run(entity_id)` / `.get_for_task(entity_id, workflow_run_id)` (in-process, sync); `app.PERSISTENT_SESSIONS_MANAGER.get_session_by_runnable_id(...)` and `.get_browser_state(session_id, organization_id)` (async, does `connect_over_cdp` on the routed address in cloud).
+- Produces: `_resolve_browser_state` now returns a `BrowserState` for task/workflow-run entities whose browser lives behind a persistent session even when this process holds no in-process state (the cloud API server case). `wait_for_browser_state`, the screencast loop, and `cdp_input.py` (which imports `_resolve_working_page`) inherit the behavior unchanged.
+
+**Why:** the run/task stream endpoints run on the API server; in cloud, the in-process `BROWSER_MANAGER` belongs to the temporal worker, so `get_for_workflow_run`/`get_for_task` return `None` there. The run's browser is still reachable: entity → persistent session → `get_browser_state` (remote CDP).
+
+- [ ] **Step 1: Write the failing tests**
+
+Append to `tests/unit_tests/test_streaming_screencast.py` (reuse its fake-browser-state helpers; `@pytest.mark.asyncio` is this file's convention):
+
+```python
+async def test_resolve_workflow_run_falls_back_to_persistent_session(monkeypatch: pytest.MonkeyPatch) -> None:
+ state = object()
+ session = AsyncMock()
+ session.persistent_browser_session_id = "pbs_7"
+ monkeypatch.setattr(app, "BROWSER_MANAGER", MagicMock(get_for_workflow_run=MagicMock(return_value=None)))
+ monkeypatch.setattr(
+ app,
+ "PERSISTENT_SESSIONS_MANAGER",
+ AsyncMock(
+ get_session_by_runnable_id=AsyncMock(return_value=session),
+ get_browser_state=AsyncMock(return_value=state),
+ ),
+ )
+
+ resolved = await screencast._resolve_browser_state("wr_1", "workflow_run", organization_id="org_1")
+ assert resolved is state
+
+
+async def test_resolve_workflow_run_without_session_or_org_returns_none(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(app, "BROWSER_MANAGER", MagicMock(get_for_workflow_run=MagicMock(return_value=None)))
+ monkeypatch.setattr(
+ app, "PERSISTENT_SESSIONS_MANAGER", AsyncMock(get_session_by_runnable_id=AsyncMock(return_value=None))
+ )
+
+ assert await screencast._resolve_browser_state("wr_1", "workflow_run", organization_id="org_1") is None
+ assert await screencast._resolve_browser_state("wr_1", "workflow_run", organization_id=None) is None
+```
+
+Guard against `MagicMock` auto-attributes making assertions vacuous: assert identity (`is state`), not truthiness.
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `uv run pytest tests/unit_tests/test_streaming_screencast.py -v -k persistent_session`
+Expected: FAIL — current code returns whatever `BROWSER_MANAGER` returned (`None`) with no fallback.
+
+- [ ] **Step 3: Implement the fallback**
+
+Replace `_resolve_browser_state` in `screencast.py`:
+
+```python
+async def _resolve_browser_state(
+ entity_id: str,
+ entity_type: str,
+ workflow_run_id: str | None = None,
+ organization_id: str | None = None,
+) -> BrowserState | None:
+ if entity_type == "workflow_run":
+ state = app.BROWSER_MANAGER.get_for_workflow_run(entity_id)
+ return state if state is not None else await _state_via_persistent_session(entity_id, organization_id)
+ if entity_type == "task":
+ state = app.BROWSER_MANAGER.get_for_task(entity_id, workflow_run_id)
+ return state if state is not None else await _state_via_persistent_session(entity_id, organization_id)
+ if entity_type == "browser_session":
+ return await app.PERSISTENT_SESSIONS_MANAGER.get_browser_state(entity_id, organization_id)
+ return None
+
+
+async def _state_via_persistent_session(runnable_id: str, organization_id: str | None) -> BrowserState | None:
+ """The runnable's browser may live behind a persistent session another process owns; reach it remotely."""
+ if not organization_id:
+ return None
+ try:
+ session = await app.PERSISTENT_SESSIONS_MANAGER.get_session_by_runnable_id(
+ organization_id=organization_id, runnable_id=runnable_id
+ )
+ if not session:
+ return None
+ return await app.PERSISTENT_SESSIONS_MANAGER.get_browser_state(
+ session.persistent_browser_session_id, organization_id
+ )
+ except Exception:
+ LOG.debug("Could not resolve browser state via persistent session", runnable_id=runnable_id, exc_info=True)
+ return None
+```
+
+- [ ] **Step 4: Run the full screencast suite**
+
+Run: `uv run pytest tests/unit_tests/test_streaming_screencast.py tests/unit/forge/sdk/routes/streaming/ -v`
+Expected: PASS — including the SKY-11215 rebind tests (the fallback only engages when the manager returns `None`, so the resolve-each-poll behavior is unchanged).
+
+- [ ] **Step 5: Compile check and commit**
+
+```bash
+uv run python -m py_compile skyvern/forge/sdk/routes/streaming/screencast.py
+git add skyvern/forge/sdk/routes/streaming/screencast.py tests/unit_tests/test_streaming_screencast.py
+git commit -m "feat(SKY-13291): resolve screencast browser state via persistent sessions when out-of-process"
+```
+
+---
+
+### Task 4: `stream_transport` on `BrowserSessionResponse`
+
+**Files:**
+- Modify: `skyvern/webeye/schemas.py` (field near `vnc_streaming_supported` at line 76; factory `from_browser_session` near line 176)
+- Test: `tests/unit/webeye/test_browser_session_response.py` (extend)
+
+**Interfaces:**
+- Consumes: `app.AGENT_FUNCTION.resolve_stream_transport(...)` from Task 1.
+- Produces: `BrowserSessionResponse.stream_transport: str | None` — the value the frontend reads in Task 6. `"vnc"` or `"cdp"`; `None` only for old serialized payloads.
+
+**Note:** additive optional field ⇒ Fern SDK drift. Follow `cloud_docs/fern-sdk/` regeneration policy in the PR (drift check is advisory; call the field out in the PR body).
+
+- [ ] **Step 1: Write the failing test**
+
+Append to `tests/unit/webeye/test_browser_session_response.py` (reuse its session/factory fixtures):
+
+```python
+async def test_response_carries_stream_transport(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(
+ app, "AGENT_FUNCTION", AsyncMock(resolve_stream_transport=AsyncMock(return_value="cdp"), **_agent_function_defaults())
+ )
+ response = await BrowserSessionResponse.from_browser_session(_make_browser_session())
+ assert response.stream_transport == "cdp"
+```
+
+Where `_agent_function_defaults()` supplies whatever other AgentFunction methods this factory already calls (`resolve_browser_session_connect_url`, `supports_live_view`, …) — reuse the file's existing stubbing pattern rather than inventing a new one.
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `uv run pytest tests/unit/webeye/test_browser_session_response.py -v -k stream_transport`
+Expected: FAIL — `stream_transport` not a field.
+
+- [ ] **Step 3: Add the field and populate it**
+
+Field (next to `vnc_streaming_supported`, line ~76):
+
+```python
+ stream_transport: str | None = Field(
+ None,
+ description='Live-view transport for this session: "vnc" or "cdp".',
+ examples=["vnc", "cdp"],
+ )
+```
+
+In `from_browser_session`, alongside the `vnc_streaming_supported=` argument:
+
+```python
+ stream_transport=await app.AGENT_FUNCTION.resolve_stream_transport(
+ browser_session_id=browser_session.persistent_browser_session_id,
+ organization_id=browser_session.organization_id,
+ ),
+```
+
+- [ ] **Step 4: Run tests to verify they pass**
+
+Run: `uv run pytest tests/unit/webeye/test_browser_session_response.py tests/cloud/test_vendor_identity_nondisclosure.py -v`
+Expected: PASS — including the non-disclosure suite: the field carries only a neutral transport word, never a vendor identity or URL.
+
+- [ ] **Step 5: Compile check and commit**
+
+```bash
+uv run python -m py_compile skyvern/webeye/schemas.py
+git add skyvern/webeye/schemas.py tests/unit/webeye/test_browser_session_response.py
+git commit -m "feat(SKY-13291): expose per-session stream transport on browser session responses"
+```
+
+---
+
+### Task 5: Cloud override — vendor-held sessions stream via CDP
+
+**Files:**
+- Modify: `cloud/agent_functions.py` (next to `supports_live_view`, ~line 868)
+- Test: `tests/cloud/test_stream_transport_resolution.py` (new — per-function cloud test files are this repo's pattern, cf. `test_browser_session_connect_url.py`)
+
+**Interfaces:**
+- Consumes: `cloud_db.browser_session_infra.get_browser_session_infra(persistent_browser_session_id) -> SessionInfra` (missing row ⇒ `FIRST_PARTY`; `SessionInfra.is_vendor: bool`). Both already imported/used in this module by `supports_live_view`.
+- Produces: cloud `resolve_stream_transport` returning `"cdp"` for vendor rows, else `settings.BROWSER_STREAMING_MODE`.
+
+- [ ] **Step 1: Write the failing tests**
+
+Create `tests/cloud/test_stream_transport_resolution.py` (use the same infra-row fixtures as `tests/cloud/test_browser_session_infra.py` / the stubbing style of `test_browser_session_connect_url.py`):
+
+```python
+from unittest.mock import AsyncMock
+
+import pytest
+
+from cloud import agent_functions as cloud_agent_functions
+from cloud.db.repositories.browser_session_infra import FIRST_PARTY, SessionInfra
+from cloud.webeye.browser_types import BrowserInfraProvider
+
+
+async def test_vendor_session_streams_cdp(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(
+ cloud_agent_functions.cloud_db.browser_session_infra,
+ "get_browser_session_infra",
+ AsyncMock(return_value=SessionInfra(provider=EXTERNAL_PROVIDER)),
+ )
+ fn = cloud_agent_functions.CloudAgentFunction()
+ assert await fn.resolve_stream_transport(browser_session_id="pbs_1", organization_id="org_1") == "cdp"
+
+
+async def test_first_party_session_uses_deployment_setting(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(
+ cloud_agent_functions.cloud_db.browser_session_infra,
+ "get_browser_session_infra",
+ AsyncMock(return_value=FIRST_PARTY),
+ )
+ monkeypatch.setattr(cloud_agent_functions.settings, "BROWSER_STREAMING_MODE", "vnc")
+ fn = cloud_agent_functions.CloudAgentFunction()
+ assert await fn.resolve_stream_transport(browser_session_id="pbs_1", organization_id="org_1") == "vnc"
+
+
+async def test_infra_lookup_failure_falls_back_to_setting(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(
+ cloud_agent_functions.cloud_db.browser_session_infra,
+ "get_browser_session_infra",
+ AsyncMock(side_effect=RuntimeError("db down")),
+ )
+ monkeypatch.setattr(cloud_agent_functions.settings, "BROWSER_STREAMING_MODE", "vnc")
+ fn = cloud_agent_functions.CloudAgentFunction()
+ assert await fn.resolve_stream_transport(browser_session_id="pbs_1", organization_id="org_1") == "vnc"
+
+
+async def test_no_session_id_uses_deployment_setting(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(cloud_agent_functions.settings, "BROWSER_STREAMING_MODE", "vnc")
+ fn = cloud_agent_functions.CloudAgentFunction()
+ assert await fn.resolve_stream_transport(browser_session_id=None, organization_id="org_1") == "vnc"
+```
+
+Adjust the class name if the override class in `cloud/agent_functions.py` is not `CloudAgentFunction` — use whatever class holds `supports_live_view`, and construct it the way `test_browser_session_connect_url.py` does.
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `uv run pytest tests/cloud/test_stream_transport_resolution.py -v`
+Expected: FAIL — the override does not exist, so the base returns the setting and the vendor test asserts `"cdp"` ≠ `"vnc"`.
+
+- [ ] **Step 3: Implement the override**
+
+In `cloud/agent_functions.py`, directly below `supports_live_view`:
+
+```python
+ async def resolve_stream_transport(
+ self, *, browser_session_id: str | None, organization_id: str | None
+ ) -> str:
+ """A vendor-held browser has no websockify sidecar to relay RFB from, but its CDP endpoint
+ is reachable through the session router — so it streams via CDP screencast. First-party
+ sessions keep the deployment default."""
+ if browser_session_id:
+ try:
+ infra = await cloud_db.browser_session_infra.get_browser_session_infra(browser_session_id)
+ except Exception:
+ LOG.warning(
+ "Could not resolve session infrastructure for stream transport; using the deployment default",
+ browser_session_id=browser_session_id,
+ exc_info=True,
+ )
+ return settings.BROWSER_STREAMING_MODE
+ if infra.is_vendor:
+ return "cdp"
+ return settings.BROWSER_STREAMING_MODE
+```
+
+- [ ] **Step 4: Run the cloud suites**
+
+Run: `uv run pytest tests/cloud/test_stream_transport_resolution.py tests/cloud/test_vendor_identity_nondisclosure.py tests/cloud/test_browser_session_infra.py -v`
+Expected: PASS.
+
+- [ ] **Step 5: Compile check and commit**
+
+```bash
+uv run python -m py_compile cloud/agent_functions.py
+git add cloud/agent_functions.py tests/cloud/test_stream_transport_resolution.py
+git commit -m "feat(SKY-13291): stream vendor-held sessions via CDP screencast transport"
+```
+
+---
+
+### Task 6: Frontend — session type, transport resolver, and hook
+
+**Files:**
+- Modify: `skyvern-frontend/src/routes/workflows/types/browserSessionTypes.ts` (line ~18, next to `vnc_streaming_supported`)
+- Modify: `skyvern-frontend/src/hooks/useRuntimeConfig.ts`
+- Test: `skyvern-frontend/src/hooks/useRuntimeConfig.test.ts` (extend)
+
+**Interfaces:**
+- Consumes: `BrowserSessionResponse.stream_transport` from Task 4; existing `normalizeBrowserStreamingMode`, `useBrowserStreamingMode`, `getClient`, `useCredentialGetter`.
+- Produces (Task 7 relies on these exact names):
+ - `resolveStreamTransport(globalMode: BrowserStreamingMode, sessionTransport: string | null | undefined): BrowserStreamingMode` (pure, exported)
+ - `useStreamTransport(browserSessionId?: string | null): { streamTransport: BrowserStreamingMode }` (exported hook; falls back to the global mode while loading or when the id is absent)
+ - `BrowserSessionType.stream_transport?: string | null`
+
+- [ ] **Step 1: Write the failing test**
+
+Append to `skyvern-frontend/src/hooks/useRuntimeConfig.test.ts`:
+
+```ts
+import { resolveStreamTransport } from "./useRuntimeConfig";
+
+describe("resolveStreamTransport", () => {
+ it("prefers the session's transport over the global mode", () => {
+ expect(resolveStreamTransport("vnc", "cdp")).toBe("cdp");
+ expect(resolveStreamTransport("cdp", "vnc")).toBe("vnc");
+ });
+
+ it("falls back to the global mode when the session gives none or garbage", () => {
+ expect(resolveStreamTransport("vnc", null)).toBe("vnc");
+ expect(resolveStreamTransport("vnc", undefined)).toBe("vnc");
+ expect(resolveStreamTransport("cdp", "")).toBe("cdp");
+ expect(resolveStreamTransport("vnc", "webrtc")).toBe("vnc");
+ });
+});
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `cd skyvern-frontend && npx vitest run src/hooks/useRuntimeConfig.test.ts`
+Expected: FAIL — `resolveStreamTransport` is not exported.
+
+- [ ] **Step 3: Implement type, resolver, and hook**
+
+`browserSessionTypes.ts` — add below `vnc_streaming_supported`:
+
+```ts
+ stream_transport?: string | null;
+```
+
+`useRuntimeConfig.ts` — add (and export) below `useBrowserStreamingMode`; add `import { useCredentialGetter } from "@/hooks/useCredentialGetter";` (same import path `BrowserSession.tsx` uses):
+
+```ts
+function resolveStreamTransport(
+ globalMode: BrowserStreamingMode,
+ sessionTransport: string | null | undefined,
+): BrowserStreamingMode {
+ if (!sessionTransport || !STREAMING_MODES.has(sessionTransport.trim().toLowerCase())) {
+ return globalMode;
+ }
+ return normalizeBrowserStreamingMode(sessionTransport);
+}
+
+function useStreamTransport(browserSessionId?: string | null) {
+ const { browserStreamingMode } = useBrowserStreamingMode();
+ const credentialGetter = useCredentialGetter();
+ const query = useQuery<{ stream_transport?: string | null }>({
+ queryKey: ["browserSession", browserSessionId],
+ queryFn: async () => {
+ const client = await getClient(credentialGetter, "sans-api-v1");
+ return client
+ .get(`/browser_sessions/${browserSessionId}`)
+ .then((response) => response.data);
+ },
+ enabled: Boolean(browserSessionId),
+ staleTime: 5 * 60 * 1000,
+ refetchOnWindowFocus: false,
+ retry: 1,
+ });
+
+ return {
+ streamTransport: resolveStreamTransport(
+ browserStreamingMode,
+ query.data?.stream_transport,
+ ),
+ };
+}
+```
+
+Add both to the file's export list. Note: the `["browserSession", id]` queryKey deliberately matches `BrowserSession.tsx`'s existing session query so react-query dedupes the fetch on pages that already load the session.
+
+- [ ] **Step 4: Run tests and type check**
+
+Run: `cd skyvern-frontend && npx vitest run src/hooks/useRuntimeConfig.test.ts && npx tsc --noEmit`
+Expected: PASS / clean.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add skyvern-frontend/src/hooks/useRuntimeConfig.ts skyvern-frontend/src/hooks/useRuntimeConfig.test.ts \
+ skyvern-frontend/src/routes/workflows/types/browserSessionTypes.ts
+git commit -m "feat(SKY-13291): per-session stream transport resolver and hook"
+```
+
+---
+
+### Task 7: Frontend — switch the stream-selection call sites
+
+**Files (all Modify):**
+- `skyvern-frontend/src/routes/browserSessions/BrowserSession.tsx` (~line 51)
+- `skyvern-frontend/src/routes/workflows/studio/StreamPresenter.tsx` (~line 30)
+- `skyvern-frontend/src/routes/workflows/studio/runview/RunLiveStream.tsx` (~line 26)
+- `skyvern-frontend/src/routes/tasks/detail/TaskActions.tsx` (~line 130)
+- `skyvern-frontend/src/routes/workflows/workflowRun/WorkflowRunOverview.tsx` (~lines 51, 123)
+- `skyvern-frontend/src/routes/workflows/editor/Workspace.tsx` (~lines 532, 582–585)
+
+**Interfaces:**
+- Consumes: `useStreamTransport(browserSessionId)` and `resolveStreamTransport(...)` from Task 6.
+- Produces: no new symbols. `BrowserPaneHeader.tsx`'s `StreamModeBadge` intentionally keeps the global mode — it is a DEV-only diagnostic of deployment config, not a per-session control.
+
+Per-site edits (each is: swap the hook, keep the rest of the expression identical):
+
+- [ ] **Step 1: BrowserSession.tsx** — it already fetches the full session object, so use the pure resolver, not the fetching hook. Replace lines 51–52:
+
+```tsx
+ const { browserStreamingMode } = useBrowserStreamingMode();
+ const isCdpMode = browserStreamingMode === "cdp";
+```
+
+with (the `browserSession` query result already exists below; move this derivation after it):
+
+```tsx
+ const { browserStreamingMode } = useBrowserStreamingMode();
+ const isCdpMode =
+ resolveStreamTransport(browserStreamingMode, browserSession?.stream_transport) === "cdp";
+```
+
+- [ ] **Step 2: StreamPresenter.tsx** — replace lines 30–31:
+
+```tsx
+ const { browserStreamingMode } = useBrowserStreamingMode();
+ const useCdp = browserStreamingMode === "cdp" && !isRecording;
+```
+
+with:
+
+```tsx
+ const { streamTransport } = useStreamTransport(browserSessionId);
+ const useCdp = streamTransport === "cdp" && !isRecording;
+```
+
+- [ ] **Step 3: RunLiveStream.tsx** — replace lines 26 and 34:
+
+```tsx
+ const { streamTransport } = useStreamTransport(browserSessionId);
+ ...
+ const useVnc =
+ Boolean(browserSessionId) && streamTransport !== "cdp" && !vncFailed;
+```
+
+- [ ] **Step 4: TaskActions.tsx** — replace lines 130–131 (note `browserSessionId` is declared on line 129, before the current hook call):
+
+```tsx
+ const { streamTransport } = useStreamTransport(browserSessionId);
+ const shouldUseCdpStream = streamTransport === "cdp";
+```
+
+- [ ] **Step 5: WorkflowRunOverview.tsx** — line 62 already has `const browserSessionId = workflowRun?.browser_session_id;`. Replace line 51's hook and line 123:
+
+```tsx
+ const { streamTransport } = useStreamTransport(browserSessionId);
+ ...
+ const shouldUseCdpStream = streamTransport === "cdp";
+```
+
+Move the hook call below the `workflowRun` query so `browserSessionId` is in scope (hooks order stays unconditional).
+
+- [ ] **Step 6: Workspace.tsx** — the active debug session provides the id. Replace line 532's hook and lines 581–585:
+
+```tsx
+ const { streamTransport } = useStreamTransport(activeDebugSession?.browser_session_id);
+ ...
+ const isCdpStreamingMode = streamTransport === "cdp" && !recordingStore.isRecording;
+ // Record Browser exfiltration requires VNC even when the transport is CDP streaming.
+ const preferVncStream = streamTransport !== "cdp" || recordingStore.isRecording;
+```
+
+If `activeDebugSession` is declared after line 532, move the hook call to just after that declaration (unconditional call order preserved). Update the comment at line ~2547 (`mode comes from BROWSER_STREAMING_MODE / runtime config`) to `mode comes from the session's stream transport, falling back to runtime config`.
+
+- [ ] **Step 7: Type check, lint, and grep for leftovers**
+
+```bash
+cd skyvern-frontend && npx tsc --noEmit && npx eslint \
+ src/routes/browserSessions/BrowserSession.tsx src/routes/workflows/studio/StreamPresenter.tsx \
+ src/routes/workflows/studio/runview/RunLiveStream.tsx src/routes/tasks/detail/TaskActions.tsx \
+ src/routes/workflows/workflowRun/WorkflowRunOverview.tsx src/routes/workflows/editor/Workspace.tsx
+grep -rn "useBrowserStreamingMode" src/routes | grep -v BrowserPaneHeader
+```
+
+Expected: clean; the grep returns only `BrowserSession.tsx` (resolver pattern keeps the global hook there) and nothing else outside the intentionally-global badge.
+
+- [ ] **Step 8: Commit**
+
+```bash
+git add skyvern-frontend/src/routes/browserSessions/BrowserSession.tsx \
+ skyvern-frontend/src/routes/workflows/studio/StreamPresenter.tsx \
+ skyvern-frontend/src/routes/workflows/studio/runview/RunLiveStream.tsx \
+ skyvern-frontend/src/routes/tasks/detail/TaskActions.tsx \
+ skyvern-frontend/src/routes/workflows/workflowRun/WorkflowRunOverview.tsx \
+ skyvern-frontend/src/routes/workflows/editor/Workspace.tsx
+git commit -m "feat(SKY-13291): select live-stream component per session transport"
+```
+
+---
+
+### Task 8: Capability flip — vendors serve live view (rollout switch, own PR)
+
+**Files:**
+- Modify: `cloud/webeye/browser_types.py` (`PROVIDER_CAPABILITIES`, ~lines 133–145, and the `live_view` comment ~line 121)
+- Test: `tests/cloud/test_pbs_infra_routing.py` (extend/adjust)
+
+**Interfaces:**
+- Consumes: everything above — this is the switch that routes `needs_live_view=True` sessions to vendors AND (through the existing cloud `supports_live_view`, which returns `capabilities_for(infra.provider).live_view`) makes `vnc_streaming_supported=True` on vendor session responses.
+- Produces: `live_view is True` in `PROVIDER_CAPABILITIES` for every external provider.
+
+**Do not merge until Tasks 1–7 are deployed and a staging vendor session shows a working stream.** Reverting this one PR restores the pre-feature routing exactly.
+
+- [ ] **Step 1: Write/adjust the failing tests**
+
+In `tests/cloud/test_pbs_infra_routing.py`, find the test asserting `needs_live_view` forces first-party (capability override) and split the assertion:
+
+```python
+async def test_needs_live_view_no_longer_forces_first_party() -> None:
+ for provider in EXTERNAL_PROVIDERS:
+ assert capabilities_for(provider).live_view is True
+```
+
+Also update (do not delete) any existing test that pinned `live_view=False` — its name documents the old deliberate design; rewrite it to pin the new one.
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `uv run pytest tests/cloud/test_pbs_infra_routing.py -v`
+Expected: FAIL on the new assertion.
+
+- [ ] **Step 3: Flip the table**
+
+In `PROVIDER_CAPABILITIES`, set `live_view=True` for every external provider entry. Replace the `live_view` field comment (lines ~118–121) with:
+
+```python
+ # First-party sessions stream RFB from the pod's websockify sidecar; vendor-held sessions
+ # stream CDP screencast through the session router (resolve_stream_transport decides per row).
+ live_view: bool
+```
+
+- [ ] **Step 4: Run the cloud suites**
+
+Run: `uv run pytest tests/cloud/test_pbs_infra_routing.py tests/cloud/test_stream_transport_resolution.py tests/cloud/test_pbs_vendor_session_lifecycle.py -v`
+Expected: PASS.
+
+- [ ] **Step 5: Compile check and commit**
+
+```bash
+uv run python -m py_compile cloud/webeye/browser_types.py
+git add cloud/webeye/browser_types.py tests/cloud/test_pbs_infra_routing.py
+git commit -m "feat(SKY-13291): vendors serve live view via CDP screencast transport"
+```
+
+---
+
+## End-to-end verification (before the Task 8 PR merges)
+
+1. Backend: `uv run pytest tests/unit/forge/sdk/routes/streaming/ tests/unit_tests/test_streaming_screencast.py tests/unit/webeye/test_browser_session_response.py tests/cloud/ -q` and `uv run alembic check` (no schema change expected).
+2. Local smoke (OSS mode unchanged): `BROWSER_STREAMING_MODE=cdp ./run_skyvern.sh`, open a browser session page, confirm the screencast still streams (base hook returns the global setting — behavior identical to today).
+3. Staging smoke (the new path): create a vendor-routed session via the API (`DYNAMIC_BROWSER_TYPE` flag arm or the infra flag), GET `/browser_sessions/{id}` → `stream_transport == "cdp"` and `vnc_streaming_supported == true` (after Task 8), open the session page → `InteractiveStreamView` renders frames. Capture before/after screenshots for the FE PR.
+4. Watch router load during the staging soak: screencast frames traverse the cdp-proxy (`skyvern.cdp_proxy.*` metrics); SKY-12502 (frame throttle/downscale via the policy engine) is the companion if per-viewer byte rates need bounding.
+
+## Known risks
+
+- `useStreamTransport` adds one `/browser_sessions/{id}` fetch on run pages that didn't previously load the session (deduped elsewhere via the shared queryKey). Acceptable; revisit only if endpoint latency shows up.
+- The vendor's own operator UI could theoretically hold a screencast on the same page target; the spike saw no interference, but if frames stall while automation runs, check for a competing `Page.startScreencast` client before blaming the router.
+- One provider's `cdp_url` is `http(s)` requiring `/json/version` discovery — irrelevant here (Playwright's `connect_over_cdp` handles it, and the API server dials the routed `browser_address`), but any future raw-websocket consumer must replicate it.
diff --git a/skyvern-frontend/src/hooks/useRuntimeConfig.test.ts b/skyvern-frontend/src/hooks/useRuntimeConfig.test.ts
index 03db7e5d5..d89e52f1c 100644
--- a/skyvern-frontend/src/hooks/useRuntimeConfig.test.ts
+++ b/skyvern-frontend/src/hooks/useRuntimeConfig.test.ts
@@ -30,3 +30,27 @@ describe("runtime config helpers", () => {
expect(normalizeBrowserStreamingMode(undefined)).toBe("vnc");
});
});
+
+describe("resolveStreamTransport", () => {
+ afterEach(() => {
+ vi.unstubAllEnvs();
+ vi.resetModules();
+ });
+
+ it("prefers the session's transport over the global mode", async () => {
+ const { resolveStreamTransport } = await loadRuntimeConfigHelpers();
+
+ expect(resolveStreamTransport("vnc", "cdp")).toBe("cdp");
+ expect(resolveStreamTransport("cdp", "vnc")).toBe("vnc");
+ });
+
+ it("falls back to the global mode when the session gives none or garbage", async () => {
+ const { resolveStreamTransport } = await loadRuntimeConfigHelpers();
+
+ expect(resolveStreamTransport("vnc", null)).toBe("vnc");
+ expect(resolveStreamTransport("vnc", undefined)).toBe("vnc");
+ expect(resolveStreamTransport("cdp", "")).toBe("cdp");
+ expect(resolveStreamTransport("vnc", "webrtc")).toBe("vnc");
+ expect(resolveStreamTransport("cdp", "webrtc")).toBe("cdp");
+ });
+});
diff --git a/skyvern-frontend/src/hooks/useRuntimeConfig.ts b/skyvern-frontend/src/hooks/useRuntimeConfig.ts
index 8717395b0..018ddee25 100644
--- a/skyvern-frontend/src/hooks/useRuntimeConfig.ts
+++ b/skyvern-frontend/src/hooks/useRuntimeConfig.ts
@@ -1,4 +1,5 @@
import { getClient } from "@/api/AxiosClient";
+import { useCredentialGetter } from "@/hooks/useCredentialGetter";
import { browserStreamingMode as buildTimeBrowserStreamingMode } from "@/util/env";
import { useQuery } from "@tanstack/react-query";
@@ -56,9 +57,56 @@ function useBrowserStreamingMode() {
};
}
+function resolveStreamTransport(
+ globalMode: BrowserStreamingMode,
+ sessionTransport: string | null | undefined,
+): BrowserStreamingMode {
+ const normalized = (sessionTransport ?? "").trim().toLowerCase();
+ if (!STREAMING_MODES.has(normalized)) {
+ return globalMode;
+ }
+ return normalized as BrowserStreamingMode;
+}
+
+function useStreamTransport(browserSessionId?: string | null) {
+ const { browserStreamingMode } = useBrowserStreamingMode();
+ const credentialGetter = useCredentialGetter();
+ const query = useQuery<{ stream_transport?: string | null }>({
+ // Deliberately the same key BrowserSession.tsx uses for its session query,
+ // so react-query dedupes the fetch on pages that already load the session.
+ queryKey: ["browserSession", browserSessionId],
+ queryFn: async () => {
+ const client = await getClient(credentialGetter, "sans-api-v1");
+ return client
+ .get(`/browser_sessions/${browserSessionId}`)
+ .then((response) => response.data);
+ },
+ enabled: Boolean(browserSessionId),
+ staleTime: 5 * 60 * 1000,
+ refetchOnWindowFocus: false,
+ retry: 1,
+ });
+
+ // Undefined until the session answers. Reporting the deployment default in the meantime is a
+ // different claim from "this session streams that way", and a consumer acting on it opens a
+ // stream the session may not serve, then swaps once the real answer lands.
+ const pending = Boolean(browserSessionId) && query.isPending;
+
+ return {
+ streamTransport: pending
+ ? undefined
+ : resolveStreamTransport(
+ browserStreamingMode,
+ query.data?.stream_transport,
+ ),
+ };
+}
+
export {
browserStreamingLabel,
normalizeBrowserStreamingMode,
+ resolveStreamTransport,
useBrowserStreamingMode,
useRuntimeConfig,
+ useStreamTransport,
};
diff --git a/skyvern-frontend/src/routes/browserSessions/BrowserSession.tsx b/skyvern-frontend/src/routes/browserSessions/BrowserSession.tsx
index b56940dcf..1b5de540a 100644
--- a/skyvern-frontend/src/routes/browserSessions/BrowserSession.tsx
+++ b/skyvern-frontend/src/routes/browserSessions/BrowserSession.tsx
@@ -26,7 +26,10 @@ import { SaveSessionAsBrowserProfileDialog } from "@/routes/browserProfiles/Save
import { useBackgroundBrowserProfileCreate } from "@/routes/browserProfiles/hooks/useBackgroundBrowserProfileCreate";
import { CopyText } from "@/routes/workflows/editor/Workspace";
import { type BrowserSession as BrowserSessionType } from "@/routes/workflows/types/browserSessionTypes";
-import { useBrowserStreamingMode } from "@/hooks/useRuntimeConfig";
+import {
+ resolveStreamTransport,
+ useBrowserStreamingMode,
+} from "@/hooks/useRuntimeConfig";
import {
StreamModeBadge,
type StreamMode,
@@ -49,7 +52,6 @@ function BrowserSession() {
const [isSaveProfileDialogOpen, setIsSaveProfileDialogOpen] = useState(false);
const [vncFailed, setVncFailed] = useState(false);
const { browserStreamingMode } = useBrowserStreamingMode();
- const isCdpMode = browserStreamingMode === "cdp";
useEffect(() => {
setVncFailed(false);
@@ -72,6 +74,11 @@ function BrowserSession() {
});
const browserSession = query.data;
+ const isCdpMode =
+ resolveStreamTransport(
+ browserStreamingMode,
+ browserSession?.stream_transport,
+ ) === "cdp";
const streamMode: StreamMode = isCdpMode
? "cdp"
: browserSession?.vnc_streaming_supported
diff --git a/skyvern-frontend/src/routes/tasks/detail/TaskActions.tsx b/skyvern-frontend/src/routes/tasks/detail/TaskActions.tsx
index c9de36933..3aafa2f85 100644
--- a/skyvern-frontend/src/routes/tasks/detail/TaskActions.tsx
+++ b/skyvern-frontend/src/routes/tasks/detail/TaskActions.tsx
@@ -7,7 +7,7 @@ import { toast } from "@/components/ui/use-toast";
import { ZoomableImage } from "@/components/ZoomableImage";
import { useCostCalculator } from "@/hooks/useCostCalculator";
import { useCredentialGetter } from "@/hooks/useCredentialGetter";
-import { useBrowserStreamingMode } from "@/hooks/useRuntimeConfig";
+import { useStreamTransport } from "@/hooks/useRuntimeConfig";
import { getCredentialParam } from "@/util/env";
import {
StreamStatusPanel,
@@ -127,8 +127,8 @@ function TaskActions() {
const taskIsNotFinalized = task && statusIsNotFinalized(task);
const taskIsRunningOrQueued = task && statusIsRunningOrQueued(task);
const browserSessionId = task?.browser_session_id;
- const { browserStreamingMode } = useBrowserStreamingMode();
- const shouldUseCdpStream = browserStreamingMode === "cdp";
+ const { streamTransport } = useStreamTransport(browserSessionId);
+ const shouldUseCdpStream = streamTransport === "cdp";
useEffect(() => {
// In VNC mode, BrowserStream handles live sessions. In CDP mode, this
diff --git a/skyvern-frontend/src/routes/workflows/ImportWorkflowButton.tsx b/skyvern-frontend/src/routes/workflows/ImportWorkflowButton.tsx
index 717283622..5faf35944 100644
--- a/skyvern-frontend/src/routes/workflows/ImportWorkflowButton.tsx
+++ b/skyvern-frontend/src/routes/workflows/ImportWorkflowButton.tsx
@@ -13,8 +13,10 @@ import { UploadIcon } from "@radix-ui/react-icons";
import { useQueryClient } from "@tanstack/react-query";
import { useId, useRef, useState } from "react";
import {
+ ArchiveEntry,
expandFileToWorkflowYamls,
extractTitleFromYaml,
+ unzipArchive,
} from "./importWorkflowYaml";
import { WorkflowApiResponse } from "./types/workflowTypes";
import { useCredentialGetter } from "@/hooks/useCredentialGetter";
@@ -184,11 +186,42 @@ function ImportWorkflowButton({
}
};
+ const prepareDocument = async (
+ fileName: string,
+ id: string,
+ yaml: string,
+ ): Promise<PreparedYamlImport> => {
+ const title = extractTitleFromYaml(yaml);
+ let duplicateReason: DuplicateReason | null = null;
+ if (title) {
+ try {
+ const existing = await findExistingWorkflowTitle(title);
+ if (existing) {
+ duplicateReason = { kind: "existing", existingTitle: existing };
+ }
+ } catch {
+ duplicateReason = { kind: "check-failed" };
+ }
+ }
+ return { id, fileName, yaml, title, duplicateReason };
+ };
+
+ const expandToDocuments = (text: string): string[] => {
+ try {
+ return expandFileToWorkflowYamls(text);
+ } catch {
+ return [text];
+ }
+ };
+
const handleFiles = async (fileList: FileList) => {
const files = Array.from(fileList);
const pdfFiles = files.filter((f) => f.name.toLowerCase().endsWith(".pdf"));
+ const zipFiles = files.filter((f) => f.name.toLowerCase().endsWith(".zip"));
const yamlLikeFiles = files.filter(
- (f) => !f.name.toLowerCase().endsWith(".pdf"),
+ (f) =>
+ !f.name.toLowerCase().endsWith(".pdf") &&
+ !f.name.toLowerCase().endsWith(".zip"),
);
let anyPdfStarted = false;
@@ -208,42 +241,46 @@ function ImportWorkflowButton({
const preparedPerFile = await Promise.all(
yamlLikeFiles.map(async (file) => {
- const text = await file.text();
- let documents: string[];
+ const documents = expandToDocuments(await file.text());
+ return Promise.all(
+ documents.map((yaml, index) =>
+ prepareDocument(file.name, `${file.name}#${index}`, yaml),
+ ),
+ );
+ }),
+ );
+
+ const preparedPerZip = await Promise.all(
+ zipFiles.map(async (file) => {
+ let entries: ArchiveEntry[];
try {
- documents = expandFileToWorkflowYamls(text);
- } catch {
- documents = [text];
+ entries = unzipArchive(new Uint8Array(await file.arrayBuffer()));
+ } catch (error) {
+ toast({
+ variant: "destructive",
+ title: `Error reading ${file.name}`,
+ description: getErrorMessage(error, "Failed to read zip archive"),
+ });
+ return [];
}
- return Promise.all(
- documents.map(async (yaml, index) => {
- const title = extractTitleFromYaml(yaml);
- let duplicateReason: DuplicateReason | null = null;
- if (title) {
- try {
- const existing = await findExistingWorkflowTitle(title);
- if (existing) {
- duplicateReason = {
- kind: "existing",
- existingTitle: existing,
- };
- }
- } catch {
- duplicateReason = { kind: "check-failed" };
- }
- }
- return {
- id: `${file.name}#${index}`,
- fileName: file.name,
- yaml,
- title,
- duplicateReason,
- };
+ const preparedPerEntry = await Promise.all(
+ entries.map(async ({ name, text }) => {
+ const documents = expandToDocuments(text);
+ return Promise.all(
+ documents.map((yaml, index) =>
+ prepareDocument(name, `${file.name}#${name}#${index}`, yaml),
+ ),
+ );
}),
);
+ return preparedPerEntry.flat();
}),
);
- const prepared: PreparedYamlImport[] = preparedPerFile.flat();
+
+ const prepared: PreparedYamlImport[] = [
+ ...preparedPerFile.flat(),
+ ...preparedPerZip.flat(),
+ ];
const titleCounts = new Map<string, number>();
for (const p of prepared) {
@@ -331,7 +368,7 @@ function ImportWorkflowButton({
ref={inputRef}
id={inputId}
type="file"
- accept=".yaml,.yml,.json,.pdf"
+ accept=".yaml,.yml,.json,.zip,.pdf"
multiple
className="hidden"
disabled={isBusy}
@@ -364,7 +401,7 @@ function ImportWorkflowButton({
</Label>
</TooltipTrigger>
<TooltipContent>
- Import one or more agents from YAML, JSON, or PDF files
+ Import one or more agents from YAML, JSON, ZIP, or PDF files
</TooltipContent>
</Tooltip>
</TooltipProvider>
diff --git a/skyvern-frontend/src/routes/workflows/components/BulkActionBar.tsx b/skyvern-frontend/src/routes/workflows/components/BulkActionBar.tsx
index de6591813..4206d13c7 100644
--- a/skyvern-frontend/src/routes/workflows/components/BulkActionBar.tsx
+++ b/skyvern-frontend/src/routes/workflows/components/BulkActionBar.tsx
@@ -30,6 +30,7 @@ import {
TokensIcon,
} from "@radix-ui/react-icons";
import { useQueryClient } from "@tanstack/react-query";
+import { strToU8, zipSync } from "fflate";
import { stringify as convertToYAML } from "yaml";
import { convert } from "../editor/workflowEditorUtils";
import { Tag, TagKey } from "../types/tagTypes";
@@ -52,8 +53,12 @@ type Props = {
};
// Blob URLs avoid the ~2MB data-URI cap that truncates large workflow exports.
-function downloadFile(fileName: string, contents: string) {
- const blob = new Blob([contents], { type: "text/plain;charset=utf-8" });
+function downloadFile(
+ fileName: string,
+ contents: BlobPart,
+ mimeType = "text/plain;charset=utf-8",
+) {
+ const blob = new Blob([contents], { type: mimeType });
const url = URL.createObjectURL(blob);
const element = document.createElement("a");
element.href = url;
@@ -66,6 +71,22 @@ function downloadFile(fileName: string, contents: string) {
setTimeout(() => URL.revokeObjectURL(url), 60_000);
}
+function sanitizeFileName(name: string): string {
+ return name.replace(/[/\\:*?"<>|]/g, "_");
+}
+
+// Two selected agents can share a title; number repeats so every archive
+// entry name stays unique.
+function uniqueFileName(used: Set<string>, title: string, ext: string): string {
+ const base = sanitizeFileName(title);
+ let candidate = `${base}.${ext}`;
+ for (let suffix = 2; used.has(candidate); suffix++) {
+ candidate = `${base}-${suffix}.${ext}`;
+ }
+ used.add(candidate);
+ return candidate;
+}
+
function BulkActionBar({
selectedWorkflows,
isOperating,
@@ -225,7 +246,7 @@ function BulkActionBar({
function handleBulkExport(type: "json" | "yaml") {
if (selectedWorkflows.length === 1) {
const workflow = selectedWorkflows[0]!;
- const safeTitle = workflow.title.replace(/[/\\:*?"<>|]/g, "_");
+ const safeTitle = sanitizeFileName(workflow.title);
const contents =
type === "json"
? JSON.stringify(convert(workflow), null, 2)
@@ -233,16 +254,24 @@ function BulkActionBar({
downloadFile(`${safeTitle}.${type}`, contents);
return;
}
- // One bundled file; per-agent downloads trip the browser's
- // multiple-downloads permission prompt.
- const converted = selectedWorkflows.map((workflow) => convert(workflow));
- const contents =
- type === "json"
- ? JSON.stringify(converted, null, 2)
- : converted
- .map((definition) => convertToYAML(definition))
- .join("---\n");
- downloadFile(`agents-export-${selectedWorkflows.length}.${type}`, contents);
+ // A ZIP with one file per agent triggers a single download. N sequential
+ // downloadFile() calls would trip the browser's multiple-downloads
+ // permission prompt instead.
+ const usedNames = new Set<string>();
+ const entries: Record<string, Uint8Array> = {};
+ for (const workflow of selectedWorkflows) {
+ const contents =
+ type === "json"
+ ? JSON.stringify(convert(workflow), null, 2)
+ : convertToYAML(convert(workflow));
+ entries[uniqueFileName(usedNames, workflow.title, type)] =
+ strToU8(contents);
+ }
+ downloadFile(
+ `agents-export-${selectedWorkflows.length}.zip`,
+ new Uint8Array(zipSync(entries)),
+ "application/zip",
+ );
}
const agentNoun = count === 1 ? "agent" : "agents";
diff --git a/skyvern-frontend/src/routes/workflows/editor/Workspace.tsx b/skyvern-frontend/src/routes/workflows/editor/Workspace.tsx
index 666a595ed..935a41574 100644
--- a/skyvern-frontend/src/routes/workflows/editor/Workspace.tsx
+++ b/skyvern-frontend/src/routes/workflows/editor/Workspace.tsx
@@ -50,7 +50,7 @@ import { useIsGlobalWorkflow } from "../hooks/useIsGlobalWorkflow";
import { resolveWorkspaceBrowserSessionBindings } from "./browserSessionBindings";
import { useBlockScriptsQuery } from "@/routes/workflows/hooks/useBlockScriptsQuery";
import { BrowserSessionStream } from "@/routes/browserSessions/BrowserSessionStream";
-import { useBrowserStreamingMode } from "@/hooks/useRuntimeConfig";
+import { useStreamTransport } from "@/hooks/useRuntimeConfig";
import {
StreamModeBadge,
StreamStatusPanel,
@@ -201,6 +201,7 @@ import { useWorkflowYamlEditorLifecycle } from "./hooks/useWorkflowYamlEditorLif
import {
preservedFinallyBlockLabel,
workflowVersionFromSaveData,
+ yamlCommitInputs,
} from "./workflowVersionFromSaveData";
import "./workspace-styles.css";
@@ -529,7 +530,6 @@ function Workspace({
const { data: workflowRun } = useWorkflowRunQuery();
const studioRunId = useStudioRunId();
const isFinalized = workflowRun ? statusIsFinalized(workflowRun) : false;
- const { browserStreamingMode } = useBrowserStreamingMode();
const [openCycleBrowserDialogue, setOpenCycleBrowserDialogue] =
useState(false);
@@ -578,11 +578,6 @@ function Workspace({
const [isCopilotTurnActive, setIsCopilotTurnActive] = useState(false);
const blockScriptStore = useBlockScriptStore();
const recordingStore = useRecordingStore();
- const isCdpStreamingMode =
- browserStreamingMode === "cdp" && !recordingStore.isRecording;
- // Record Browser exfiltration requires VNC even when the org default is CDP streaming.
- const preferVncStream =
- browserStreamingMode !== "cdp" || recordingStore.isRecording;
const cacheKey = workflow?.cache_key ?? "";
// Block delete confirmation dialog state
@@ -775,6 +770,15 @@ function Workspace({
const activeDebugSession = debugSession ?? null;
+ const { streamTransport } = useStreamTransport(
+ activeDebugSession?.browser_session_id,
+ );
+ const isCdpStreamingMode =
+ streamTransport === "cdp" && !recordingStore.isRecording;
+ // Record Browser exfiltration requires VNC even when the transport is CDP streaming.
+ const preferVncStream =
+ streamTransport !== "cdp" || recordingStore.isRecording;
+
const workflowChangesStore = useWorkflowHasChangesStore();
const showBreakoutButton =
@@ -1776,12 +1780,16 @@ function Workspace({
);
return false;
}
+ const { definition: draftDefinition, definitionYaml } = yamlCommitInputs(
+ parsed,
+ yamlStore.draft,
+ );
try {
const client = await getClient(credentialGetter, "sans-api-v1");
const response = await client.post<WorkflowYAMLConversionResponse>(
"/workflow/copilot/convert-yaml-to-blocks",
{
- workflow_definition_yaml: yamlStore.draft,
+ workflow_definition_yaml: definitionYaml,
workflow_id: saveData.workflow.workflow_id,
},
);
@@ -1858,7 +1866,7 @@ function Workspace({
try {
await saveWorkflow.mutateAsync({
blocks: upgradedBlocks,
- parameters: parsed?.parameters ?? [],
+ parameters: draftDefinition?.parameters ?? [],
workflowDefinitionVersion: upgradedVersion,
settings: { ...saveData.settings, finallyBlockLabel },
});
@@ -2544,7 +2552,7 @@ function Workspace({
</div>
)}
- {/* Live browser: mode comes from BROWSER_STREAMING_MODE / runtime config */}
+ {/* Live browser: mode comes from the session's stream transport, falling back to runtime config */}
{showVncBrowserPanel && (
<div className="skyvern-vnc-browser flex h-full w-[calc(100%_-_6rem)] flex-1 flex-col items-center justify-center">
<div key={reloadKey} className="w-full flex-1">
diff --git a/skyvern-frontend/src/routes/workflows/editor/workflowVersionFromSaveData.test.ts b/skyvern-frontend/src/routes/workflows/editor/workflowVersionFromSaveData.test.ts
index db0cca628..b1bf86f34 100644
--- a/skyvern-frontend/src/routes/workflows/editor/workflowVersionFromSaveData.test.ts
+++ b/skyvern-frontend/src/routes/workflows/editor/workflowVersionFromSaveData.test.ts
@@ -1,4 +1,5 @@
import { describe, expect, test } from "vitest";
+import { parse } from "yaml";
import { ProxyLocation } from "@/api/types";
import type { WorkflowSaveData } from "@/store/WorkflowHasChangesStore";
@@ -10,6 +11,7 @@ import type {
import {
preservedFinallyBlockLabel,
workflowVersionFromSaveData,
+ yamlCommitInputs,
} from "./workflowVersionFromSaveData";
const definition: WorkflowDefinition = {
@@ -138,3 +140,90 @@ describe("preservedFinallyBlockLabel", () => {
expect(preservedFinallyBlockLabel("", ["step_1"])).toBe(null);
});
});
+
+describe("yamlCommitInputs", () => {
+ test("unwraps a full export and preserves its parameters", () => {
+ const nested = {
+ version: 2,
+ parameters: [
+ {
+ key: "target_url",
+ parameter_type: "workflow",
+ workflow_parameter_type: "string",
+ },
+ ],
+ blocks: [
+ {
+ label: "open_target",
+ block_type: "goto_url",
+ url: "https://example.test/",
+ },
+ ],
+ };
+ const fullExport = {
+ title: "wf",
+ proxy_location: "RESIDENTIAL",
+ workflow_definition: nested,
+ run_sequentially: false,
+ };
+ const draft = "title: wf\nworkflow_definition:\n parameters: []\n";
+
+ const { definition, definitionYaml } = yamlCommitInputs(
+ fullExport as unknown as typeof nested,
+ draft,
+ );
+ expect(definition).toBe(nested);
+ expect(definition.parameters).toHaveLength(1);
+ expect(definition.parameters[0]?.key).toBe("target_url");
+ expect(definitionYaml).not.toBe(draft);
+ const roundTrip = parse(definitionYaml);
+ expect(roundTrip.blocks).toEqual(nested.blocks);
+ expect(roundTrip.parameters).toEqual(nested.parameters);
+ expect(roundTrip.title).toBeUndefined();
+ });
+
+ test("returns a definition-only draft byte-exact", () => {
+ const draft =
+ "parameters: []\nblocks:\n # user's comment\n - label: a\n block_type: goto_url\n";
+ const parsed = parse(draft);
+
+ const { definition, definitionYaml } = yamlCommitInputs(parsed, draft);
+ expect(definition).toBe(parsed);
+ expect(definitionYaml).toBe(draft);
+ });
+
+ test("keeps raw text for a NaN draft", () => {
+ expect(yamlCommitInputs(NaN, ".nan # keep").definitionYaml).toBe(
+ ".nan # keep",
+ );
+ });
+
+ test("prefers top-level blocks when both shapes are present", () => {
+ const input = {
+ blocks: [{ label: "a", block_type: "goto_url" }],
+ workflow_definition: { parameters: [], blocks: [] },
+ };
+ const draft = "blocks: []\nworkflow_definition:\n blocks: []\n";
+
+ const { definition, definitionYaml } = yamlCommitInputs(input, draft);
+ expect(definition).toBe(input);
+ expect(definitionYaml).toBe(draft);
+ });
+
+ test("leaves malformed and non-object shapes alone", () => {
+ const draft = "workflow_definition: invalid\n";
+ const inputs = [
+ { workflow_definition: "invalid" },
+ { workflow_definition: [] },
+ [],
+ null,
+ undefined,
+ ];
+
+ for (const input of inputs) {
+ const { definition, definitionYaml } = yamlCommitInputs(input, draft);
+ expect(definition).toBe(input);
+ expect(definitionYaml).toBe(draft);
+ }
+ });
+});
diff --git a/skyvern-frontend/src/routes/workflows/editor/workflowVersionFromSaveData.ts b/skyvern-frontend/src/routes/workflows/editor/workflowVersionFromSaveData.ts
index 6597a537c..1b1b9eac8 100644
--- a/skyvern-frontend/src/routes/workflows/editor/workflowVersionFromSaveData.ts
+++ b/skyvern-frontend/src/routes/workflows/editor/workflowVersionFromSaveData.ts
@@ -1,3 +1,5 @@
+import { stringify as convertToYAML } from "yaml";
+
import type { WorkflowSaveData } from "@/store/WorkflowHasChangesStore";
import type { WorkflowVersion } from "../hooks/useWorkflowVersionsQuery";
@@ -74,3 +76,36 @@ export function workflowVersionFromSaveData(
import_error: workflow.import_error ?? null,
};
}
+
+// A pasted "Export as YAML" document (WorkflowCreateYAMLRequest) nests the
+// definition under workflow_definition. The pane edits only the definition, so
+// unwrap it; top-level export settings are ignored, matching enterYamlMode.
+function unwrapWorkflowDefinition<T>(parsed: T): T {
+ const doc = parsed as
+ | { blocks?: unknown; workflow_definition?: unknown }
+ | null
+ | undefined;
+ const nested = doc?.workflow_definition;
+ return doc?.blocks == null &&
+ typeof nested === "object" &&
+ nested !== null &&
+ !Array.isArray(nested)
+ ? (nested as T)
+ : parsed;
+}
+
+// The pane edits the workflow definition, but a user can paste a full "Export
+// as YAML" document; a definition-only draft is POSTed as the user's raw text
+// so their formatting and comments survive the round trip.
+export function yamlCommitInputs<T>(
+ parsed: T,
+ draft: string,
+): { definition: T; definitionYaml: string } {
+ const definition = unwrapWorkflowDefinition(parsed);
+ return {
+ definition,
+ definitionYaml: Object.is(definition, parsed)
+ ? draft
+ : convertToYAML(definition),
+ };
+}
diff --git a/skyvern-frontend/src/routes/workflows/importWorkflowYaml.test.ts b/skyvern-frontend/src/routes/workflows/importWorkflowYaml.test.ts
index 0f15f0a8e..36a56e98f 100644
--- a/skyvern-frontend/src/routes/workflows/importWorkflowYaml.test.ts
+++ b/skyvern-frontend/src/routes/workflows/importWorkflowYaml.test.ts
@@ -1,9 +1,11 @@
+import { strToU8, zipSync } from "fflate";
import { describe, expect, it } from "vitest";
import { parse as parseYAML, stringify as convertToYAML } from "yaml";
import {
expandFileToWorkflowYamls,
extractTitleFromYaml,
+ unzipArchive,
} from "./importWorkflowYaml";
const workflowA = { title: "Workflow A", workflow_definition: { blocks: [] } };
@@ -13,6 +15,12 @@ function titlesOf(yamls: string[]): Array<string | null> {
return yamls.map((yaml) => extractTitleFromYaml(yaml));
}
+// fflate is loaded in Node's realm while Vitest runs tests in jsdom's realm.
+// Copy encoded bytes so zipSync recognizes them as Uint8Array file contents.
+function archiveBytes(text: string): Uint8Array {
+ return new Uint8Array(strToU8(text));
+}
+
describe("expandFileToWorkflowYamls", () => {
it("splits the bulk YAML export format into one workflow per document", () => {
// Matches BulkActionBar.handleBulkExport: docs joined by "---\n".
@@ -82,6 +90,77 @@ describe("expandFileToWorkflowYamls", () => {
});
});
+describe("unzipArchive", () => {
+ it("extracts one text entry per file in the archive", () => {
+ // Mirrors BulkActionBar.handleBulkExport's ZIP branch: one sanitized,
+ // deduped entry per agent.
+ const zipped = zipSync({
+ "Workflow A.yaml": archiveBytes(convertToYAML(workflowA)),
+ "Workflow B.yaml": archiveBytes(convertToYAML(workflowB)),
+ });
+
+ const entries = unzipArchive(zipped);
+ const byName = new Map(entries.map((entry) => [entry.name, entry.text]));
+
+ expect(entries).toHaveLength(2);
+ expect(parseYAML(byName.get("Workflow A.yaml")!)).toEqual(workflowA);
+ expect(parseYAML(byName.get("Workflow B.yaml")!)).toEqual(workflowB);
+ });
+
+ it("round-trips zipped per-agent files back into individual workflows", () => {
+ const zipped = zipSync({
+ "Workflow A.yaml": archiveBytes(convertToYAML(workflowA)),
+ "Workflow B.yaml": archiveBytes(convertToYAML(workflowB)),
+ });
+
+ const expanded = unzipArchive(zipped).flatMap((entry) =>
+ expandFileToWorkflowYamls(entry.text),
+ );
+
+ expect(titlesOf(expanded)).toEqual(["Workflow A", "Workflow B"]);
+ });
+
+ it("ignores empty entries", () => {
+ const zipped = zipSync({
+ "empty.yaml": new Uint8Array(0),
+ "Workflow A.yaml": archiveBytes(convertToYAML(workflowA)),
+ });
+
+ const entries = unzipArchive(zipped);
+
+ expect(entries.map((entry) => entry.name)).toEqual(["Workflow A.yaml"]);
+ });
+
+ it("ignores non-workflow files, including macOS zip metadata junk", () => {
+ const zipped = zipSync({
+ "Workflow A.yaml": archiveBytes(convertToYAML(workflowA)),
+ "__MACOSX/._Workflow A.yaml": archiveBytes("junk"),
+ ".DS_Store": archiveBytes("junk"),
+ "notes.txt": archiveBytes("not a workflow"),
+ });
+
+ const entries = unzipArchive(zipped);
+
+ expect(entries.map((entry) => entry.name)).toEqual(["Workflow A.yaml"]);
+ });
+
+ it("rejects an archive over the size limit before attempting to unzip it", () => {
+ const oversized = new Uint8Array(21 * 1024 * 1024);
+
+ expect(() => unzipArchive(oversized)).toThrow(/too large/i);
+ });
+
+ it("rejects an archive with too many entries", () => {
+ const files: Record<string, Uint8Array> = {};
+ for (let i = 0; i < 201; i++) {
+ files[`workflow-${i}.yaml`] = archiveBytes(convertToYAML(workflowA));
+ }
+ const zipped = zipSync(files);
+
+ expect(() => unzipArchive(zipped)).toThrow(/too many files/i);
+ });
+});
+
describe("extractTitleFromYaml", () => {
it("reads and trims a top-level title", () => {
expect(extractTitleFromYaml("title: ' Padded '\nfoo: 1")).toBe("Padded");
diff --git a/skyvern-frontend/src/routes/workflows/importWorkflowYaml.ts b/skyvern-frontend/src/routes/workflows/importWorkflowYaml.ts
index 6330fcb3c..7cd3a4a09 100644
--- a/skyvern-frontend/src/routes/workflows/importWorkflowYaml.ts
+++ b/skyvern-frontend/src/routes/workflows/importWorkflowYaml.ts
@@ -1,3 +1,4 @@
+import { strFromU8, unzipSync } from "fflate";
import {
parse as parseYAML,
parseAllDocuments,
@@ -46,6 +47,45 @@ export function expandFileToWorkflowYamls(text: string): string[] {
return workflows.map((workflow) => convertToYAML(workflow));
}
+export type ArchiveEntry = { name: string; text: string };
+
+// unzipSync runs synchronously on the UI thread; cap size/entry count so an
+// oversized archive fails fast instead of hanging the tab.
+const MAX_ZIP_BYTES = 20 * 1024 * 1024;
+const MAX_ZIP_ENTRIES = 200;
+
+function isWorkflowArchiveEntry(name: string): boolean {
+ if (name.startsWith("__MACOSX/")) {
+ return false;
+ }
+ const basename = name.split("/").pop() ?? "";
+ if (basename.startsWith(".")) {
+ return false; // .DS_Store, AppleDouble "._*" resource forks
+ }
+ return /\.(ya?ml|json)$/i.test(name);
+}
+
+// Bulk export (N > 1) writes one ZIP with one file per agent. Unpack it back
+// into individual entries so each can go through the normal single-file
+// import path (which itself calls expandFileToWorkflowYamls per entry).
+export function unzipArchive(bytes: Uint8Array): ArchiveEntry[] {
+ if (bytes.length > MAX_ZIP_BYTES) {
+ throw new Error(
+ `Archive is too large to import (max ${MAX_ZIP_BYTES / (1024 * 1024)}MB).`,
+ );
+ }
+ const unzipped = unzipSync(bytes);
+ const entries = Object.entries(unzipped)
+ .filter(([name, data]) => isWorkflowArchiveEntry(name) && data.length > 0)
+ .map(([name, data]) => ({ name, text: strFromU8(data) }));
+ if (entries.length > MAX_ZIP_ENTRIES) {
+ throw new Error(
+ `Archive has too many files to import (max ${MAX_ZIP_ENTRIES}).`,
+ );
+ }
+ return entries;
+}
+
export function extractTitleFromYaml(yaml: string): string | null {
try {
const parsed = parseYAML(yaml);
diff --git a/skyvern-frontend/src/routes/workflows/studio/BrowserTab.test.tsx b/skyvern-frontend/src/routes/workflows/studio/BrowserTab.test.tsx
index ec0408e3d..8ea8f398c 100644
--- a/skyvern-frontend/src/routes/workflows/studio/BrowserTab.test.tsx
+++ b/skyvern-frontend/src/routes/workflows/studio/BrowserTab.test.tsx
@@ -51,6 +51,7 @@ vi.mock("../hooks/useWorkflowRunsQuery", () => ({
}));
vi.mock("@/hooks/useRuntimeConfig", () => ({
useBrowserStreamingMode: () => ({ browserStreamingMode: "vnc" }),
+ useStreamTransport: () => ({ streamTransport: "vnc" }),
}));
vi.mock("posthog-js/react", () => ({
usePostHog: () => ({ capture: vi.fn() }),
diff --git a/skyvern-frontend/src/routes/workflows/studio/StreamPresenter.test.tsx b/skyvern-frontend/src/routes/workflows/studio/StreamPresenter.test.tsx
index 5dfac9cca..a481a7ad6 100644
--- a/skyvern-frontend/src/routes/workflows/studio/StreamPresenter.test.tsx
+++ b/skyvern-frontend/src/routes/workflows/studio/StreamPresenter.test.tsx
@@ -7,6 +7,7 @@ import { StreamPresenter } from "./StreamPresenter";
const runtimeConfigMock = vi.hoisted(() => ({
browserStreamingMode: "cdp" as "cdp" | "vnc",
+ transportPending: false,
}));
const browserStreamProps = vi.hoisted(
@@ -18,6 +19,11 @@ vi.mock("@/hooks/useRuntimeConfig", () => ({
useBrowserStreamingMode: () => ({
browserStreamingMode: runtimeConfigMock.browserStreamingMode,
}),
+ useStreamTransport: () => ({
+ streamTransport: runtimeConfigMock.transportPending
+ ? undefined
+ : runtimeConfigMock.browserStreamingMode,
+ }),
}));
vi.mock("@/components/BrowserStream", () => ({
@@ -36,6 +42,17 @@ describe("StreamPresenter transport-swap recording", () => {
cleanup();
browserStreamProps.last = null;
runtimeConfigMock.browserStreamingMode = "cdp";
+ runtimeConfigMock.transportPending = false;
+ });
+
+ it("shows neither stream until the session's transport is known", () => {
+ runtimeConfigMock.transportPending = true;
+ // Global mode is cdp, but an unanswered session must not be painted as either: a session
+ // that turns out to stream the other way would have shown a stream that cannot connect.
+ render(<StreamPresenter browserSessionId="pbs_test" />);
+
+ expect(screen.queryByTestId("cdp-stream")).toBeNull();
+ expect(screen.queryByTestId("vnc-stream")).toBeNull();
});
it("shows the CDP stream when not recording in cdp mode", () => {
diff --git a/skyvern-frontend/src/routes/workflows/studio/StreamPresenter.tsx b/skyvern-frontend/src/routes/workflows/studio/StreamPresenter.tsx
index 318453927..0e3648156 100644
--- a/skyvern-frontend/src/routes/workflows/studio/StreamPresenter.tsx
+++ b/skyvern-frontend/src/routes/workflows/studio/StreamPresenter.tsx
@@ -1,5 +1,5 @@
import { BrowserStream } from "@/components/BrowserStream";
-import { useBrowserStreamingMode } from "@/hooks/useRuntimeConfig";
+import { useStreamTransport } from "@/hooks/useRuntimeConfig";
import { BrowserSessionStream } from "@/routes/browserSessions/BrowserSessionStream";
type StreamPresenterProps = {
@@ -27,8 +27,13 @@ export function StreamPresenter({
onUrlChange,
onActivity,
}: StreamPresenterProps) {
- const { browserStreamingMode } = useBrowserStreamingMode();
- const useCdp = browserStreamingMode === "cdp" && !isRecording;
+ const { streamTransport } = useStreamTransport(browserSessionId);
+ const useCdp = streamTransport === "cdp" && !isRecording;
+
+ if (!streamTransport) {
+ // Mounting either transport now would open a connection this session may not serve.
+ return null;
+ }
if (useCdp) {
// CDP frames must be explicitly centered; VNC handles this in its own CSS.
diff --git a/skyvern-frontend/src/routes/workflows/studio/StudioBrowserStream.test.tsx b/skyvern-frontend/src/routes/workflows/studio/StudioBrowserStream.test.tsx
index ae6f4f9ea..b41079476 100644
--- a/skyvern-frontend/src/routes/workflows/studio/StudioBrowserStream.test.tsx
+++ b/skyvern-frontend/src/routes/workflows/studio/StudioBrowserStream.test.tsx
@@ -40,6 +40,9 @@ vi.mock("@/hooks/useRuntimeConfig", () => ({
useBrowserStreamingMode: () => ({
browserStreamingMode: runtimeConfigMock.browserStreamingMode,
}),
+ useStreamTransport: () => ({
+ streamTransport: runtimeConfigMock.browserStreamingMode,
+ }),
}));
vi.mock("@/components/BrowserStream", () => ({
diff --git a/skyvern-frontend/src/routes/workflows/studio/runview/RunLiveStream.tsx b/skyvern-frontend/src/routes/workflows/studio/runview/RunLiveStream.tsx
index 8842f95dc..b8890fcba 100644
--- a/skyvern-frontend/src/routes/workflows/studio/runview/RunLiveStream.tsx
+++ b/skyvern-frontend/src/routes/workflows/studio/runview/RunLiveStream.tsx
@@ -1,7 +1,7 @@
import { useEffect, useState } from "react";
import { BrowserStream } from "@/components/BrowserStream";
-import { useBrowserStreamingMode } from "@/hooks/useRuntimeConfig";
+import { useStreamTransport } from "@/hooks/useRuntimeConfig";
import { WorkflowRunStream } from "../../workflowRun/WorkflowRunStream";
@@ -23,7 +23,7 @@ export function RunLiveStream({
interactive,
onUrlChange,
}: RunLiveStreamProps) {
- const { browserStreamingMode } = useBrowserStreamingMode();
+ const { streamTransport } = useStreamTransport(browserSessionId);
const [vncFailed, setVncFailed] = useState(false);
useEffect(() => {
@@ -31,7 +31,13 @@ export function RunLiveStream({
}, [browserSessionId]);
const useVnc =
- Boolean(browserSessionId) && browserStreamingMode !== "cdp" && !vncFailed;
+ Boolean(browserSessionId) && streamTransport !== "cdp" && !vncFailed;
+
+ if (browserSessionId && !streamTransport) {
+ // Defaulting to VNC here opens a socket an externally hosted session cannot serve, and the
+ // failure is what swaps the transport — a visible drop rather than a first paint.
+ return null;
+ }
if (useVnc && browserSessionId) {
return (
diff --git a/skyvern-frontend/src/routes/workflows/types/browserSessionTypes.ts b/skyvern-frontend/src/routes/workflows/types/browserSessionTypes.ts
index a8b675871..3cadfb6a2 100644
--- a/skyvern-frontend/src/routes/workflows/types/browserSessionTypes.ts
+++ b/skyvern-frontend/src/routes/workflows/types/browserSessionTypes.ts
@@ -16,6 +16,7 @@ interface BrowserSession {
browser_type?: BrowserSessionType | null;
generate_browser_profile?: boolean;
vnc_streaming_supported: boolean;
+ stream_transport?: string | null;
}
interface Recording {
diff --git a/skyvern-frontend/src/routes/workflows/workflowRun/WorkflowRunOverview.tsx b/skyvern-frontend/src/routes/workflows/workflowRun/WorkflowRunOverview.tsx
index d1d90688d..16fb00292 100644
--- a/skyvern-frontend/src/routes/workflows/workflowRun/WorkflowRunOverview.tsx
+++ b/skyvern-frontend/src/routes/workflows/workflowRun/WorkflowRunOverview.tsx
@@ -25,7 +25,7 @@ import {
import { Skeleton } from "@/components/ui/skeleton";
import { useQueryClient } from "@tanstack/react-query";
import { useCallback, useEffect, useState } from "react";
-import { useBrowserStreamingMode } from "@/hooks/useRuntimeConfig";
+import { useStreamTransport } from "@/hooks/useRuntimeConfig";
import { RunHealChip } from "./RunHealChip";
import { RunReliabilityUplink } from "./RunReliabilityUplink";
@@ -48,7 +48,6 @@ function WorkflowRunOverview() {
const activeIteration = parseActiveIterationParam(iterationParam);
const queryClient = useQueryClient();
const [vncFailed, setVncFailed] = useState(false);
- const { browserStreamingMode } = useBrowserStreamingMode();
const { data: workflowRun, isLoading: workflowRunIsLoading } =
useWorkflowRunWithWorkflowQuery();
@@ -60,6 +59,7 @@ function WorkflowRunOverview() {
const workflowPermanentId = workflow?.workflow_permanent_id;
const browserSessionId = workflowRun?.browser_session_id;
+ const { streamTransport } = useStreamTransport(browserSessionId);
const invalidateQueries = useCallback(() => {
if (workflowRunId) {
@@ -120,7 +120,7 @@ function WorkflowRunOverview() {
selection.block_type === "human_interaction"))
);
- const shouldUseCdpStream = browserStreamingMode === "cdp";
+ const shouldUseCdpStream = streamTransport === "cdp";
const shouldShowBrowserStream =
wantsVncStream && !shouldUseCdpStream && !vncFailed;
const shouldShowScreencastFallback =
diff --git a/skyvern/browser_extension/runtime.py b/skyvern/browser_extension/runtime.py
index 5eb36afb6..775bdaedd 100644
--- a/skyvern/browser_extension/runtime.py
+++ b/skyvern/browser_extension/runtime.py
@@ -77,8 +77,9 @@ class BrowserExtensionRuntime:
await _cleanup_failed_start(relay, adapter)
if exc.errno == errno.EADDRINUSE:
raise BrowserExtensionError(
- f"Browser extension relay port {resolved_port} is already in use; "
- f"set {_PORT_ENV} to a free port"
+ f"Another Skyvern MCP session likely owns the browser-extension bridge on port {resolved_port}; "
+ f"close that session, or set {_PORT_ENV} to a free port and update the bridge port in the Skyvern "
+ "extension popup to match"
) from exc
raise
except BaseException:
diff --git a/skyvern/cli/run_commands.py b/skyvern/cli/run_commands.py
index a331b2bad..f40b124d3 100644
--- a/skyvern/cli/run_commands.py
+++ b/skyvern/cli/run_commands.py
@@ -234,9 +234,16 @@ async def _run_mcp_with_cleanup(run_async: Any, *, browser_extension: bool = Fal
extension_runtime: BrowserExtensionRuntime | None = None
try:
if browser_extension:
+ from skyvern.browser_extension.errors import BrowserExtensionError # noqa: PLC0415
from skyvern.browser_extension.runtime import BrowserExtensionRuntime # noqa: PLC0415
- extension_runtime = await BrowserExtensionRuntime.get_or_start()
+ try:
+ extension_runtime = await BrowserExtensionRuntime.get_or_start()
+ except BrowserExtensionError as exc:
+ LOG.warning(
+ "Browser extension bridge startup failed; the bridge will be retried on first browser-tool use",
+ error=str(exc),
+ )
await run_async(**kwargs)
finally:
_mcp_cleanup_in_progress = True
diff --git a/skyvern/exceptions.py b/skyvern/exceptions.py
index 0b03fea5d..2605c6ee1 100644
--- a/skyvern/exceptions.py
+++ b/skyvern/exceptions.py
@@ -362,6 +362,12 @@ class MissingBrowserStatePage(SkyvernException):
super().__init__(f"Browser state page is missing. {task_str} {workflow_run_str}")
+class BrowserProfileNotApplied(SkyvernException):
+ def __init__(self, browser_profile_id: str) -> None:
+ self.browser_profile_id = browser_profile_id
+ super().__init__(f"Browser profile {browser_profile_id} was not applied by the created browser")
+
+
class MissingWorkflowRunBrowserState(SkyvernException):
def __init__(self, workflow_run_id: str, task_id: str) -> None:
super().__init__(f"Browser state for workflow run {workflow_run_id} and task {task_id} is missing.")
diff --git a/skyvern/forge/agent_functions.py b/skyvern/forge/agent_functions.py
index 6152eff2a..30f129c19 100644
--- a/skyvern/forge/agent_functions.py
+++ b/skyvern/forge/agent_functions.py
@@ -921,6 +921,16 @@ class AgentFunction:
"""
return True
+ async def resolve_stream_transport(
+ self, *, browser_session_id: str | None, organization_id: str | None, ip_address: str | None = None
+ ) -> str:
+ """Which live-view transport serves this session: "vnc" or "cdp".
+
+ A self-hosted deployment streams every browser the same way, so the
+ deployment-wide setting decides.
+ """
+ return settings.BROWSER_STREAMING_MODE
+
async def select_browser_session_recordings(
self,
*,
@@ -1079,6 +1089,22 @@ class AgentFunction:
"""
return False
+ async def should_route_to_secure_runner_pool(
+ self,
+ *,
+ workflow_run_id: str,
+ organization_id: str | None,
+ workflow_permanent_id: str | None = None,
+ workflow_id: str | None = None,
+ ) -> bool:
+ """Whether this run must be dispatched to a worker pool that has the runner sidecar.
+
+ Answered at dispatch, before a queue is chosen, so it is run-level: no block context
+ exists yet. OSS has no runner and returns False; cloud overrides to consult the same
+ gate as should_use_codeblock_runner.
+ """
+ return False
+
async def execute_code_block_override(
self,
*,
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
index ca9435d3f..23bd2616f 100644
--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
@@ -238,7 +238,7 @@ BUILD-TIME DISCOVERY:
* `site_or_url` accepts: a URL with or without scheme (`example.com/login` is fine), a bare domain (`example.com`), or a single brand word. A brand word returns a candidate only after an exact provider-backed official-site association and guarded public-network HTTPS navigation to that associated origin; otherwise it returns `failure_reason=could_not_resolve_site_name`. English phrases ("the X website") also return that failure — `ASK_QUESTION` for a URL instead.
* When discovery returns a `candidate_url`, USE IT — call `update_and_run_blocks` immediately with a `goto_url` block whose `url:` is that `candidate_url`. Do NOT `ASK_QUESTION` for confirmation the user already implied by naming the site; confirmation is appropriate only when discovery returned a `failure_reason`.
* Discovery v3 returns `contract_version=discover_workflow_entrypoint_v3`, `candidate_url`, `candidate_form_fields`, and an `evidence_trail`; evidence-backed brand candidates also return bounded association provenance, guarded-hop evidence, and final-origin binding. Use `candidate_url` as the `url:` value on a `goto_url` block. Do NOT paste evidence verbatim into workflow YAML.
-* Budget: one successful call per turn, three per chat, eight page hops, sixty seconds. On any `failure_reason` (`step_limit`, `no_candidate`, `could_not_resolve_site_name`, `login_wall`, `anti_bot_wall`, `discovery_budget_exhausted_for_chat`), `ASK_QUESTION` for a specific URL — do not propose a vendor of your own.
+* Budget: one successful call per turn, three per chat, eight page hops, sixty seconds. On any `failure_reason` (`step_limit`, `no_candidate`, `could_not_resolve_site_name`, `anti_bot_wall`, `discovery_budget_exhausted_for_chat`), `ASK_QUESTION` for a specific URL — do not propose a vendor of your own.
* Discovery navigates and reads pages. It will NOT type, click form buttons, run JavaScript, or submit forms.
COMPOSITION PAGE INSPECTION:
diff --git a/skyvern/forge/sdk/agents/context.py b/skyvern/forge/sdk/agents/context.py
index 9c17cf643..ede3ed8c7 100644
--- a/skyvern/forge/sdk/agents/context.py
+++ b/skyvern/forge/sdk/agents/context.py
@@ -51,6 +51,7 @@ def compact_agent_messages_for_llm(
summarize_tool_output: Callable[[str], str] | None = None,
summarize_tool_arguments: Callable[[str], str] | None = None,
tool_output_truncation_suffix: str = DEFAULT_TRUNCATION_SUFFIX,
+ on_recent_truncation: Callable[[int, int], None] | None = None,
token_budget: int | None = None,
estimate_tokens: Callable[[list[Any]], int] | None = None,
is_synthetic_message: Callable[[Any], bool] | None = None,
@@ -75,6 +76,7 @@ def compact_agent_messages_for_llm(
summarize_tool_output=summarize_tool_output,
summarize_tool_arguments=summarize_tool_arguments,
suffix=tool_output_truncation_suffix,
+ on_recent_truncation=on_recent_truncation,
)
if token_budget is None:
return items
@@ -147,6 +149,7 @@ def _compact_tool_payloads(
summarize_tool_output: Callable[[str], str] | None,
summarize_tool_arguments: Callable[[str], str] | None,
suffix: str,
+ on_recent_truncation: Callable[[int, int], None] | None = None,
) -> list[Any]:
output_indices = [i for i, item in enumerate(items) if _tool_output_field(item) is not None]
call_indices = [i for i, item in enumerate(items) if get_agent_message_field(item, "type") == "function_call"]
@@ -154,6 +157,8 @@ def _compact_tool_payloads(
recent_calls = set(call_indices[-keep_recent_tool_outputs:]) if keep_recent_tool_outputs > 0 else set()
compacted: list[Any] = []
+ recent_truncated_count = 0
+ recent_truncated_largest = 0
for i, item in enumerate(items):
output_field = _tool_output_field(item)
if output_field is not None:
@@ -164,6 +169,9 @@ def _compact_tool_payloads(
if i in recent_outputs
else _summarize_or_truncate(output, summarize_tool_output, max_recent_tool_output_chars, suffix)
)
+ if i in recent_outputs and new_output != output:
+ recent_truncated_count += 1
+ recent_truncated_largest = max(recent_truncated_largest, len(output))
item = replace_agent_message_field(item, output_field, new_output) if new_output != output else item
elif get_agent_message_field(item, "type") == "function_call" and i not in recent_calls:
arguments = get_agent_message_field(item, "arguments")
@@ -180,6 +188,8 @@ def _compact_tool_payloads(
else item
)
compacted.append(item)
+ if recent_truncated_count and on_recent_truncation is not None:
+ on_recent_truncation(recent_truncated_count, recent_truncated_largest)
return compacted
diff --git a/skyvern/forge/sdk/copilot/code_block_synthesis.py b/skyvern/forge/sdk/copilot/code_block_synthesis.py
index 0398cb022..b956cdfaf 100644
--- a/skyvern/forge/sdk/copilot/code_block_synthesis.py
+++ b/skyvern/forge/sdk/copilot/code_block_synthesis.py
@@ -4102,6 +4102,11 @@ def _render_artifact_metadata_block(metadata: Mapping[str, Any]) -> str:
return json.dumps(metadata, indent=2, sort_keys=True)
+# The rendered offer's message content must begin with this sentinel; the
+# supersede-collapse and synthetic-turn classification key on the prefix.
+SYNTHESIZED_OFFER_SENTINEL = "SYNTHESIZED CODE BLOCK (offered once)."
+
+
def render_synthesized_offer_text(
synthesized: SynthesizedCodeBlock,
trajectory: Sequence[Mapping[str, Any]] | None = None,
@@ -4120,7 +4125,7 @@ def render_synthesized_offer_text(
)
)
parts = [
- "SYNTHESIZED CODE BLOCK (offered once). The page interactions you scouted were compiled into a "
+ SYNTHESIZED_OFFER_SENTINEL + " The page interactions you scouted were compiled into a "
"deterministic Playwright snippet. Persist it VERBATIM as a `code` block labeled "
f"`{_SYNTHESIZED_BLOCK_LABEL}` via update_workflow / update_and_run_blocks. {extraction_instruction}",
"```python",
diff --git a/skyvern/forge/sdk/copilot/enforcement.py b/skyvern/forge/sdk/copilot/enforcement.py
index 4a46f3d9a..f40891c6c 100644
--- a/skyvern/forge/sdk/copilot/enforcement.py
+++ b/skyvern/forge/sdk/copilot/enforcement.py
@@ -39,6 +39,7 @@ from skyvern.forge.sdk.copilot.code_block_synthesis import (
CREDENTIAL_FILL_TOOL_NAME,
LIVE_SCOUT_CREDENTIAL_FIELDS,
ONE_TIME_CODE_CREDENTIAL_FIELD,
+ SYNTHESIZED_OFFER_SENTINEL,
ObligationFinding,
credential_scout_gap,
credential_submit_boundary_index,
@@ -230,7 +231,13 @@ TOKENS_PER_RESIZED_IMAGE = 765
# Keep the last N function_call_output items at full (head-truncated) size.
# Older outputs collapse to a compact synopsis so context doesn't grow linearly.
KEEP_RECENT_TOOL_OUTPUTS = 3
-_RECENT_TOOL_OUTPUT_CHAR_CAP = 2000
+# A tripwire against pathological payloads, not a routine ration: code-bearing tool
+# results (synthesized blocks, edit re-anchor echoes) run 3-30KB plus JSON escaping
+# and must reach the model whole, so the cap sits above that class with headroom.
+_RECENT_TOOL_OUTPUT_CHAR_CAP = 50_000
+# Older tool-call arguments that fail to summarize keep only this much; that channel
+# exists to shrink replayed context and must not follow the recent-window cap.
+_SUMMARIZED_TOOL_ARGUMENT_CHAR_CAP = 2000
_TOOL_OUTPUT_SUMMARIZE_THRESHOLD = 300
_TOOL_OUTPUT_TRUNCATION_SUFFIX = "\n... [older tool output truncated]"
# Head-truncation marker for the recent tool-output window. Kept on a
@@ -1648,9 +1655,31 @@ def _is_nudge_message(item: Any) -> bool:
return isinstance(content, str) and content.startswith(NUDGE_SENTINEL)
+def _is_synthesized_offer_message(item: Any) -> bool:
+ if _item_field(item, "role") != "user":
+ return False
+ content = _item_field(item, "content")
+ return isinstance(content, str) and content.startswith(SYNTHESIZED_OFFER_SENTINEL)
+
+
def is_synthetic_user_message(item: Any) -> bool:
- """Return True if item is a screenshot or nudge (not a real user turn)."""
- return is_screenshot_message(item) or _is_nudge_message(item)
+ """Return True if item is a screenshot, nudge, or synthesized-block offer
+ (not a real user turn)."""
+ return is_screenshot_message(item) or _is_nudge_message(item) or _is_synthesized_offer_message(item)
+
+
+def collapse_superseded_synthesized_offers(items: list[Any]) -> list[Any]:
+ """Drop every synthesized-block offer except the newest: a refreshed offer supersedes its
+ predecessors, and offers ride as user messages no other compaction rung touches. Applied on
+ every model-input assembly path before token estimation; the opening item is never dropped.
+ """
+ offer_indices = [i for i, item in enumerate(items) if i > 0 and _is_synthesized_offer_message(item)]
+ if len(offer_indices) <= 1:
+ return items
+ stale = set(offer_indices[:-1])
+ dropped_chars = sum(len(_item_field(items[i], "content") or "") for i in stale)
+ LOG.info("copilot_superseded_offers_dropped", dropped=len(stale), dropped_chars=dropped_chars)
+ return [item for i, item in enumerate(items) if i not in stale]
def _truncated_output_fallback(output: str) -> str:
@@ -1680,6 +1709,9 @@ def _summarize_tool_output(output: str) -> str:
data = parsed.get("data")
if isinstance(data, dict):
+ code = data.get("code")
+ if isinstance(code, str) and code:
+ synopsis["code_chars_elided"] = len(code)
for key in ("overall_status", "workflow_run_id", "failure_reason", "url", "message"):
val = data.get(key)
if val is None or val == "":
@@ -1750,9 +1782,9 @@ def _summarize_tool_arguments(args_json: str) -> str:
try:
parsed = json.loads(args_json)
except (TypeError, ValueError):
- return args_json[:_RECENT_TOOL_OUTPUT_CHAR_CAP] + _TOOL_OUTPUT_TRUNCATION_SUFFIX
+ return args_json[:_SUMMARIZED_TOOL_ARGUMENT_CHAR_CAP] + _TOOL_OUTPUT_TRUNCATION_SUFFIX
if not isinstance(parsed, dict):
- return args_json[:_RECENT_TOOL_OUTPUT_CHAR_CAP] + _TOOL_OUTPUT_TRUNCATION_SUFFIX
+ return args_json[:_SUMMARIZED_TOOL_ARGUMENT_CHAR_CAP] + _TOOL_OUTPUT_TRUNCATION_SUFFIX
compact: dict[str, Any] = {}
for key, val in parsed.items():
if isinstance(val, str) and len(val) > 500:
@@ -1766,7 +1798,16 @@ def _summarize_tool_arguments(args_json: str) -> str:
try:
return json.dumps(compact, separators=(",", ":"))
except (TypeError, ValueError):
- return args_json[:_RECENT_TOOL_OUTPUT_CHAR_CAP] + _TOOL_OUTPUT_TRUNCATION_SUFFIX
+ return args_json[:_SUMMARIZED_TOOL_ARGUMENT_CHAR_CAP] + _TOOL_OUTPUT_TRUNCATION_SUFFIX
+
+
+def log_recent_tool_output_truncation(truncated_count: int, largest_original_chars: int) -> None:
+ LOG.warning(
+ "copilot_recent_tool_output_truncated",
+ truncated_count=truncated_count,
+ cap=_RECENT_TOOL_OUTPUT_CHAR_CAP,
+ largest_original_chars=largest_original_chars,
+ )
def _prune_input_list(items: list[Any]) -> list[Any]:
@@ -1778,6 +1819,7 @@ def _prune_input_list(items: list[Any]) -> list[Any]:
function_call items keep the last KEEP_RECENT_TOOL_OUTPUTS at full size
(head-truncated); older ones collapse to JSON synopses.
"""
+ items = collapse_superseded_synthesized_offers(items)
screenshot_indices = [i for i, item in enumerate(items) if is_screenshot_message(item)]
drop_indices = set(screenshot_indices[:-1])
@@ -1788,6 +1830,8 @@ def _prune_input_list(items: list[Any]) -> list[Any]:
recent_fc_set = set(fc_indices[-KEEP_RECENT_TOOL_OUTPUTS:])
result: list[Any] = []
+ recent_truncated_count = 0
+ recent_truncated_largest = 0
for i, item in enumerate(items):
if i in drop_indices:
result.append({"role": "user", "content": SCREENSHOT_PLACEHOLDER})
@@ -1798,11 +1842,13 @@ def _prune_input_list(items: list[Any]) -> list[Any]:
output = _item_field(item, "output")
if isinstance(output, str):
if i in recent_fco_set:
- new_output = (
- output[:_RECENT_TOOL_OUTPUT_CHAR_CAP] + _TOOL_OUTPUT_HEAD_TRUNCATION_SUFFIX
- if len(output) > _RECENT_TOOL_OUTPUT_CHAR_CAP
- else output
- )
+ if len(output) > _RECENT_TOOL_OUTPUT_CHAR_CAP:
+ new_output = output[:_RECENT_TOOL_OUTPUT_CHAR_CAP] + _TOOL_OUTPUT_HEAD_TRUNCATION_SUFFIX
+ if new_output != output:
+ recent_truncated_count += 1
+ recent_truncated_largest = max(recent_truncated_largest, len(output))
+ else:
+ new_output = output
else:
new_output = _summarize_tool_output(output)
if new_output != output:
@@ -1815,6 +1861,8 @@ def _prune_input_list(items: list[Any]) -> list[Any]:
item = _replace_item_field(item, "arguments", new_args)
result.append(item)
+ if recent_truncated_count:
+ log_recent_tool_output_truncation(recent_truncated_count, recent_truncated_largest)
return result
diff --git a/skyvern/forge/sdk/copilot/request_policy.py b/skyvern/forge/sdk/copilot/request_policy.py
index 954840e6f..4c0e1355c 100644
--- a/skyvern/forge/sdk/copilot/request_policy.py
+++ b/skyvern/forge/sdk/copilot/request_policy.py
@@ -218,22 +218,6 @@ _INVALID_CONDITIONAL_CONTAINER_MARKERS = (
"inside conditional",
"within conditional",
)
-# The leading (?<!\w) keeps a contraction's apostrophe ("I've", "can't") from opening a quote and
-# pairing with a later one, which would swallow the run between them as a bogus credential name.
-_QUOTED_CREDENTIAL_NAME_RE = re.compile(r"(?:`([^`]{1,100})`|\"([^\"]{1,100})\"|(?<!\w)'([^']{1,100})')")
-_NAMED_CREDENTIAL_TOKEN_RE = re.compile(
- r"\b(?:saved\s+credential|credential)\s+(?:named|called)\s+([A-Za-z0-9_.@:-]{2,100})\b",
- re.I,
-)
-_DIRECT_CREDENTIAL_TOKEN_RE = re.compile(
- r"\b(?:use|using|with)\s+(?:the\s+)?(?:saved\s+)?credential"
- r"(?:\s+(?:named|called))?\s+([A-Za-z0-9_.@:-]{2,100})\b",
- re.I,
-)
-_POSTFIX_CREDENTIAL_TOKEN_RE = re.compile(
- r"\b(?:use|using|with)\s+(?:my\s+|the\s+)?(?:saved\s+)?([A-Za-z0-9_.@:-]{2,100})\s+credential\b",
- re.I,
-)
_CREDENTIAL_REPLACEMENT_TARGET_RES = (
re.compile(
r"\bswitch(?:ing)?\s+to\b"
@@ -263,7 +247,6 @@ _CREDENTIAL_REPLACEMENT_OBJECT_RE = re.compile(
r"(?:\s+(?:(?:id|named|called)\s+)?[A-Za-z0-9_.@:-]{2,100})?)\s*",
re.I,
)
-_CREDENTIAL_QUOTE_CONTEXT_RE = re.compile(r"\b(?:credentials?|log[\s-]?in)\b", re.I)
_NEGATED_CREDENTIAL_CONTEXT_RE = re.compile(
r"(?:"
r"\b(?:do\s+not|don't|never|without|avoid(?:ing)?|exclud(?:e|ed|es|ing))\b"
@@ -278,9 +261,6 @@ _EXPLICIT_LOGIN_ACTION_RE = re.compile(
r"(?!\s+(?:form|page|path|route|screen|url)\b)",
re.I,
)
-_CREDENTIAL_REFERENCE_STOPWORDS = frozenset(
- {"a", "an", "credential", "for", "login", "my", "saved", "stored", "the", "to"}
-)
_CODE_BLOCK_AUTHORING_MARKERS = ("code block", "code-block", "codeblock")
_LOGIN_BLOCK_BAN_MARKERS = ("do not create a login block", "don't create a login block", "no login block")
_CREDENTIAL_CODE_MARKERS = ("saved credential", "login_credentials", ".otp()", "one-time-code")
@@ -4441,10 +4421,6 @@ async def _classify_request(
return policy
-def _quote_in_credential_context(user_message: str, quote_start: int) -> bool:
- return bool(_CREDENTIAL_QUOTE_CONTEXT_RE.search(user_message[max(0, quote_start - 48) : quote_start]))
-
-
def _credential_reference_is_negated(user_message: str, reference_start: int) -> bool:
clause_start = max(user_message.rfind(delimiter, 0, reference_start) for delimiter in (".", "!", "?", "\n", ";"))
context = user_message[clause_start + 1 : reference_start]
@@ -4480,71 +4456,20 @@ def _credential_authority_text(user_message: str) -> str:
return f"use {user_message[max(replacement_targets) :].lstrip()}"
-def _exact_credential_name_candidates(user_message: str) -> list[str]:
- text = _credential_authority_text(user_message or "")
- mentions: list[tuple[str, int, bool, bool]] = []
-
- def add_mention(value: str, start: int, *, context_matches: bool = True) -> None:
- value = value.strip()
- if not context_matches or not value or value.lower() in _CREDENTIAL_REFERENCE_STOPWORDS:
- return
- mentions.append(
- (
- value,
- start,
- _credential_reference_is_negated(text, start),
- _credential_reference_is_unrelated_replacement(text, start),
- )
- )
-
- for match in _QUOTED_CREDENTIAL_NAME_RE.finditer(text):
- group_index = next((index for index, group in enumerate(match.groups(), start=1) if group), 1)
- value = (match.group(group_index) or "").strip()
- add_mention(
- value,
- match.start(),
- context_matches=_quote_in_credential_context(text, match.start()),
- )
- for match in _NAMED_CREDENTIAL_TOKEN_RE.finditer(text):
- add_mention(match.group(1), match.start(1))
- for match in _DIRECT_CREDENTIAL_TOKEN_RE.finditer(text):
- add_mention(match.group(1), match.start(1))
- for match in _POSTFIX_CREDENTIAL_TOKEN_RE.finditer(text):
- add_mention(match.group(1), match.start(1))
-
- last_negated_mention = {
- value: start for value, start, is_negated, is_unrelated in mentions if is_negated and not is_unrelated
- }
- return _clean_list(
- [
- value
- for value, start, is_negated, is_unrelated in mentions
- if not is_negated and not is_unrelated and last_negated_mention.get(value, -1) < start
- ]
- )
-
-
-def _credential_name_candidates(user_message: str) -> list[str]:
- # Classifier refs are model-authored hints, not current-turn credential
- # authority. Resolve names only from deterministic, affirmative references
- # in the latest user message.
- return _exact_credential_name_candidates(user_message)
-
-
_CREDENTIAL_NAME_MIN_DISTINCTIVE_LENGTH = 4
-# Labels that mean the classifier read this turn as referring to a stored credential at all.
-_CREDENTIAL_REFERENCE_INPUT_KINDS = frozenset({"credential_id", "credential_name", "website_stored_credential"})
-def _explicit_login_action_asserted(user_message: str) -> bool:
- """Whether the message asks for a login in its own words, ignoring phrasings that refuse one.
+def _asserted_in_own_words(user_message: str, phrase: re.Pattern[str]) -> bool:
+ """Whether the message uses the phrase affirmatively, ignoring occurrences that refuse one.
Shares the negation rule with saved-name matching, so "do not log in with prod" authorizes neither.
"""
text = _credential_authority_text(user_message or "")
- return any(
- not _credential_reference_is_negated(text, match.start()) for match in _EXPLICIT_LOGIN_ACTION_RE.finditer(text)
- )
+ return any(not _credential_reference_is_negated(text, match.start()) for match in phrase.finditer(text))
+
+
+def _explicit_login_action_asserted(user_message: str) -> bool:
+ return _asserted_in_own_words(user_message, _EXPLICIT_LOGIN_ACTION_RE)
def _wrapper_start(text: str, index: int) -> int:
@@ -4554,47 +4479,78 @@ def _wrapper_start(text: str, index: int) -> int:
return index
-def _name_stated_affirmatively(text: str, name: str) -> bool:
+def _name_stated_affirmatively(text: str, name: str, *, min_length: int = 1) -> bool:
name = (name or "").strip()
- if len(name) < _CREDENTIAL_NAME_MIN_DISTINCTIVE_LENGTH or name.lower() in _CREDENTIAL_REFERENCE_STOPWORDS:
+ if len(name) < min_length:
return False
- pattern = rf"(?<![A-Za-z0-9_.@:-]){re.escape(name)}(?![A-Za-z0-9_.@:-])"
+ # A dot only continues the name when a name character follows it, so a saved name ending a
+ # sentence still matches while `prod` never claims `prod.staging`.
+ pattern = rf"(?<![A-Za-z0-9_.@:-]){re.escape(name)}(?!\.?[A-Za-z0-9_@:-])"
+ last_affirmative = last_negated = -1
for match in re.finditer(pattern, text, re.IGNORECASE):
# Negation reads the clause before the reference, and any wrapper around the name belongs to
# it: from inside the quotes, "credential, `prod`" no longer reads as an appositive.
start = _wrapper_start(text, match.start())
- if not _credential_reference_is_negated(text, start) and not _credential_reference_is_unrelated_replacement(
- text, start
- ):
- return True
- return False
+ if _credential_reference_is_unrelated_replacement(text, start):
+ continue
+ if _credential_reference_is_negated(text, start):
+ last_negated = start
+ else:
+ last_affirmative = start
+ # A name the message goes on to exclude is withdrawn, however affirmatively it was stated first.
+ return last_affirmative > last_negated
-def _saved_credential_names_mentioned(user_message: str, credentials: list[Credential]) -> list[str]:
- """Saved names a message states outright, e.g. "use skyvern-datadog to login".
+def _saved_credential_names_mentioned(
+ policy: RequestPolicy, user_message: str, credentials: list[Credential]
+) -> list[str]:
+ """Saved names this turn refers to: the names read out of the message, kept when the org has a
+ credential by that name.
- The saved names are known, so match on them rather than on the prose around them.
+ Searching the message for each saved name instead would match any text that happens to equal
+ one — "rename the hex block", or a `/hex` path inside a URL — so the reading is left to the
+ model and this checks its answer against what the org actually has. The message is still read
+ for negation, which the extraction does not carry.
"""
text = _credential_authority_text(user_message or "")
+ stated = {
+ ref.strip().casefold()
+ for ref in policy.credential_refs
+ if isinstance(ref, str) and ref.strip() and _name_stated_affirmatively(text, ref.strip())
+ }
+ # Names are not unique and differ freely in case, so every saved name a ref could mean is a
+ # candidate and the picker asks, rather than one of them being bound on the org's row order.
return _clean_list(
[
(credential.name or "").strip()
for credential in credentials
- if _name_stated_affirmatively(text, credential.name or "")
+ if (credential.name or "").strip().casefold() in stated
]
)
-def _classifier_ref_stated_in_message(policy: RequestPolicy, user_message: str) -> bool:
- """Whether a credential-labelled turn's extracted name appears affirmatively in this message.
+def _classifier_refs_stated_in_message(
+ policy: RequestPolicy, user_message: str, *, min_length: int = _CREDENTIAL_NAME_MIN_DISTINCTIVE_LENGTH
+) -> list[str]:
+ """Names read out of this message that it also states affirmatively.
- Which credential label the classifier chose must not decide whether a name the user wrote is
- looked up by name (SKY-12917), but a turn it never called a credential request may not scan.
+ Not credential authority — the org's saved names decide that; this only lets the turn name a
+ credential it could not find. Which label the reading carried does not gate it (SKY-12917).
"""
- if policy.classifier_status == "fallback" or policy.credential_input_kind not in _CREDENTIAL_REFERENCE_INPUT_KINDS:
- return False
+ if policy.classifier_status == "fallback":
+ return []
text = _credential_authority_text(user_message or "")
- return any(isinstance(ref, str) and _name_stated_affirmatively(text, ref) for ref in policy.credential_refs)
+ return _clean_list(
+ [
+ ref
+ for ref in policy.credential_refs
+ if isinstance(ref, str)
+ and not _CREDENTIAL_ID_RE.fullmatch(ref)
+ # A model-authored ref is not a known name, so it still needs to look like one before it
+ # can be echoed back to the user.
+ and _name_stated_affirmatively(text, ref, min_length=min_length)
+ ]
+ )
def _explicit_credential_ids(user_message: str) -> list[str]:
@@ -4725,7 +4681,9 @@ def _has_resolvable_credential_scope(policy: RequestPolicy, user_message: str =
return False
if _explicit_credential_ids(user_message):
return True
- if _credential_name_candidates(user_message):
+ # Whether a name is distinctive enough to quote back at the user is a different question from
+ # whether the turn named a credential at all, and a saved `qa` answers the second one.
+ if _classifier_refs_stated_in_message(policy, user_message, min_length=1):
return True
if policy.login_page_urls:
return True
@@ -4955,9 +4913,9 @@ def _can_defer_unresolved_credential_name_for_draft(
) -> bool:
if policy.clarification_reason != "credential_name_unresolved":
return False
- # A classifier-carried name may justify leaving an unvalidated placeholder
- # in a draft, but _credential_name_candidates still requires current-turn
- # anchoring before the name can enter resolved_credentials and authorize a run.
+ # A classifier-carried name may justify leaving an unvalidated placeholder in a draft, but a
+ # name still has to match a saved credential before it can enter resolved_credentials and
+ # authorize a run.
if policy.credential_refs:
return True
if _has_resolvable_credential_scope(policy, user_message):
@@ -5026,26 +4984,21 @@ async def _resolve_credentials(
return
# `login_intent` is a classifier-authored hint that misses on runs where the model does not set
- # it, so a message asking for a login in its own words must still count. A classifier that failed
- # outright is a different case: a degraded turn never enumerates the org's saved credentials.
+ # it, so a message asking for a login or naming a credential in its own words must still count. A
+ # classifier that failed outright is a different case: a degraded turn never enumerates the org's
+ # saved credentials.
+ stated_refs = _classifier_refs_stated_in_message(policy, user_message)
login_phrase_asserted = _explicit_login_action_asserted(user_message)
- may_scan_saved_names = (
- policy.login_intent
- or (policy.classifier_status != "fallback" and login_phrase_asserted)
- or _classifier_ref_stated_in_message(policy, user_message)
- )
- name_candidates = _credential_name_candidates(user_message)
+ # Only a turn that reported a name has anything to look up, so a message about nothing of the
+ # kind never reads the org's credentials.
credentials: list[Credential] | None = None
- # Only a request that already asserts a login or credential intent may be matched against the
- # org's saved names; without that gate an unrelated message would scan the credential list.
- if not name_candidates and may_scan_saved_names:
+ name_candidates: list[str] = []
+ if policy.credential_refs:
credentials = await _load_credentials(organization_id)
- name_candidates = _saved_credential_names_mentioned(user_message, credentials)
- if name_candidates:
- if credentials is None:
- credentials = await _load_credentials(organization_id)
+ name_candidates = _saved_credential_names_mentioned(policy, user_message, credentials)
+ if name_candidates or stated_refs:
named_matches = _deduplicate_credentials(
- [credential for credential in credentials if credential.name in name_candidates]
+ [credential for credential in credentials or [] if credential.name in name_candidates]
)
if len(named_matches) == 1:
policy.resolved_credentials = named_matches
@@ -5087,9 +5040,9 @@ async def _resolve_credentials(
policy.allow_run_blocks = False
return
question = (
- f"I could not find a stored credential named `{name_candidates[0]}`. Please choose an existing credential "
+ f"I could not find a stored credential named `{stated_refs[0]}`. Please choose an existing credential "
"by exact name or a credential ID beginning with cred_."
- if policy.credential_input_kind == "credential_name"
+ if policy.credential_input_kind == "credential_name" and stated_refs
else _clarification_question(policy)
)
_block(
diff --git a/skyvern/forge/sdk/copilot/session_factory.py b/skyvern/forge/sdk/copilot/session_factory.py
index f6dfbf380..a53ceb1cb 100644
--- a/skyvern/forge/sdk/copilot/session_factory.py
+++ b/skyvern/forge/sdk/copilot/session_factory.py
@@ -30,17 +30,28 @@ from skyvern.forge.sdk.copilot.enforcement import (
_summarize_tool_arguments,
_summarize_tool_output,
aggressive_prune,
+ collapse_superseded_synthesized_offers,
estimate_tokens,
is_screenshot_message,
is_synthetic_user_message,
+ log_recent_tool_output_truncation,
)
LOG = structlog.get_logger()
RECENT_REAL_TURNS = 2
+TOOL_OUTPUT_TRUNCATE_SOFT = 2000
TOOL_OUTPUT_TRUNCATE_EMERGENCY = 300
+def _emergency_truncate_all(items: list[Any], cap: int) -> list[Any]:
+ truncated_items = [_truncate_tool_output(item, cap) for item in items]
+ truncated_count = sum(1 for old, new in zip(items, truncated_items) if new is not old)
+ if truncated_count:
+ LOG.warning("copilot_tool_output_emergency_truncated", truncated_count=truncated_count, cap=cap)
+ return truncated_items
+
+
def create_copilot_session(chat_id: str) -> SQLiteSession:
"""Create an in-memory SQLiteSession scoped to a single copilot request."""
return SQLiteSession(session_id=chat_id, db_path=":memory:")
@@ -54,9 +65,21 @@ def _compact_tool_items(items: list[Any]) -> list[Any]:
summarize_tool_output=_summarize_tool_output,
summarize_tool_arguments=_summarize_tool_arguments,
tool_output_truncation_suffix=_TOOL_OUTPUT_HEAD_TRUNCATION_SUFFIX,
+ on_recent_truncation=log_recent_tool_output_truncation,
)
+def _screenshot_with_image(item: Any) -> bool:
+ """A sentinel-prefixed item only counts as the keepable frame when it actually
+ carries an image part; text-only lookalikes and placeholders must not shadow it."""
+ if not is_screenshot_message(item):
+ return False
+ content = get_agent_message_field(item, "content")
+ if not isinstance(content, list):
+ return False
+ return any(get_agent_message_field(block, "type") == "input_image" for block in content)
+
+
def copilot_session_input_callback(
history_items: list[Any],
new_items: list[Any],
@@ -65,12 +88,12 @@ def copilot_session_input_callback(
payloads in the middle region.
Keeps the original goal (first item) at full fidelity and preserves the
- last ``RECENT_REAL_TURNS`` real user turns. Within the remaining middle
- region, older ``function_call_output`` / ``function_call`` items are
- compacted using the same ``KEEP_RECENT_TOOL_OUTPUTS`` rule that
- ``enforcement._prune_input_list`` uses in the non-session path, so
- first-turn transcripts with a long tool chain get compacted identically
- regardless of user-turn count.
+ last ``RECENT_REAL_TURNS`` real user turns — in production every injected
+ copilot message is synthetic, so the whole post-goal history is the middle
+ region. Within it, older ``function_call_output`` / ``function_call`` items
+ are compacted using the same ``KEEP_RECENT_TOOL_OUTPUTS`` rule that
+ ``enforcement._prune_input_list`` uses in the non-session path, and the
+ newest screenshot survives unless a newer one rides in recent/new items.
"""
if not history_items:
return new_items
@@ -95,9 +118,17 @@ def copilot_session_input_callback(
recent = []
pruned_middle = _compact_tool_items(middle)
+ # The newest frame must survive the merge: with every injected message classified
+ # synthetic, the whole post-goal history is middle, and blanket-replacing would
+ # blind the agent to the frame it just captured once it rotates into history.
+ middle_screenshots = [i for i, item in enumerate(pruned_middle) if _screenshot_with_image(item)]
+ has_newer_screenshot = any(_screenshot_with_image(item) for item in list(recent) + list(new_items))
+ keep_screenshot = middle_screenshots[-1] if middle_screenshots and not has_newer_screenshot else None
pruned_middle = [
- {"role": "user", "content": SCREENSHOT_PLACEHOLDER} if is_screenshot_message(item) else item
- for item in pruned_middle
+ {"role": "user", "content": SCREENSHOT_PLACEHOLDER}
+ if is_screenshot_message(item) and i != keep_screenshot
+ else item
+ for i, item in enumerate(pruned_middle)
]
return [history_items[0]] + pruned_middle + list(recent) + list(new_items)
@@ -165,11 +196,12 @@ def _maybe_dump_model_input(data: CallModelData[Any], model_data: ModelInputData
def _filter_to_budget(data: CallModelData[Any], *, token_budget: int) -> ModelInputData:
"""Token-budget enforcement applied just before each model call.
- Graduated pruning:
+ Graduated pruning, after dropping superseded synthesized-block offers:
1. Compact older tool outputs + function-call arguments using the
KEEP_RECENT_TOOL_OUTPUTS rule (mirrors ``enforcement._prune_input_list``).
2. If still over budget: drop all screenshots except the most recent.
- 3. If still over budget: truncate ALL tool outputs to 300 chars.
+ 3. If still over budget: truncate ALL tool outputs — first to 2000 chars,
+ then to 300 only if the softer pass was not enough.
4. If still over budget: aggressive prune as last resort.
"""
model_data = data.model_data
@@ -181,6 +213,11 @@ def _filter_to_budget(data: CallModelData[Any], *, token_budget: int) -> ModelIn
est = estimate_tokens(items)
LOG.info("Token estimate before filtering", tokens=est)
+ # Superseded offers are pure waste, dropped before the budget rungs so the freed
+ # tokens stop triggering emergency truncation. The estimate above deliberately
+ # stays pre-collapse: it is the metric that surfaces a recurrence of the stacking.
+ items = collapse_superseded_synthesized_offers(items)
+
# Re-run compaction here even though ``copilot_session_input_callback``
# already compacted on session merge. The KEEP_RECENT_TOOL_OUTPUTS window
# shifts whenever new items get appended — an output that was "recent" on
@@ -207,8 +244,17 @@ def _filter_to_budget(data: CallModelData[Any], *, token_budget: int) -> ModelIn
LOG.info("Within budget after screenshot drop", tokens=est)
return ModelInputData(input=items, instructions=model_data.instructions)
- # Layer 3: Truncate ALL tool outputs to 300 chars
- items = [_truncate_tool_output(item, TOOL_OUTPUT_TRUNCATE_EMERGENCY) for item in items]
+ # Layer 3a: bring every tool output down to the pre-raise bound before resorting
+ # to the harsher pass, so a code-bearing recent output degrades gracefully.
+ items = _emergency_truncate_all(items, TOOL_OUTPUT_TRUNCATE_SOFT)
+
+ est = estimate_tokens(items)
+ if est <= token_budget:
+ LOG.info("Within budget after soft emergency truncation", tokens=est)
+ return ModelInputData(input=items, instructions=model_data.instructions)
+
+ # Layer 3b: Truncate ALL tool outputs to 300 chars
+ items = _emergency_truncate_all(items, TOOL_OUTPUT_TRUNCATE_EMERGENCY)
est = estimate_tokens(items)
if est <= token_budget:
diff --git a/skyvern/forge/sdk/copilot/tools/banned_blocks.py b/skyvern/forge/sdk/copilot/tools/banned_blocks.py
index 756dbde5d..5b022ca18 100644
--- a/skyvern/forge/sdk/copilot/tools/banned_blocks.py
+++ b/skyvern/forge/sdk/copilot/tools/banned_blocks.py
@@ -280,8 +280,9 @@ Runtime facts:
with challenge controls directly.
- After challenge handling, wait for either the observed one-time-code field or
a real authenticated-page anchor. If an OTP field appears, fill it with
- `await <credential_key>.otp()`, submit the observed Next/Verify control, and
- then wait for the authenticated anchor. Do not treat disappearance of the
+ `await <credential_key>.otp()`, submit it with the exact control the scout
+ demonstrated (its recorded selector and label, never a guessed one), and then
+ wait for the authenticated anchor. Do not treat disappearance of the
login fields as proof of authentication.
- Return JSON-safe structured data plus visible evidence text for records, totals,
confirmations, and identifiers.
diff --git a/skyvern/forge/sdk/copilot/tools/discovery.py b/skyvern/forge/sdk/copilot/tools/discovery.py
index 7bc9cd910..b4d841e8e 100644
--- a/skyvern/forge/sdk/copilot/tools/discovery.py
+++ b/skyvern/forge/sdk/copilot/tools/discovery.py
@@ -104,6 +104,38 @@ def _user_provided_entry_url(copilot_ctx: CopilotContext) -> str | None:
return urls[0] if len(urls) == 1 else None
+def _site_name_names_host(site_name: str, url: str) -> bool:
+ """Whether a hostname label opens with the requested name.
+
+ A label may extend the name (`datadog` names `datadoghq`) but may not merely contain it, so
+ `ups` does not name `groupsupport`.
+ """
+ normalized = _normalize_discovery_name(site_name or "")
+ if len(normalized) < 3:
+ return False
+ try:
+ host = urlparse(url).hostname or ""
+ except ValueError:
+ return False
+ return any(_normalize_discovery_name(label).startswith(normalized) for label in host.split("."))
+
+
+def _credential_entry_url(copilot_ctx: CopilotContext, site_name: str) -> str | None:
+ """The login page a credential resolved for this turn already signs in at.
+
+ Only a credential whose host carries the requested name answers, because approvals persist
+ across turns and a later request for a different site would otherwise open this one.
+ """
+ policy = copilot_ctx.request_policy
+ credentials = list(policy.resolved_credentials) if policy is not None else []
+ named: list[str] = []
+ for credential in credentials:
+ url = (credential.tested_url or "").strip()
+ if url and url not in named and _site_name_names_host(site_name, url):
+ named.append(url)
+ return named[0] if len(named) == 1 else None
+
+
def _resolve_discovery_entry_url(site_or_url: str) -> tuple[str | None, str]:
"""Resolve the user-supplied site name/URL into a navigable URL.
@@ -632,7 +664,9 @@ async def _discovery_walk(
anti_bot_has_no_candidate_evidence = (
not form_fields and candidate_title_score == 0 and candidate_anchor_score == 0
)
- if anti_bot_detected or login_wall_detected:
+ # A sign-in page carries no lead of its own when nothing on it matches the intent.
+ login_page_is_the_only_lead = candidate_title_score == 0 and candidate_anchor_score == 0
+ if anti_bot_detected or (login_wall_detected and not login_page_is_the_only_lead):
# Tag walls so the scorer-miss fallback refuses them even when they degrade to no_candidate.
evidence_trail[-1]["wall"] = True
if anti_bot_detected and anti_bot_has_no_candidate_evidence:
@@ -654,13 +688,19 @@ async def _discovery_walk(
confidence=0.0,
failure_reason="anti_bot_wall",
)
- if login_wall_detected:
+ if login_wall_detected and login_page_is_the_only_lead and entrypoint_candidate is None:
+ # A sign-in page is where a credentialed workflow starts, so a URL the user or their
+ # own credential supplied is reported as the candidate rather than refused. Sign-in
+ # pages routinely ship verification widgets, which the run solves; a page carrying a
+ # challenge and nothing else already refused above. A machine-selected candidate falls
+ # through to the binding checks below instead.
+ evidence_trail[-1]["page_kind"] = "login"
return build_result(
- candidate_url=None,
- candidate_form_fields=[],
+ candidate_url=current_url,
+ candidate_form_fields=form_fields,
evidence_trail=evidence_trail,
- confidence=0.0,
- failure_reason="login_wall",
+ confidence=1.0,
+ failure_reason=None,
)
if entrypoint_candidate is not None:
@@ -949,6 +989,12 @@ async def _discover_workflow_entrypoint_impl(
else:
entry_url = _user_provided_entry_url(copilot_ctx)
kind = "user_provided_site" if entry_url else "unresolved"
+ if entry_url is None:
+ # A multi-word site name is not a bare word, so this is the only point a stored login page
+ # is reachable for one.
+ entry_url = _credential_entry_url(copilot_ctx, site_or_url)
+ if entry_url:
+ kind = "credential_tested_url"
if entry_url is None:
result = _discovery_build_result(
candidate_url=None,
diff --git a/skyvern/forge/sdk/copilot/tools/scouting.py b/skyvern/forge/sdk/copilot/tools/scouting.py
index a12f65ce4..427b5ff45 100644
--- a/skyvern/forge/sdk/copilot/tools/scouting.py
+++ b/skyvern/forge/sdk/copilot/tools/scouting.py
@@ -60,7 +60,6 @@ from skyvern.forge.sdk.copilot.composition_evidence import (
from skyvern.forge.sdk.copilot.config import BlockAuthoringPolicy
from skyvern.forge.sdk.copilot.context import FillCarry
from skyvern.forge.sdk.copilot.enforcement import (
- _RECENT_TOOL_OUTPUT_CHAR_CAP,
mint_scout_observation_contract_for_ctx,
record_reached_terminal_action_observation,
record_scouted_output_coverage,
@@ -109,6 +108,14 @@ from .banned_blocks import _copilot_block_authoring_policy
LOG = structlog.get_logger()
+# Emission budget for scout/evaluate tool results: bounds how much live page content
+# a result may carry into the authoring context, so it must not follow the transcript
+# recent-window cap.
+_SCOUT_RESULT_CHAR_CAP = 2000
+# Reconnaissance evaluates keep the raw page payload by design, so they get a larger
+# budget than the steer results; this is the hard ceiling on that channel, not a target.
+_SCOUT_RECON_RESULT_CHAR_CAP = 20_000
+
_FILL_CARRY_RETRYABLE_VALIDATION_FAILURES = frozenset({"page_mismatch", "selector_absent_from_page_evidence"})
@@ -1424,7 +1431,7 @@ def _shed_scout_page_summary_section(summary: dict[str, Any]) -> bool:
def _attach_scout_page_summary(result: dict[str, Any], page_evidence: dict[str, Any]) -> None:
"""Attach a compact page summary at result["data"]["page"], keeping the whole
- serialized result under the recent-output pruner cap by shedding sections —
+ serialized result under the scout result budget by shedding sections —
never by slicing the serialized JSON."""
data = result.get("data")
if not isinstance(data, dict):
@@ -1432,7 +1439,7 @@ def _attach_scout_page_summary(result: dict[str, Any], page_evidence: dict[str,
try:
summary = _build_scout_page_summary(page_evidence)
data["page"] = summary
- while len(json.dumps(result)) > _RECENT_TOOL_OUTPUT_CHAR_CAP:
+ while len(json.dumps(result)) > _SCOUT_RESULT_CHAR_CAP:
if not _shed_scout_page_summary_section(summary):
data.pop("page", None)
return
@@ -1676,18 +1683,33 @@ def _fit_evaluate_steer_under_cap(
*,
keep_raw_page_payload: bool,
) -> None:
- """Keep the serialized result under the recent-output cap without ever head-slicing it.
+ """Keep the serialized result under the scout result budget without ever head-slicing it.
Reconnaissance output needs the raw page payload available to the model; imperative steers have
enough structured evidence to shed bulky non-essential payload while always preserving the action."""
+ def over(limit: int) -> bool:
+ return len(json.dumps(result, default=str)) > limit
+
def over_cap() -> bool:
- return len(json.dumps(result, default=str)) > _RECENT_TOOL_OUTPUT_CHAR_CAP
+ return over(_SCOUT_RESULT_CHAR_CAP)
if not over_cap():
return
if keep_raw_page_payload:
data.pop("actionable_targets", None)
+ nested = data.get("result")
+ if isinstance(nested, dict):
+ for key in _EVALUATE_STEER_NESTED_BULKY_KEYS:
+ if not over(_SCOUT_RECON_RESULT_CHAR_CAP):
+ return
+ if key in nested and nested[key] != _EVALUATE_STEER_SHED_MARKER:
+ nested[key] = _EVALUATE_STEER_SHED_MARKER
+ while over(_SCOUT_RECON_RESULT_CHAR_CAP):
+ largest_key = _largest_non_essential_data_key(data)
+ if largest_key is None:
+ break
+ data[largest_key] = _EVALUATE_STEER_SHED_MARKER
return
nested = data.get("result")
if isinstance(nested, dict):
@@ -1767,7 +1789,7 @@ async def _auto_act_on_repeat(ctx: AgentContext, result: dict[str, Any], *, url:
else:
data["page"] = _build_scout_page_summary(post_evidence)
essential = _auto_act_essential_keys()
- while len(json.dumps(result, default=str)) > _RECENT_TOOL_OUTPUT_CHAR_CAP:
+ while len(json.dumps(result, default=str)) > _SCOUT_RESULT_CHAR_CAP:
largest = max(
(
(key, _serialized_len(value))
diff --git a/skyvern/forge/sdk/copilot/tools/workflow_update.py b/skyvern/forge/sdk/copilot/tools/workflow_update.py
index b23425836..40c60e282 100644
--- a/skyvern/forge/sdk/copilot/tools/workflow_update.py
+++ b/skyvern/forge/sdk/copilot/tools/workflow_update.py
@@ -7,9 +7,11 @@ import hashlib
import io
import json
import keyword
+import os
import re
import textwrap
import tokenize
+import uuid
from collections.abc import Iterable, Iterator, Mapping, Sequence
from dataclasses import dataclass
from dataclasses import field as dataclass_field
@@ -23,6 +25,8 @@ import yaml
from jinja2 import TemplateSyntaxError
from pydantic import AliasChoices, BaseModel, Field, ValidationError
+from skyvern._version import __version__
+from skyvern.config import settings
from skyvern.forge import app
from skyvern.forge.sdk.api.llm.schema_validator import validate_schema
from skyvern.forge.sdk.copilot.attribution import resolve_copilot_created_by_stamp
@@ -8641,10 +8645,66 @@ def _co_computed_metadata_repair_contract(
)
+def _dump_imposition_decision(
+ ctx: AgentContext,
+ *,
+ workflow_yaml: str,
+ result: _SynthesizedCodeImpositionResult,
+) -> None:
+ """Serialize the imposition decision for offline replay; both outcomes are captured because a
+ substitution's generated code is the artifact no log holds. The recorded branch inputs let a
+ replay say whether a divergence came from the decision or from state this capture cannot carry."""
+ directory = os.environ.get("COPILOT_DUMP_IMPOSITION_INPUTS")
+ if not directory or settings.ENV != "local":
+ return
+ try:
+ os.makedirs(directory, mode=0o700, exist_ok=True)
+ changed = result.workflow_yaml != workflow_yaml
+ prior_source, prior_yaml = _prior_yaml_source(ctx)
+ payload = {
+ "schema_version": 2,
+ "source_version": __version__,
+ "workflow_yaml": workflow_yaml,
+ "prior_source": prior_source,
+ "prior_yaml": prior_yaml,
+ "scout_trajectory": list(ctx.scout_trajectory),
+ "flow_evidence": ctx.flow_evidence,
+ "block_authoring_policy": str(_copilot_block_authoring_policy(ctx)),
+ "impose_synthesized_code_block": ctx.impose_synthesized_code_block,
+ "update_workflow_called": ctx.update_workflow_called,
+ "turn_origin": str(ctx.turn_origin),
+ "raw_code_artifact_metadata": ctx.raw_code_artifact_metadata,
+ # The branch inputs as the run left them, read from retained state: deriving the plan here
+ # would latch one the decision never latched and change the next evaluation's reach.
+ "reaches_goal": synthesized_trajectory_reaches_goal(ctx),
+ "extraction_plan_present": ctx.last_bound_requested_output_extraction_plan is not None,
+ "outcome": "substituted" if changed else "kept",
+ "violations": list(result.violations),
+ "result_workflow_yaml": result.workflow_yaml,
+ }
+ target = os.path.join(
+ directory, f"imposition-{'substituted' if changed else 'kept'}-{uuid.uuid4().hex[:8]}.json"
+ )
+ with open(os.open(target, os.O_CREAT | os.O_WRONLY | os.O_TRUNC, 0o600), "w") as handle:
+ json.dump(payload, handle, default=str)
+ except Exception:
+ LOG.info("copilot_imposition_input_dump_failed", exc_info=True)
+
+
def _maybe_impose_synthesized_code_block(
workflow_yaml: str,
ctx: AgentContext,
runtime_parameters: Mapping[str, Any] | None = None,
+) -> _SynthesizedCodeImpositionResult:
+ result = _maybe_impose_synthesized_code_block_decision(workflow_yaml, ctx, runtime_parameters)
+ _dump_imposition_decision(ctx, workflow_yaml=workflow_yaml, result=result)
+ return result
+
+
+def _maybe_impose_synthesized_code_block_decision(
+ workflow_yaml: str,
+ ctx: AgentContext,
+ runtime_parameters: Mapping[str, Any] | None = None,
) -> _SynthesizedCodeImpositionResult:
ctx.pending_goal_complete_landing = False
ctx.pending_requested_output_extraction_candidate = None
@@ -9167,6 +9227,21 @@ def _maybe_impose_synthesized_code_block(
block_label=str(code_block.get("label") or ""),
stage_count=len(durable_stage_codes),
)
+ # Every author-time check above has run and reported; only the code replacement is withheld. A
+ # submission is replaced solely when the machine has something landed to replace it with — a
+ # goal-reaching trajectory or a bound requested-output plan. With neither, substitution is pure
+ # replay of scout probes, and whether the submission is any good is the test-run's question
+ # (ADR-0022).
+ if submitted_code.strip() and not reaches_goal and extraction_plan is None:
+ LOG.info(
+ "copilot_imposition_kept_submitted_code",
+ carrier_label=_code_block_label(code_block),
+ prior_source=prior_source,
+ reaches_goal=reaches_goal,
+ **_submitted_code_fingerprint(submitted_code),
+ )
+ return _SynthesizedCodeImpositionResult(workflow_yaml=workflow_yaml)
+
# A credential fill in the trajectory opens the split on its own: a value-producing subset that
# carries no credential fill reruns on the scout's authenticated session instead of replaying login.
should_split_output_owner = (
diff --git a/skyvern/forge/sdk/routes/browser_sessions.py b/skyvern/forge/sdk/routes/browser_sessions.py
index 2a538531b..00bd70444 100644
--- a/skyvern/forge/sdk/routes/browser_sessions.py
+++ b/skyvern/forge/sdk/routes/browser_sessions.py
@@ -377,6 +377,7 @@ async def get_browser_session(
browser_session,
app.STORAGE,
fail_download_lookup=True,
+ include_stream_transport=True,
)
diff --git a/skyvern/forge/sdk/routes/streaming/screencast.py b/skyvern/forge/sdk/routes/streaming/screencast.py
index 4e8480307..cf87ed016 100644
--- a/skyvern/forge/sdk/routes/streaming/screencast.py
+++ b/skyvern/forge/sdk/routes/streaming/screencast.py
@@ -31,23 +31,44 @@ async def wait_for_browser_state(
poll_interval: float = 0.25,
) -> BrowserState | None:
elapsed = 0.0
- while elapsed < timeout:
- browser_state = await _resolve_browser_state(
- entity_id,
- entity_type,
- workflow_run_id,
- organization_id=organization_id,
- )
+ observed: BrowserState | None = None
+ try:
+ while elapsed < timeout:
+ browser_state = observed or await _resolve_browser_state(
+ entity_id,
+ entity_type,
+ workflow_run_id,
+ organization_id=organization_id,
+ )
- if browser_state is not None:
- page = await browser_state.get_working_page()
- if page is not None:
- return browser_state
+ if browser_state is not None:
+ # An observer connection belongs to this websocket, so it is held across polls
+ # rather than re-dialed while the session's first page comes up.
+ if entity_type == "browser_session":
+ observed = browser_state
+ page = await browser_state.get_working_page()
+ if page is not None:
+ # Ownership passes to the caller, who releases it when the websocket ends.
+ observed = None
+ return browser_state
- await asyncio.sleep(poll_interval)
- elapsed += poll_interval
+ await asyncio.sleep(poll_interval)
+ elapsed += poll_interval
- return None
+ return None
+ finally:
+ # Whatever is still held here was never handed to a caller, so nobody else will give it back.
+ await release_browser_state(observed, entity_type, entity_id)
+
+
+async def release_browser_state(browser_state: BrowserState | None, entity_type: str, entity_id: str) -> None:
+ """Give back whatever watching adopted. Only the manager knows whether that is anything."""
+ if browser_state is None or entity_type != "browser_session":
+ return
+ try:
+ await app.PERSISTENT_SESSIONS_MANAGER.release_observer_browser_state(entity_id, browser_state)
+ except Exception:
+ LOG.warning("Could not release the live-view browser connection", browser_session_id=entity_id, exc_info=True)
async def _resolve_browser_state(
@@ -61,7 +82,7 @@ async def _resolve_browser_state(
if entity_type == "task":
return app.BROWSER_MANAGER.get_for_task(entity_id, workflow_run_id)
if entity_type == "browser_session":
- return await app.PERSISTENT_SESSIONS_MANAGER.get_browser_state(entity_id, organization_id)
+ return await app.PERSISTENT_SESSIONS_MANAGER.get_observer_browser_state(entity_id, organization_id)
return None
@@ -73,6 +94,13 @@ async def _resolve_working_page(
organization_id: str | None = None,
fall_back_to_captured: bool = True,
) -> object | None:
+ # The viewer's own connection outlives every page the session opens, so following the active page
+ # needs no re-resolution here — and re-resolving would dial a second connection every poll.
+ if entity_type == "browser_session":
+ try:
+ return await browser_state.get_working_page()
+ except Exception:
+ return None
# Re-resolve each poll so the screencast follows a BrowserState the run swaps in post-connect (skyvern#6703).
try:
state = await _resolve_browser_state(entity_id, entity_type, workflow_run_id, organization_id)
diff --git a/skyvern/forge/sdk/routes/streaming/screenshot.py b/skyvern/forge/sdk/routes/streaming/screenshot.py
index 6ce17c33d..107f23eee 100644
--- a/skyvern/forge/sdk/routes/streaming/screenshot.py
+++ b/skyvern/forge/sdk/routes/streaming/screenshot.py
@@ -24,7 +24,12 @@ from websockets.exceptions import ConnectionClosedError, ConnectionClosedOK
from skyvern.config import settings
from skyvern.forge import app
from skyvern.forge.sdk.routes.routers import base_router, legacy_base_router
-from skyvern.forge.sdk.routes.streaming.screencast import start_screencast_loop, wait_for_browser_state
+from skyvern.forge.sdk.routes.streaming.screencast import (
+ release_browser_state,
+ start_screencast_loop,
+ wait_for_browser_state,
+)
+from skyvern.forge.sdk.routes.streaming.verify import stream_transport
from skyvern.forge.sdk.schemas.persistent_browser_sessions import is_final_status
from skyvern.forge.sdk.schemas.tasks import TaskStatus
from skyvern.forge.sdk.services.org_auth_service import get_current_org
@@ -65,6 +70,9 @@ async def task_stream(
LOG.info("Started task streaming", task_id=task_id, organization_id=organization_id)
+ # A run streams the frames its own worker publishes (``CDPFramePublisher``), which reaches a
+ # browser this process does not hold — including an externally hosted one. Only a deployment
+ # that drives the browser in-process screencasts it directly.
if settings.BROWSER_STREAMING_MODE == "cdp":
await _local_screencast_for_task(websocket, task_id, organization_id)
return
@@ -336,7 +344,7 @@ async def browser_session_streaming(
organization_id=organization_id,
)
- if settings.BROWSER_STREAMING_MODE == "cdp":
+ if await stream_transport(browser_session_id, organization_id) == "cdp":
await _local_screencast_for_browser_session(websocket, browser_session_id, organization_id)
return
@@ -355,10 +363,14 @@ async def _run_local_screencast(
wait_for_running: Callable[[], Awaitable[str | None]],
check_finalized: Callable[[], Awaitable[bool]],
get_current_status: Callable[[], Awaitable[str | None]],
+ # Required: a browser this process does not own is reachable only through the session that
+ # holds it, and that lookup is organization-scoped. Defaulting this made the remote path
+ # silently unreachable for whichever caller forgot to pass it.
+ organization_id: str,
get_workflow_run_id: Callable[[], str | None] | None = None,
- organization_id: str | None = None,
) -> None:
id_key = f"{entity_type}_id"
+ browser_state = None
try:
early_exit_status = await wait_for_running()
if early_exit_status is not None:
@@ -400,6 +412,8 @@ async def _run_local_screencast(
LOG.warning("WebSocket connection error during local screencast", **{id_key: entity_id})
except Exception:
LOG.warning("Error in local screencast", **{id_key: entity_id}, exc_info=True)
+ finally:
+ await release_browser_state(browser_state, entity_type, entity_id)
async def _local_screencast_for_workflow_run(
@@ -450,6 +464,7 @@ async def _local_screencast_for_workflow_run(
wait_for_running=wait_for_running,
check_finalized=check_finalized,
get_current_status=get_current_status,
+ organization_id=organization_id,
)
@@ -496,6 +511,7 @@ async def _local_screencast_for_task(
wait_for_running=wait_for_running,
check_finalized=check_finalized,
get_current_status=get_current_status,
+ organization_id=organization_id,
get_workflow_run_id=lambda: task_workflow_run_id,
)
diff --git a/skyvern/forge/sdk/routes/streaming/verify.py b/skyvern/forge/sdk/routes/streaming/verify.py
index f61970b89..2a1a9be9b 100644
--- a/skyvern/forge/sdk/routes/streaming/verify.py
+++ b/skyvern/forge/sdk/routes/streaming/verify.py
@@ -91,7 +91,7 @@ async def verify_browser_session(
browser_address = browser_session.browser_address if browser_session.is_browser_ready else None
if not browser_address:
- if settings.BROWSER_STREAMING_MODE == "cdp":
+ if await stream_transport(browser_session_id, organization_id) == "cdp":
browser_address = ""
else:
LOG.info(
@@ -320,3 +320,18 @@ async def loop_verify_workflow_run(verifiable: MessageChannel | VncChannel) -> N
verifiable.browser_session = browser_session
await asyncio.sleep(Constants.POLL_INTERVAL_FOR_VERIFICATION_SECONDS)
+
+
+async def stream_transport(browser_session_id: str | None, organization_id: str) -> str:
+ try:
+ return await app.AGENT_FUNCTION.resolve_stream_transport(
+ browser_session_id=browser_session_id, organization_id=organization_id
+ )
+ except Exception:
+ LOG.warning(
+ "Could not resolve the stream transport; using the deployment default",
+ browser_session_id=browser_session_id,
+ organization_id=organization_id,
+ exc_info=True,
+ )
+ return settings.BROWSER_STREAMING_MODE
diff --git a/skyvern/forge/sdk/services/bitwarden.py b/skyvern/forge/sdk/services/bitwarden.py
index b71583da8..0b98a022d 100644
--- a/skyvern/forge/sdk/services/bitwarden.py
+++ b/skyvern/forge/sdk/services/bitwarden.py
@@ -169,7 +169,7 @@ def get_bitwarden_item_type_code(item_type: BitwardenItemType) -> int:
def get_list_response_item_from_bitwarden_item(item: dict) -> CredentialItem:
if item["type"] == BitwardenItemType.LOGIN:
login = item["login"]
- totp = BitwardenService.normalize_totp_config(login.get("totp", ""))
+ totp = BitwardenService.normalize_totp_config(login.get("totp") or "")
return CredentialItem(
item_id=item["id"],
credential=PasswordCredential(
@@ -1099,7 +1099,7 @@ class BitwardenService:
raise BitwardenGetItemError(f"Failed to get login item by ID: {item_id}")
login = response["data"]["login"]
- totp = BitwardenService.normalize_totp_config(login.get("totp", ""))
+ totp = BitwardenService.normalize_totp_config(login.get("totp") or "")
if not login:
raise BitwardenGetItemError(f"Item with ID: {item_id} is not a login item")
@@ -1412,7 +1412,7 @@ class BitwardenService:
credential=PasswordCredential(
username=login_item["username"] or "",
password=login_item["password"] or "",
- totp=login_item["totp"],
+ totp=BitwardenService.normalize_totp_config(login_item.get("totp") or ""),
metadata=_password_metadata_from_bitwarden_item(response["data"]),
),
login_uris=_whole_site_login_uris(login_item),
diff --git a/skyvern/forge/sdk/workflow/models/_jinja.py b/skyvern/forge/sdk/workflow/models/_jinja.py
index be99f716d..2d3daf2f4 100644
--- a/skyvern/forge/sdk/workflow/models/_jinja.py
+++ b/skyvern/forge/sdk/workflow/models/_jinja.py
@@ -8,7 +8,7 @@ import tokenize
from typing import Any, Callable, cast
import structlog
-from jinja2 import StrictUndefined
+from jinja2 import ChainableUndefined, StrictUndefined
from jinja2.sandbox import SandboxedEnvironment
from skyvern.config import settings
@@ -120,8 +120,37 @@ def render_templates_in_json_value(value: object, render_string: Callable[[str],
return value
-jinja_json_finalize_env = SandboxedEnvironment(finalize=_json_finalize)
-jinja_json_finalize_env.filters["json"] = _json_type_filter
+class RequiredBindingUndefined(ChainableUndefined):
+ """Raises where an undefined is rendered directly into the payload.
+
+ StrictUndefined would also break `{% if %}` guards, `{% for %}` over an absent collection, and
+ `| default(...)` on a producer block that never ran — all legitimate ways to author a conditional
+ payload — so only `__str__` fails. The trade-off is that filters consuming an undefined as a
+ sequence (`join`, `length`) still emit, because raising on iteration would take `{% for %}` with it.
+ """
+
+ __slots__ = ()
+
+ def __str__(self) -> str:
+ self._fail_with_undefined_error()
+ raise AssertionError("unreachable")
+
+
+def _json_finalize_required_binding(value: Any) -> Any:
+ # A producer that returned null said "no value"; rendering it into a quoted JSON slot would write
+ # the text "None". An omitted binding is the loud case and is handled by the undefined type above.
+ if value is None:
+ return ""
+ return _json_finalize(value)
+
+
+# A payload a block sends to an external system cannot distinguish "the producer omitted this" from
+# "the user wanted an empty value" once an undefined reference renders as "". Rendering one under this
+# environment raises instead, and stays strict regardless of WORKFLOW_TEMPLATING_STRICTNESS.
+jinja_json_finalize_required_binding_env = SandboxedEnvironment(
+ undefined=RequiredBindingUndefined, finalize=_json_finalize_required_binding
+)
+jinja_json_finalize_required_binding_env.filters["json"] = _json_type_filter
if settings.WORKFLOW_TEMPLATING_STRICTNESS == "strict":
jinja_json_finalize_strict_env = SandboxedEnvironment(undefined=StrictUndefined, finalize=_json_finalize)
diff --git a/skyvern/forge/sdk/workflow/models/block.py b/skyvern/forge/sdk/workflow/models/block.py
index 1b2e81f45..eda197c8f 100644
--- a/skyvern/forge/sdk/workflow/models/block.py
+++ b/skyvern/forge/sdk/workflow/models/block.py
@@ -736,6 +736,7 @@ class Block(BaseModel, abc.ABC):
*,
force_include_secrets: bool = False,
env: SandboxedEnvironment | None = None,
+ skip_missing_variable_preflight: bool = False,
) -> str:
"""
Format a template string using the workflow run context.
@@ -844,7 +845,9 @@ class Block(BaseModel, abc.ABC):
template_data["workflow_run_outputs"] = workflow_run_context.workflow_run_outputs
template_data["workflow_run_summary"] = workflow_run_context.build_workflow_run_summary()
- if settings.WORKFLOW_TEMPLATING_STRICTNESS == "strict":
+ # A caller whose environment decides for itself what an absent binding means renders instead of
+ # failing here, so `| default(...)` still reaches an undefined the preflight would reject.
+ if settings.WORKFLOW_TEMPLATING_STRICTNESS == "strict" and not skip_missing_variable_preflight:
if missing_variables := get_missing_variables(potential_template, template_data):
raise MissingJinjaVariables(
template=potential_template,
@@ -1503,18 +1506,30 @@ class BaseTaskBlock(Block):
# so that cookie-based authentication can redirect or restore the session
# BEFORE the agent starts interacting with the page.
if workflow_run.browser_profile_id:
- LOG.info(
- "Browser profile loaded — waiting for page to settle before agent acts",
- browser_profile_id=workflow_run.browser_profile_id,
- workflow_run_id=workflow_run.workflow_run_id,
- )
- try:
- await working_page.wait_for_load_state("networkidle", timeout=10000)
- except Exception:
- LOG.debug(
- "networkidle timeout after browser profile load (non-fatal)",
+ applied_profile_id = browser_state.browser_artifacts.applied_browser_profile_id
+ if applied_profile_id != workflow_run.browser_profile_id and not browser_session_id:
+ LOG.warning(
+ "Stamped browser profile was not applied to this browser — continuing without saved state",
+ browser_profile_id=workflow_run.browser_profile_id,
+ applied_browser_profile_id=applied_profile_id,
workflow_run_id=workflow_run.workflow_run_id,
)
+ else:
+ # A persistent session loads its profile session-side, invisible to these
+ # artifacts — keep the settle wait so cookie redirects finish either way.
+ LOG.info(
+ "Browser profile loaded — waiting for page to settle before agent acts",
+ browser_profile_id=workflow_run.browser_profile_id,
+ applied_browser_profile_id=applied_profile_id,
+ workflow_run_id=workflow_run.workflow_run_id,
+ )
+ try:
+ await working_page.wait_for_load_state("networkidle", timeout=10000)
+ except Exception:
+ LOG.debug(
+ "networkidle timeout after browser profile load (non-fatal)",
+ workflow_run_id=workflow_run.workflow_run_id,
+ )
except Exception as e:
LOG.exception(
diff --git a/skyvern/forge/sdk/workflow/models/google_sheets_blocks.py b/skyvern/forge/sdk/workflow/models/google_sheets_blocks.py
index 22c8c4038..b0754eb66 100644
--- a/skyvern/forge/sdk/workflow/models/google_sheets_blocks.py
+++ b/skyvern/forge/sdk/workflow/models/google_sheets_blocks.py
@@ -4,10 +4,14 @@ from dataclasses import dataclass
from typing import Any, Literal, cast
import structlog
+from jinja2 import UndefinedError
from skyvern.forge import app
from skyvern.forge.sdk.workflow.context_manager import WorkflowRunContext
-from skyvern.forge.sdk.workflow.models._jinja import jinja_json_finalize_env
+from skyvern.forge.sdk.workflow.exceptions import FailedToFormatJinjaStyleParameter
+from skyvern.forge.sdk.workflow.models._jinja import (
+ jinja_json_finalize_required_binding_env,
+)
from skyvern.forge.sdk.workflow.models.block import Block
from skyvern.forge.sdk.workflow.models.parameter import PARAMETER_TYPE
from skyvern.schemas.google_sheets import (
@@ -413,9 +417,26 @@ class GoogleSheetsWriteBlock(Block):
self.credential_id, workflow_run_context
)
if self.values:
- self.values = self.format_block_parameter_template_from_workflow_run_context(
- self.values, workflow_run_context, env=jinja_json_finalize_env
+ self.values = self._render_values_or_raise(workflow_run_context)
+
+ def _render_values_or_raise(self, workflow_run_context: WorkflowRunContext) -> str:
+ """A reference no upstream block answered would otherwise render as "" and be appended as a
+ blank cell the Sheets API reports as a successful write. `| default(...)` passes an empty."""
+ try:
+ return self.format_block_parameter_template_from_workflow_run_context(
+ self.values,
+ workflow_run_context,
+ env=jinja_json_finalize_required_binding_env,
+ skip_missing_variable_preflight=True,
)
+ except FailedToFormatJinjaStyleParameter as exc:
+ if not isinstance(exc.__cause__, UndefinedError):
+ raise
+ raise ValueError(
+ f"block `{self.label}` field `values` references a value no upstream block produced: "
+ f"{exc.__cause__}. Return that key from the producing block, or write an explicit "
+ "default (e.g. {{ block_label.field | default('') }}) if an empty cell is intended."
+ ) from exc
def _coerce_values(self, raw: Any, *, column_offset: int = 0, absolute_columns: bool = False) -> list[list[Any]]:
"""Rows for the write payload.
diff --git a/skyvern/forge/sdk/workflow/service.py b/skyvern/forge/sdk/workflow/service.py
index 3f9418ab1..733603573 100644
--- a/skyvern/forge/sdk/workflow/service.py
+++ b/skyvern/forge/sdk/workflow/service.py
@@ -45,6 +45,7 @@ from skyvern.exceptions import (
BlockedHost,
BlockNotFound,
BrowserActionPolicyNotEnforceable,
+ BrowserProfileNotApplied,
BrowserProfileNotFound,
BrowserSessionNotFound,
BrowserSessionNotRenewable,
@@ -228,7 +229,11 @@ from skyvern.utils.url_validators import validate_url as validate_url_with_block
from skyvern.webeye.actions.action_types import ActionType
from skyvern.webeye.browser_state import BrowserState
from skyvern.webeye.profile_cookie_merge import cookie_delta, seed_cookie_values, union_cookies_into_profile_dir
-from skyvern.webeye.session_cookies import persist_session_cookies, read_persisted_session_cookies
+from skyvern.webeye.session_cookies import (
+ persist_session_cookies,
+ read_persisted_session_cookies,
+ refresh_banked_cookies,
+)
LOG = structlog.get_logger()
@@ -5672,14 +5677,28 @@ class WorkflowService:
url=login_url,
)
# A prior block may already have opened this run's browser; get_or_create_for_workflow_run
- # then returns that cached context and ignores browser_profile_id, so the credential
- # profile never loads. Stamping the run credential-seeded in that case would let the
- # healthy-run bank whole-dir the unrelated context into the shared credential profile.
- # Only credential-seed when we will actually load the profile into a fresh browser;
- # otherwise degrade to fresh — the credential seed holds only when its
- # profile loaded.
- browser_already_open = app.BROWSER_MANAGER.get_for_workflow_run(workflow_run_id) is not None
- if browser_already_open:
+ # then returns that cached context and ignores browser_profile_id. Stamping the run
+ # credential-seeded when that context did NOT boot from this profile would let the
+ # healthy-run bank whole-dir an unrelated context into the shared credential profile.
+ # Credential-seed only when the profile is genuinely in the browser: either we load it
+ # into a fresh browser below, or the open browser already booted from it.
+ browser_state_open = app.BROWSER_MANAGER.get_for_workflow_run(workflow_run_id)
+ browser_already_open = browser_state_open is not None
+ # A script/setup boot applies the setup-stamped seed itself; when the open browser was
+ # booted from exactly the resolved profile, the credential seed is genuinely loaded.
+ open_browser_booted_this_profile = (
+ browser_state_open is not None
+ and browser_state_open.browser_artifacts.applied_browser_profile_id == resolved_browser_profile_id
+ )
+ if browser_already_open and open_browser_booted_this_profile:
+ LOG.info(
+ "Credential login block reached with a browser already booted from the resolved "
+ "profile — keeping the credential seed",
+ workflow_run_id=workflow_run_id,
+ block_label=block.label,
+ browser_profile_id=resolved_browser_profile_id,
+ )
+ if browser_already_open and not open_browser_booted_this_profile:
LOG.warning(
"Credential login block reached with a browser already open; running fresh login "
"instead of credential-seeding to avoid banking an unrelated context",
@@ -5721,7 +5740,7 @@ class WorkflowService:
# Create the browser with the saved profile and navigate to the login block's URL. The
# user enters the post-login target URL; the saved cookies authenticate once it loads.
- profile_loaded = bool(login_url) and not browser_already_open
+ profile_loaded = open_browser_booted_this_profile or (bool(login_url) and not browser_already_open)
if login_url and not browser_already_open:
try:
browser_state = await app.BROWSER_MANAGER.get_or_create_for_workflow_run(
@@ -5730,6 +5749,14 @@ class WorkflowService:
browser_profile_id=resolved_browser_profile_id,
browser_session_id=decision.attach_browser_session_id,
)
+ if (
+ not decision.attach_browser_session_id
+ and browser_state.browser_artifacts.applied_browser_profile_id
+ != resolved_browser_profile_id
+ ):
+ # The chosen browser creator (e.g. a remote/vendor browser) accepted the
+ # profile id but never loaded it; the run must not stay credential-stamped.
+ raise BrowserProfileNotApplied(resolved_browser_profile_id)
working_page = await browser_state.get_working_page()
if working_page and working_page.url == "about:blank":
await browser_state.navigate_to_url(page=working_page, url=login_url)
@@ -6434,7 +6461,7 @@ class WorkflowService:
]
if patched_branches != block.branch_conditions:
block = block.model_copy(update={"branch_conditions": patched_branches})
- elif block.next_block_label == finally_block_label:
+ if block.next_block_label == finally_block_label:
block = block.model_copy(update={"next_block_label": None})
result.append(block)
return result
@@ -9119,6 +9146,20 @@ class WorkflowService:
browser_profile_id=sink_profile_id,
)
return
+ if (
+ workflow_run.browser_profile_id
+ and browser_state.browser_artifacts.applied_browser_profile_id != workflow_run.browser_profile_id
+ ):
+ # The stamped seed was never applied (e.g. a vendor-routed boot) — this directory does not
+ # extend the sink's accumulated state, so writing it back would erase the saved archive.
+ # Seedless accumulate rows (browser_profile_id None, sink set) still write by design.
+ LOG.warning(
+ "Skipped persisting sink browser profile — stamped seed profile was not applied to this browser",
+ workflow_run_id=workflow_run.workflow_run_id,
+ browser_profile_id=sink_profile_id,
+ seed_browser_profile_id=workflow_run.browser_profile_id,
+ )
+ return
if effective_workflow_run_status != WorkflowRunStatus.completed:
LOG.info(
"Skipped persisting browser sink profile for non-completed workflow run",
@@ -9176,6 +9217,20 @@ class WorkflowService:
browser_profile_id=sink_profile_id,
)
return
+ try:
+ # The seed archive's banked-cookie sidecar rides this directory into the write; refresh it
+ # from the live jar, or drop it, so a later boot can't replay seed-era cookies over fresher.
+ await refresh_banked_cookies(
+ None if close_browser_on_completion else browser_state.browser_context,
+ browser_state.browser_artifacts.browser_session_dir,
+ )
+ except Exception:
+ LOG.warning(
+ "Failed to refresh the banked-cookies sidecar before the sink write-back",
+ workflow_run_id=workflow_run.workflow_run_id,
+ browser_profile_id=sink_profile_id,
+ exc_info=True,
+ )
await app.STORAGE.store_browser_profile(
workflow_run.organization_id,
profile_id=sink_profile_id,
diff --git a/skyvern/webeye/browser_artifacts.py b/skyvern/webeye/browser_artifacts.py
index 33ccfca93..db67ff363 100644
--- a/skyvern/webeye/browser_artifacts.py
+++ b/skyvern/webeye/browser_artifacts.py
@@ -30,6 +30,10 @@ class BrowserArtifacts(BaseModel):
needs_cdp_frame_publisher: bool = False
# Optional opaque identifier for a remote browser session.
remote_browser_session_id: str | None = None
+ # The saved browser profile the creator actually applied to this browser's user-data-dir.
+ # None when none was requested or the chosen creator could not load one (remote/vendor browsers,
+ # storage miss, corruption fallback) — consumers must treat None as "profile not applied".
+ applied_browser_profile_id: str | None = None
_browser_console_log_lock: asyncio.Lock = PrivateAttr(default_factory=asyncio.Lock)
# Tombstoned synchronously before any await, so set_popup_video_listener can't
# re-register a page's video after RealBrowserState decides to discard it.
diff --git a/skyvern/webeye/browser_factory.py b/skyvern/webeye/browser_factory.py
index 05e1510ba..39bada4fb 100644
--- a/skyvern/webeye/browser_factory.py
+++ b/skyvern/webeye/browser_factory.py
@@ -680,6 +680,17 @@ class BrowserContextFactory:
if not creator:
raise UnknownBrowserType(browser_type)
browser_context, browser_artifacts, cleanup_func = await creator(playwright, **creator_kwargs)
+ requested_profile_id = cast(str | None, kwargs.get("browser_profile_id"))
+ if requested_profile_id and browser_artifacts.applied_browser_profile_id != requested_profile_id:
+ LOG.warning(
+ "Browser profile was requested but not applied by the chosen browser creator — "
+ "run continues without the saved profile",
+ browser_profile_id=requested_profile_id,
+ browser_type=browser_type,
+ workflow_run_id=kwargs.get("workflow_run_id"),
+ task_id=kwargs.get("task_id"),
+ organization_id=kwargs.get("organization_id"),
+ )
await restore_session_cookies(browser_context, browser_artifacts.browser_session_dir)
# After session cookies so a verified-login heal (banked by the credential living-profile
# engine) wins over the profile's own older session cookies on a key clash. Gated on the
@@ -878,6 +889,8 @@ async def _create_headless_chromium(
har_path=browser_args["record_har_path"],
browser_session_dir=user_data_dir,
)
+ if loaded_from_saved_profile:
+ browser_artifacts.applied_browser_profile_id = browser_profile_id
try:
browser_context = await playwright.chromium.launch_persistent_context(**browser_args)
except Exception as launch_error:
@@ -971,6 +984,8 @@ async def _create_headful_chromium(
har_path=browser_args["record_har_path"],
browser_session_dir=user_data_dir,
)
+ if loaded_from_saved_profile:
+ browser_artifacts.applied_browser_profile_id = browser_profile_id
try:
browser_context = await playwright.chromium.launch_persistent_context(**browser_args)
except Exception as launch_error:
diff --git a/skyvern/webeye/default_persistent_sessions_manager.py b/skyvern/webeye/default_persistent_sessions_manager.py
index f67f8f112..5f1d3fc86 100644
--- a/skyvern/webeye/default_persistent_sessions_manager.py
+++ b/skyvern/webeye/default_persistent_sessions_manager.py
@@ -313,6 +313,16 @@ class DefaultPersistentSessionsManager(PersistentSessionsManager):
browser_session = self._browser_sessions.get(session_id)
return browser_session.browser_state if browser_session else None
+ async def get_observer_browser_state(
+ self, session_id: str, organization_id: str | None = None
+ ) -> BrowserState | None:
+ """The state this process already drives, so watching it adopts nothing."""
+ return await self.get_browser_state(session_id, organization_id)
+
+ async def release_observer_browser_state(self, session_id: str, browser_state: BrowserState) -> None:
+ """Nothing was adopted: this process drives the browser, and the run still needs it."""
+ return None
+
async def set_browser_state(
self, session_id: str, browser_state: BrowserState, organization_id: str | None = None
) -> None:
diff --git a/skyvern/webeye/persistent_sessions_manager.py b/skyvern/webeye/persistent_sessions_manager.py
index 858d3252a..4cd2bd09a 100644
--- a/skyvern/webeye/persistent_sessions_manager.py
+++ b/skyvern/webeye/persistent_sessions_manager.py
@@ -92,6 +92,22 @@ class PersistentSessionsManager(Protocol):
"""Get the browser state for a session."""
...
+ async def get_observer_browser_state(
+ self, session_id: str, organization_id: str | None = None
+ ) -> BrowserState | None:
+ """Get the browser state for a session to watch it, never to drive it.
+
+ A watcher must not arm request interception on a browser a run is driving, and must not
+ publish what it connects into any cache a run could pick up. Pair every call with
+ ``release_observer_browser_state`` — whether that releases anything is the implementation's
+ to know, since a manager that returns a state it already drives must not tear it down.
+ """
+ ...
+
+ async def release_observer_browser_state(self, session_id: str, browser_state: BrowserState) -> None:
+ """Release what ``get_observer_browser_state`` handed out. Never closes the remote browser."""
+ ...
+
async def set_browser_state(
self, session_id: str, browser_state: BrowserState, organization_id: str | None = None
) -> None:
diff --git a/skyvern/webeye/real_browser_manager.py b/skyvern/webeye/real_browser_manager.py
index 4d788ea68..89e1698cb 100644
--- a/skyvern/webeye/real_browser_manager.py
+++ b/skyvern/webeye/real_browser_manager.py
@@ -934,6 +934,7 @@ class RealBrowserManager(BrowserManager):
"Creating browser state for workflow run",
sampling=True,
workflow_run_id=workflow_run.workflow_run_id,
+ browser_profile_id=browser_profile_id or "none",
)
if browser_session_id and workflow_run.organization_id:
session = await app.PERSISTENT_SESSIONS_MANAGER.get_session(
diff --git a/skyvern/webeye/schemas.py b/skyvern/webeye/schemas.py
index b88cb6094..89a9d6409 100644
--- a/skyvern/webeye/schemas.py
+++ b/skyvern/webeye/schemas.py
@@ -74,6 +74,11 @@ class BrowserSessionResponse(BaseModel):
description="Whether this session's browser profile will be saved when it ends so it can become a reusable browser profile.",
)
vnc_streaming_supported: bool = Field(False, description="Whether the browser session supports VNC streaming")
+ stream_transport: str | None = Field(
+ None,
+ description='Live-view transport for this session: "vnc" or "cdp". Resolved on the single-session fetch only; null elsewhere.',
+ examples=["vnc", "cdp"],
+ )
download_path: str | None = Field(None, description="The path where the browser session downloads files")
downloaded_files: list[FileInfo] | None = Field(
None, description="The list of files downloaded by the browser session"
@@ -96,6 +101,10 @@ class BrowserSessionResponse(BaseModel):
# False deliberately preserves the existing permissive timeout behavior for PATCH,
# active-list, and history responses; the single-session GET opts into strict lookup.
fail_download_lookup: bool = False,
+ # Resolving the transport costs a per-session infrastructure lookup, and the list
+ # endpoints serialize an unpaginated set concurrently. Only the single-session fetch —
+ # the one live view actually reads — pays for it.
+ include_stream_transport: bool = False,
) -> BrowserSessionResponse:
"""
Creates a BrowserSessionResponse from a PersistentBrowserSession object.
@@ -165,6 +174,18 @@ class BrowserSessionResponse(BaseModel):
upstream_cdp_url=browser_session.upstream_cdp_url,
)
+ stream_transport: str | None = None
+ if include_stream_transport:
+ # The response contract admits exactly two transport words; anything else a resolver
+ # produces is withheld rather than serialized to clients.
+ stream_transport = await app.AGENT_FUNCTION.resolve_stream_transport(
+ browser_session_id=browser_session.persistent_browser_session_id,
+ organization_id=browser_session.organization_id,
+ ip_address=browser_session.ip_address,
+ )
+ if stream_transport not in ("vnc", "cdp"):
+ stream_transport = None
+
return cls(
browser_session_id=browser_session.persistent_browser_session_id,
organization_id=browser_session.organization_id,
@@ -178,6 +199,7 @@ class BrowserSessionResponse(BaseModel):
browser_session.persistent_browser_session_id,
ip_address=browser_session.ip_address,
),
+ stream_transport=stream_transport,
app_url=app_url,
started_at=browser_session.started_at,
completed_at=browser_session.completed_at,
diff --git a/skyvern/webeye/session_cookies.py b/skyvern/webeye/session_cookies.py
index c08c45580..979dca5e1 100644
--- a/skyvern/webeye/session_cookies.py
+++ b/skyvern/webeye/session_cookies.py
@@ -7,7 +7,12 @@ import os
import structlog
from playwright.async_api import BrowserContext
-from skyvern.webeye.profile_cookie_merge import _ALLOWED_COOKIE_KEYS, BANKED_COOKIES_FILENAME
+from skyvern.webeye.profile_cookie_merge import (
+ _ALLOWED_COOKIE_KEYS,
+ BANKED_COOKIES_FILENAME,
+ clear_banked_cookies,
+ union_cookies_into_profile_dir,
+)
LOG = structlog.get_logger()
@@ -143,3 +148,24 @@ async def restore_banked_cookies(browser_context: BrowserContext | None, user_da
await _add_cookies_with_fallback(browser_context, sanitized, "banked")
except Exception:
LOG.warning("Failed to restore banked cookies", exc_info=True)
+
+
+async def refresh_banked_cookies(browser_context: BrowserContext | None, user_data_dir: str | None) -> None:
+ """Replace the banked-cookies sidecar in ``user_data_dir`` with the live context's cookie jar.
+
+ Chromium commits its Cookies database on a ~30s timer, so a directory archived from a running browser
+ can hold a database that predates the last sign-in and the sidecar carries what it is missing. Without
+ a live context the sidecar is only dropped: the browser has flushed its database, and a sidecar
+ inherited from the seed archive would otherwise replay stale cookies over fresher state at every boot.
+ """
+ if not user_data_dir or not os.path.isdir(user_data_dir):
+ return
+ cookies: list[dict] = []
+ if browser_context is not None:
+ try:
+ cookies = await browser_context.cookies()
+ except Exception as exc:
+ LOG.info("Live cookie jar unavailable; dropping the banked cookies", error_type=type(exc).__name__)
+ clear_banked_cookies(user_data_dir)
+ if cookies:
+ union_cookies_into_profile_dir(cookies, user_data_dir)
diff --git a/tests/smoke_tests/browser_extension/run_extension_smoke.py b/tests/smoke_tests/browser_extension/run_extension_smoke.py
index b5d7940a1..10b2ec24a 100644
--- a/tests/smoke_tests/browser_extension/run_extension_smoke.py
+++ b/tests/smoke_tests/browser_extension/run_extension_smoke.py
@@ -4,7 +4,6 @@ import argparse
import asyncio
import base64
import hashlib
-import inspect
import json
import os
import re
@@ -17,15 +16,16 @@ from pathlib import Path
from urllib.parse import urlsplit
from aiohttp import web
+from fastmcp import Client
from playwright.async_api import Browser, BrowserContext, Page, Playwright, async_playwright
from skyvern.browser_extension.protocol import EXTENSION_ID
from skyvern.browser_extension.runtime import BrowserExtensionRuntime
+from tests.evals.mcp.task import build_mcp_stdio_transport, unwrap_tool_result
EXPECTED_EXTENSION_ID = EXTENSION_ID
TOKEN_ENV = "SKYVERN_BROWSER_EXTENSION_TOKEN"
-BROWSER_TYPE_ENV = "BROWSER_TYPE"
-MCP_BROWSER_TYPE = "extension-connect"
+PORT_ENV = "SKYVERN_BROWSER_EXTENSION_PORT"
SMOKE_TEXT = "Skyvern extension bridge smoke"
CHECKS = (
"setup",
@@ -33,21 +33,17 @@ CHECKS = (
"navigate",
"screenshot",
"trusted selector click",
- "coordinate click",
- "type",
"iframe evaluate",
"new tab",
"close tab",
"chrome survives disconnect",
"mcp session/navigate/observe",
+ "mcp coordinate click",
+ "mcp coordinate type",
)
CDP_URL_PATTERN = re.compile(r"ws://127\.0\.0\.1:\d+/cdp/[A-Za-z0-9_-]+")
-class McpUnavailableError(RuntimeError):
- pass
-
-
class SmokeReport:
def __init__(self) -> None:
self.passed = 0
@@ -131,7 +127,8 @@ def _safe_text(value: object, redactions: Sequence[str]) -> str:
for secret_value in redactions:
if secret_value:
text = text.replace(secret_value, "<redacted>")
- return " ".join(text.split())[:600]
+ text = " ".join(text.split())
+ return text if len(text) <= 600 else f"{text[:300]} ... {text[-295:]}"
async def _discover_extension_id(context: BrowserContext, timeout_seconds: float = 15.0) -> str:
@@ -193,13 +190,21 @@ async def _start_fixture_servers() -> FixtureServers:
async def outer_handler(_request: web.Request) -> web.Response:
html = f"""<!doctype html><html><head><meta charset="utf-8">
-<title>Extension Bridge Fixture</title></head><body>
+<title>Extension Bridge Fixture</title><style>
+#smoke-button {{ position: fixed; left: 40px; top: 40px; width: 160px; height: 40px; }}
+#text-input {{ position: fixed; left: 40px; top: 120px; width: 240px; height: 32px; }}
+#click-result {{ position: fixed; left: 40px; top: 180px; }}
+#password-input {{ position: fixed; left: 40px; top: 220px; }}
+#smoke-frame {{ position: fixed; left: 320px; top: 40px; width: 320px; height: 180px; }}
+</style></head><body>
<button id="smoke-button" type="button">Smoke button</button>
<input id="text-input" type="text"><input id="password-input" type="password">
+<div id="click-result">not-clicked</div>
<iframe id="smoke-frame" src="{iframe_url}"></iframe><script>
window.smokeState = {{clickCount: 0, lastTrusted: false}};
document.querySelector("#smoke-button").addEventListener("click", (event) => {{
window.smokeState.clickCount += 1; window.smokeState.lastTrusted = event.isTrusted;
+ document.querySelector("#click-result").textContent = `clicked:${{window.smokeState.clickCount}}`;
}});</script></body></html>"""
return web.Response(text=html, content_type="text/html", headers={"Cache-Control": "no-store"})
@@ -211,7 +216,7 @@ document.querySelector("#smoke-button").addEventListener("click", (event) => {{
return FixtureServers(
outer_runner=outer_runner,
iframe_runner=iframe_runner,
- outer_url=f"http://localhost:{outer_port}/",
+ outer_url=f"http://127.0.0.1:{outer_port}/",
)
@@ -224,44 +229,106 @@ def _tool_error(result: dict[str, object]) -> str:
return f"{code}: {message}"
-async def _run_mcp_phase(fixture_url: str) -> str:
- from skyvern.cli.mcp_tools import browser as browser_tools
- from skyvern.cli.mcp_tools import session as session_tools
+async def _wait_for_popup_connection(popup_page: Page, timeout_seconds: float = 15.0) -> None:
+ loop = asyncio.get_running_loop()
+ deadline = loop.time() + timeout_seconds
+ while loop.time() < deadline:
+ status = (await popup_page.locator("#status-label").text_content() or "").strip()
+ if status in {"Connected", "Client attached"}:
+ return
+ await asyncio.sleep(0.1)
+ error = await popup_page.locator("#connection-error").text_content()
+ raise TimeoutError(f"extension did not reconnect to MCP relay; popup error: {error or 'none'}")
+
+
+async def _wait_for_fixture_page(context: BrowserContext, fixture_url: str, timeout_seconds: float = 15.0) -> Page:
+ loop = asyncio.get_running_loop()
+ deadline = loop.time() + timeout_seconds
+ while loop.time() < deadline:
+ matches = [page for page in context.pages if not page.is_closed() and page.url == fixture_url]
+ if matches:
+ return matches[-1]
+ await asyncio.sleep(0.1)
+ raise TimeoutError(f"MCP navigation did not expose fixture page {fixture_url}")
- if MCP_BROWSER_TYPE not in inspect.getsource(session_tools):
- raise McpUnavailableError("the current MCP session layer has no extension-connect branch")
- previous_browser_type = os.environ.get(BROWSER_TYPE_ENV)
- os.environ[BROWSER_TYPE_ENV] = MCP_BROWSER_TYPE
+async def _run_mcp_phase(
+ context: BrowserContext,
+ popup_page: Page,
+ fixture_url: str,
+ relay_port: int,
+ pairing_token: str,
+ report: SmokeReport,
+) -> None:
+ transport = build_mcp_stdio_transport(
+ browser_extension=True,
+ env_overrides={
+ PORT_ENV: str(relay_port),
+ TOKEN_ENV: pairing_token,
+ # Keep the production SSRF default intact; only this disposable child may reach its loopback fixture.
+ "ALLOWED_HOSTS": json.dumps(["127.0.0.1"]),
+ },
+ )
session_created = False
- try:
- session_result = await session_tools.skyvern_browser_session_create()
- if session_result.get("ok") is not True:
- raise McpUnavailableError(f"session=FAIL; unavailable in-process ({_tool_error(session_result)})")
- session_created = True
-
- navigate_result = await browser_tools.skyvern_navigate(url=fixture_url)
- observe_result = await browser_tools.skyvern_observe(selector="body", interactive_only=True)
- statuses = (
- f"session={'PASS' if session_result.get('ok') is True else 'FAIL'} "
- f"navigate={'PASS' if navigate_result.get('ok') is True else 'FAIL'} "
- f"observe={'PASS' if observe_result.get('ok') is True else 'FAIL'}"
- )
- if navigate_result.get("ok") is not True:
- raise AssertionError(f"{statuses}; navigate error={_tool_error(navigate_result)}")
- if observe_result.get("ok") is not True:
- raise AssertionError(f"{statuses}; observe error={_tool_error(observe_result)}")
- return statuses
- finally:
- if session_created:
- try:
- await session_tools.skyvern_browser_session_close()
- except Exception:
- pass
- if previous_browser_type is None:
- os.environ.pop(BROWSER_TYPE_ENV, None)
- else:
- os.environ[BROWSER_TYPE_ENV] = previous_browser_type
+ async with Client(transport, timeout=60) as client:
+ await _wait_for_popup_connection(popup_page)
+
+ async def call_tool(name: str, arguments: dict[str, object] | None = None) -> dict[str, object]:
+ return unwrap_tool_result(await client.call_tool_mcp(name, arguments or {}))
+
+ try:
+ session_result = await call_tool("skyvern_browser_session_create")
+ if session_result.get("ok") is not True:
+ raise AssertionError(f"session create failed: {_tool_error(session_result)}")
+ session_created = True
+ assert pairing_token not in repr(session_result)
+ assert CDP_URL_PATTERN.search(repr(session_result)) is None
+
+ navigate_result = await call_tool("skyvern_navigate", {"url": fixture_url})
+ observe_result = await call_tool("skyvern_observe", {"selector": "body", "interactive_only": True})
+ statuses = (
+ f"session={'PASS' if session_result.get('ok') is True else 'FAIL'} "
+ f"navigate={'PASS' if navigate_result.get('ok') is True else 'FAIL'} "
+ f"observe={'PASS' if observe_result.get('ok') is True else 'FAIL'}"
+ )
+ if navigate_result.get("ok") is not True:
+ raise AssertionError(f"{statuses}; navigate error={_tool_error(navigate_result)}")
+ if observe_result.get("ok") is not True:
+ raise AssertionError(f"{statuses}; observe error={_tool_error(observe_result)}")
+ report.pass_check("mcp session/navigate/observe", statuses)
+
+ fixture_page = await _wait_for_fixture_page(context, fixture_url)
+ button_bounds = await fixture_page.locator("#smoke-button").bounding_box()
+ assert button_bounds is not None, "MCP fixture button has no bounding box"
+ click_x = button_bounds["x"] + button_bounds["width"] / 2
+ click_y = button_bounds["y"] + button_bounds["height"] / 2
+ click_result = await call_tool("skyvern_click", {"x": click_x, "y": click_y})
+ click_data = click_result.get("data")
+ assert click_result.get("ok") is True, _tool_error(click_result)
+ assert isinstance(click_data, dict)
+ assert click_data.get("x") == click_x
+ assert click_data.get("y") == click_y
+ assert click_data.get("resolved_target")
+ assert await fixture_page.locator("#click-result").text_content() == "clicked:1"
+ report.pass_check("mcp coordinate click", f"x={click_x:.1f}, y={click_y:.1f}")
+
+ input_bounds = await fixture_page.locator("#text-input").bounding_box()
+ assert input_bounds is not None, "MCP fixture input has no bounding box"
+ type_x = input_bounds["x"] + input_bounds["width"] / 2
+ type_y = input_bounds["y"] + input_bounds["height"] / 2
+ type_result = await call_tool("skyvern_type", {"x": type_x, "y": type_y, "text": SMOKE_TEXT})
+ type_data = type_result.get("data")
+ assert type_result.get("ok") is True, _tool_error(type_result)
+ assert isinstance(type_data, dict)
+ assert type_data.get("x") == type_x
+ assert type_data.get("y") == type_y
+ assert type_data.get("resolved_target")
+ assert await fixture_page.locator("#text-input").input_value() == SMOKE_TEXT
+ report.pass_check("mcp coordinate type", f"x={type_x:.1f}, y={type_y:.1f}")
+ finally:
+ if session_created:
+ close_result = await call_tool("skyvern_browser_session_close")
+ assert close_result.get("ok") is True, _tool_error(close_result)
async def _pair_extension(
@@ -286,15 +353,8 @@ async def _pair_extension(
async def _exercise_page(page: Page, report: SmokeReport, screenshot_path: Path) -> FixtureServers:
servers = await _start_fixture_servers()
try:
- try:
- await page.goto("https://example.com", wait_until="domcontentloaded")
- title = await page.title()
- assert "Example" in title, f"unexpected example.com title: {title!r}"
- except Exception:
- await page.goto(servers.outer_url, wait_until="domcontentloaded")
- report.pass_check("navigate", "loopback fallback")
- else:
- report.pass_check("navigate", f"title={title!r}")
+ await page.goto(servers.outer_url, wait_until="domcontentloaded")
+ report.pass_check("navigate", "loopback fixture")
await page.screenshot(path=screenshot_path)
assert screenshot_path.is_file() and screenshot_path.stat().st_size > 0
@@ -308,24 +368,7 @@ async def _exercise_page(page: Page, report: SmokeReport, screenshot_path: Path)
assert selector_state.get("lastTrusted") is True
report.pass_check("trusted selector click")
- button = page.locator("#smoke-button")
- bounds = await button.bounding_box()
- assert bounds is not None, "button has no bounding box"
- x = bounds["x"] + bounds["width"] / 2
- y = bounds["y"] + bounds["height"] / 2
- await page.mouse.click(x, y)
- coordinate_state = await page.evaluate("() => ({...window.smokeState})")
- assert isinstance(coordinate_state, dict)
- assert coordinate_state.get("clickCount") == 2
- assert coordinate_state.get("lastTrusted") is True
- report.pass_check("coordinate click", f"x={x:.1f}, y={y:.1f}")
-
- text_input = page.locator("#text-input")
- await text_input.focus()
- await page.keyboard.type(SMOKE_TEXT)
- assert await text_input.input_value() == SMOKE_TEXT
assert await page.locator("#password-input").count() == 1
- report.pass_check("type")
iframe_element = await page.wait_for_selector("#smoke-frame")
frame = await iframe_element.content_frame()
@@ -350,6 +393,7 @@ async def _cleanup(
runtime: BrowserExtensionRuntime | None,
playwright: Playwright | None,
profile_dir: tempfile.TemporaryDirectory[str] | None,
+ evidence_dir: tempfile.TemporaryDirectory[str] | None,
) -> list[str]:
errors: list[str] = []
cleanup_steps: list[tuple[str, Callable[[], Awaitable[object]]]] = []
@@ -368,11 +412,12 @@ async def _cleanup(
await cleanup_step()
except Exception as exc:
errors.append(f"{label}: {type(exc).__name__}: {exc}")
- if profile_dir is not None:
- try:
- profile_dir.cleanup()
- except Exception as exc:
- errors.append(f"temporary profile: {type(exc).__name__}: {exc}")
+ for label, temp_dir in (("temporary profile", profile_dir), ("temporary evidence", evidence_dir)):
+ if temp_dir is not None:
+ try:
+ temp_dir.cleanup()
+ except Exception as exc:
+ errors.append(f"{label}: {type(exc).__name__}: {exc}")
return errors
@@ -388,8 +433,8 @@ async def run_smoke() -> int:
bridge_browser: Browser | None = None
fixture_servers: FixtureServers | None = None
profile_dir: tempfile.TemporaryDirectory[str] | None = None
- screenshot_dir = Path(tempfile.mkdtemp(prefix="skyvern-extension-smoke-evidence-"))
- screenshot_path = (screenshot_dir / "smoke_navigate.png").resolve()
+ evidence_dir = tempfile.TemporaryDirectory(prefix="skyvern-extension-smoke-evidence-")
+ screenshot_path = (Path(evidence_dir.name) / "smoke_navigate.png").resolve()
try:
port_reservation, relay_port = _reserve_free_port()
@@ -421,7 +466,7 @@ async def run_smoke() -> int:
extension_id = discovered_extension_id
report.pass_check("setup", f"relay port={relay_port}, extension ID={extension_id}")
- await _pair_extension(persistent_context, runtime, extension_id, relay_port, pairing_token)
+ popup_page = await _pair_extension(persistent_context, runtime, extension_id, relay_port, pairing_token)
assert runtime.extension_connected
report.pass_check("extension paired")
@@ -449,12 +494,16 @@ async def run_smoke() -> int:
assert runtime.extension_connected
report.pass_check("chrome survives disconnect")
- try:
- mcp_statuses = await _run_mcp_phase(fixture_servers.outer_url)
- except McpUnavailableError as exc:
- report.skip_check("mcp session/navigate/observe", _safe_text(exc, redactions))
- else:
- report.pass_check("mcp session/navigate/observe", mcp_statuses)
+ await runtime.shutdown()
+ runtime = None
+ await _run_mcp_phase(
+ persistent_context,
+ popup_page,
+ fixture_servers.outer_url,
+ relay_port,
+ pairing_token,
+ report,
+ )
except Exception as exc:
failed_check = next((label for label in CHECKS if label not in report.recorded), "smoke")
report.fail_check(failed_check, _safe_text(f"{type(exc).__name__}: {exc}", redactions))
@@ -467,6 +516,7 @@ async def run_smoke() -> int:
runtime,
playwright,
profile_dir,
+ evidence_dir,
)
if cleanup_errors:
report.fail_check("cleanup", _safe_text("; ".join(cleanup_errors), redactions))
diff --git a/tests/unit/browser_extension/test_mcp_glue.py b/tests/unit/browser_extension/test_mcp_glue.py
index 58d29b905..7bee08b02 100644
--- a/tests/unit/browser_extension/test_mcp_glue.py
+++ b/tests/unit/browser_extension/test_mcp_glue.py
@@ -577,6 +577,50 @@ def test_run_mcp_browser_extension_flag_starts_and_stops_runtime(monkeypatch: py
runtime.shutdown.assert_awaited_once_with()
+async def test_run_mcp_with_cleanup_serves_when_extension_start_fails(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ message = "browser extension bridge unavailable"
+ get_or_start = AsyncMock(side_effect=BrowserExtensionError(message))
+ run_async = AsyncMock()
+ cleanup = AsyncMock()
+ warning = MagicMock()
+ instance = MagicMock(return_value=None)
+ monkeypatch.setattr(BrowserExtensionRuntime, "get_or_start", get_or_start)
+ monkeypatch.setattr(BrowserExtensionRuntime, "instance", instance)
+ monkeypatch.setattr(run_commands, "_cleanup_mcp_resources", cleanup)
+ monkeypatch.setattr(run_commands.LOG, "warning", warning)
+
+ await run_commands._run_mcp_with_cleanup(run_async, browser_extension=True, transport="stdio")
+
+ get_or_start.assert_awaited_once_with()
+ run_async.assert_awaited_once_with(transport="stdio")
+ cleanup.assert_awaited_once_with()
+ instance.assert_called_once_with()
+ assert warning.call_count == 1
+ assert warning.call_args.kwargs["error"] == message
+
+
+async def test_run_mcp_with_cleanup_propagates_unexpected_extension_start_error(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ get_or_start = AsyncMock(side_effect=RuntimeError("unexpected startup failure"))
+ run_async = AsyncMock()
+ cleanup = AsyncMock()
+ monkeypatch.setattr(BrowserExtensionRuntime, "get_or_start", get_or_start)
+ monkeypatch.setattr(BrowserExtensionRuntime, "instance", MagicMock(return_value=None))
+ monkeypatch.setattr(run_commands, "_cleanup_mcp_resources", cleanup)
+
+ with pytest.raises(RuntimeError, match="unexpected startup failure"):
+ await run_commands._run_mcp_with_cleanup(run_async, browser_extension=True)
+
+ get_or_start.assert_awaited_once_with()
+ run_async.assert_not_awaited()
+ cleanup.assert_awaited_once_with()
+
+
@pytest.mark.parametrize("transport", ["sse", "streamable-http"])
def test_run_mcp_rejects_browser_extension_with_http_transport(
monkeypatch: pytest.MonkeyPatch,
diff --git a/tests/unit/browser_extension/test_runtime.py b/tests/unit/browser_extension/test_runtime.py
index 44170ced3..f5024e96a 100644
--- a/tests/unit/browser_extension/test_runtime.py
+++ b/tests/unit/browser_extension/test_runtime.py
@@ -159,10 +159,13 @@ async def test_port_resolution_prefers_explicit_then_environment_then_default(mo
async def test_port_in_use_has_actionable_browser_extension_error(monkeypatch: pytest.MonkeyPatch) -> None:
_, adapters, _ = install_stubs(monkeypatch, relay_start_error=OSError(errno.EADDRINUSE, "address in use"))
- with pytest.raises(BrowserExtensionError, match="SKYVERN_BROWSER_EXTENSION_PORT") as error_info:
+ with pytest.raises(BrowserExtensionError) as error_info:
await BrowserExtensionRuntime.get_or_start(23001)
- assert "23001" in str(error_info.value)
+ message = str(error_info.value)
+ assert "23001" in message
+ assert "SKYVERN_BROWSER_EXTENSION_PORT" in message
+ assert "extension popup" in message
assert adapters[0].stop_count == 1
assert BrowserExtensionRuntime.instance() is None
diff --git a/tests/unit/forge/sdk/routes/streaming/test_stream_transport.py b/tests/unit/forge/sdk/routes/streaming/test_stream_transport.py
new file mode 100644
index 000000000..0136249d0
--- /dev/null
+++ b/tests/unit/forge/sdk/routes/streaming/test_stream_transport.py
@@ -0,0 +1,57 @@
+"""Tests for per-session stream-transport resolution (SKY-13291)."""
+
+from __future__ import annotations
+
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+from skyvern.forge.sdk.routes.streaming import verify
+
+
+class TestStreamTransport:
+ @pytest.mark.asyncio
+ async def test_delegates_to_agent_function(self) -> None:
+ with patch("skyvern.forge.sdk.routes.streaming.verify.app") as app_mock:
+ app_mock.AGENT_FUNCTION.resolve_stream_transport = AsyncMock(return_value="cdp")
+
+ assert await verify.stream_transport("pbs_1", "o_1") == "cdp"
+
+ app_mock.AGENT_FUNCTION.resolve_stream_transport.assert_awaited_once_with(
+ browser_session_id="pbs_1", organization_id="o_1"
+ )
+
+ @pytest.mark.asyncio
+ async def test_falls_back_to_setting_on_error(self) -> None:
+ with (
+ patch("skyvern.forge.sdk.routes.streaming.verify.app") as app_mock,
+ patch("skyvern.forge.sdk.routes.streaming.verify.settings") as settings_mock,
+ ):
+ app_mock.AGENT_FUNCTION.resolve_stream_transport = AsyncMock(side_effect=RuntimeError("db down"))
+ settings_mock.BROWSER_STREAMING_MODE = "vnc"
+
+ assert await verify.stream_transport("pbs_1", "o_1") == "vnc"
+
+ @pytest.mark.asyncio
+ async def test_base_agent_function_returns_deployment_setting(self) -> None:
+ from skyvern.forge.agent_functions import AgentFunction
+
+ with patch("skyvern.forge.agent_functions.settings") as settings_mock:
+ settings_mock.BROWSER_STREAMING_MODE = "cdp"
+
+ transport = await AgentFunction().resolve_stream_transport(browser_session_id=None, organization_id=None)
+
+ assert transport == "cdp"
+
+ @pytest.mark.asyncio
+ async def test_base_agent_function_ignores_the_pod_address(self) -> None:
+ from skyvern.forge.agent_functions import AgentFunction
+
+ with patch("skyvern.forge.agent_functions.settings") as settings_mock:
+ settings_mock.BROWSER_STREAMING_MODE = "cdp"
+
+ transport = await AgentFunction().resolve_stream_transport(
+ browser_session_id="pbs_1", organization_id="o_1", ip_address="10.0.0.1"
+ )
+
+ assert transport == "cdp"
diff --git a/tests/unit/forge/sdk/routes/streaming/test_verify_browser_session.py b/tests/unit/forge/sdk/routes/streaming/test_verify_browser_session.py
index 1f06045de..51c4a15b1 100644
--- a/tests/unit/forge/sdk/routes/streaming/test_verify_browser_session.py
+++ b/tests/unit/forge/sdk/routes/streaming/test_verify_browser_session.py
@@ -209,3 +209,31 @@ class TestVerifyBrowserSessionLocalShortCircuit:
result = await verify_browser_session("bs_test", "o_test")
assert result is None
+
+
+class TestVerifyBrowserSessionPerSessionTransport:
+ @pytest.mark.asyncio
+ async def test_cdp_transport_bypasses_missing_address_when_global_is_vnc(self) -> None:
+ session = _make_session(
+ browser_address=None,
+ status=PersistentBrowserSessionStatus.running,
+ )
+
+ manager = MagicMock()
+ manager.get_session = AsyncMock(return_value=session)
+ manager.get_browser_address = AsyncMock(
+ side_effect=AssertionError("must not wait for address when the session's transport is cdp"),
+ )
+
+ with (
+ patch("skyvern.forge.sdk.routes.streaming.verify.app") as app_mock,
+ patch("skyvern.forge.sdk.routes.streaming.verify.settings") as settings_mock,
+ ):
+ app_mock.PERSISTENT_SESSIONS_MANAGER = manager
+ app_mock.AGENT_FUNCTION.resolve_stream_transport = AsyncMock(return_value="cdp")
+ settings_mock.BROWSER_STREAMING_MODE = "vnc"
+ result = await verify_browser_session("bs_test", "o_test")
+
+ assert result is not None
+ assert result.browser_address == ""
+ manager.get_browser_address.assert_not_called()
diff --git a/tests/unit/forge/sdk/routes/test_browser_sessions.py b/tests/unit/forge/sdk/routes/test_browser_sessions.py
index 8d398ad1e..7d7649805 100644
--- a/tests/unit/forge/sdk/routes/test_browser_sessions.py
+++ b/tests/unit/forge/sdk/routes/test_browser_sessions.py
@@ -77,4 +77,5 @@ async def test_get_browser_session_enables_strict_download_lookup() -> None:
browser_session,
app_mock.STORAGE,
fail_download_lookup=True,
+ include_stream_transport=True,
)
diff --git a/tests/unit/test_bitwarden_login.py b/tests/unit/test_bitwarden_login.py
index ba4bfc160..b190aafd0 100644
--- a/tests/unit/test_bitwarden_login.py
+++ b/tests/unit/test_bitwarden_login.py
@@ -6,11 +6,19 @@ from skyvern.forge.sdk.schemas.credentials import PasswordCredential
from skyvern.forge.sdk.services import bitwarden as bitwarden_module
from skyvern.forge.sdk.services.bitwarden import (
BITWARDEN_CUSTOM_FIELD_TYPE_HIDDEN,
+ BitwardenItemType,
BitwardenService,
RunCommandResult,
get_list_response_item_from_bitwarden_item,
)
+# The vault server omits `totp` entirely for a login saved without a two-factor secret,
+# and returns it as JSON null for some hand-made items.
+MISSING_TOTP_LOGINS = [
+ pytest.param({"username": "[email protected]", "password": "pw"}, id="totp-absent"),
+ pytest.param({"username": "[email protected]", "password": "pw", "totp": None}, id="totp-null"),
+]
+
@pytest.mark.asyncio
async def test_login_ignores_data_file_creation_notice_on_stderr(
@@ -69,3 +77,39 @@ async def test_server_login_item_round_trips_metadata(monkeypatch: pytest.Monkey
]
assert listed_item.credential.metadata == metadata
assert fetched_item.credential.metadata == metadata
+
+
[email protected]("login", MISSING_TOTP_LOGINS)
+def test_list_response_item_reads_login_without_totp(login: dict) -> None:
+ item = {"id": "item-1", "name": "Login", "type": BitwardenItemType.LOGIN, "login": login}
+
+ listed_item = get_list_response_item_from_bitwarden_item(item)
+
+ assert listed_item.credential.totp == ""
+
+
[email protected]("login", MISSING_TOTP_LOGINS)
+async def test_get_login_item_by_id_reads_login_without_totp(monkeypatch: pytest.MonkeyPatch, login: dict) -> None:
+ get_json = AsyncMock(return_value={"success": True, "data": {"login": login}})
+ monkeypatch.setattr(bitwarden_module, "aiohttp_get_json", get_json)
+
+ credential = await BitwardenService._get_login_item_by_id_using_server("item-1")
+
+ assert credential.totp == ""
+
+
[email protected]("login", MISSING_TOTP_LOGINS)
+async def test_get_credential_item_by_id_reads_login_without_totp(monkeypatch: pytest.MonkeyPatch, login: dict) -> None:
+ get_json = AsyncMock(
+ return_value={
+ "success": True,
+ "data": {"id": "item-1", "name": "Login", "type": BitwardenItemType.LOGIN, "login": login},
+ }
+ )
+ monkeypatch.setattr(bitwarden_module, "aiohttp_get_json", get_json)
+
+ fetched_item = await BitwardenService._get_credential_item_by_id_using_server("item-1")
+
+ assert fetched_item.credential.totp == ""
diff --git a/tests/unit/test_browser_context_factory_marker.py b/tests/unit/test_browser_context_factory_marker.py
index 04b6bb1a3..f27dfc7b5 100644
--- a/tests/unit/test_browser_context_factory_marker.py
+++ b/tests/unit/test_browser_context_factory_marker.py
@@ -104,3 +104,105 @@ async def test_ordinary_local_creator_leaves_marker_false(monkeypatch: pytest.Mo
_, artifacts, _ = await BrowserContextFactory.create_browser_context(playwright=object())
assert artifacts.needs_cdp_frame_publisher is False
+
+
+def _factory_harness(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(factory_module, "restore_session_cookies", AsyncMock())
+ monkeypatch.setattr(factory_module, "restore_banked_cookies", AsyncMock())
+ monkeypatch.setattr(factory_module, "set_browser_console_log", lambda **_: None)
+ monkeypatch.setattr(factory_module, "set_popup_video_listener", lambda **_: None)
+ monkeypatch.setattr(factory_module, "set_download_file_listener", lambda **_: None)
+ monkeypatch.setattr(factory_module, "set_dialog_handler", lambda **_: None)
+
+ class _FakeAgentFunction:
+ async def setup_browser_context_extensions(self, **_: Any) -> None:
+ return None
+
+ async def should_apply_banked_cookies(self, organization_id: str | None) -> bool:
+ return False
+
+ class _FakeApp:
+ AGENT_FUNCTION = _FakeAgentFunction()
+
+ monkeypatch.setattr(factory_module, "app", _FakeApp())
+
+
+async def test_factory_warns_when_requested_profile_not_applied(monkeypatch: pytest.MonkeyPatch) -> None:
+ from skyvern.webeye.browser_artifacts import BrowserArtifacts
+ from skyvern.webeye.browser_factory import BrowserContextFactory
+
+ async def _profile_blind_creator(playwright: Any, **kwargs: Any) -> tuple[Any, BrowserArtifacts, None]:
+ # Mirrors remote/vendor creators: accepts browser_profile_id but never applies it.
+ return object(), BrowserArtifacts(), None
+
+ _factory_harness(monkeypatch)
+ log = MagicMock()
+ monkeypatch.setattr(factory_module, "LOG", log)
+
+ BrowserContextFactory.register_type("test-profile-blind", _profile_blind_creator)
+ monkeypatch.setattr(factory_module.settings, "BROWSER_TYPE", "test-profile-blind")
+
+ await BrowserContextFactory.create_browser_context(
+ playwright=object(), browser_profile_id="bp_x", organization_id="o_x"
+ )
+
+ assert any("not applied" in str(call.args[0]) for call in log.warning.call_args_list)
+
+
+async def test_factory_stays_quiet_when_requested_profile_applied(monkeypatch: pytest.MonkeyPatch) -> None:
+ from skyvern.webeye.browser_artifacts import BrowserArtifacts
+ from skyvern.webeye.browser_factory import BrowserContextFactory
+
+ async def _profile_applying_creator(playwright: Any, **kwargs: Any) -> tuple[Any, BrowserArtifacts, None]:
+ return object(), BrowserArtifacts(applied_browser_profile_id=str(kwargs.get("browser_profile_id"))), None
+
+ _factory_harness(monkeypatch)
+ log = MagicMock()
+ monkeypatch.setattr(factory_module, "LOG", log)
+
+ BrowserContextFactory.register_type("test-profile-applying", _profile_applying_creator)
+ monkeypatch.setattr(factory_module.settings, "BROWSER_TYPE", "test-profile-applying")
+
+ await BrowserContextFactory.create_browser_context(
+ playwright=object(), browser_profile_id="bp_x", organization_id="o_x"
+ )
+
+ assert not any("not applied" in str(call.args[0]) for call in log.warning.call_args_list)
+
+
+async def test_headless_chromium_stamps_applied_browser_profile_id(
+ monkeypatch: pytest.MonkeyPatch, tmp_path: Any
+) -> None:
+ from skyvern.forge import app
+ from skyvern.webeye.browser_factory import BrowserContextFactory
+
+ monkeypatch.setattr(app.STORAGE, "retrieve_browser_profile", AsyncMock(return_value=str(tmp_path / "profile")))
+ monkeypatch.setattr(BrowserContextFactory, "update_chromium_browser_preferences", MagicMock())
+ monkeypatch.setattr(
+ BrowserContextFactory,
+ "build_browser_args",
+ MagicMock(return_value={"record_har_path": str(tmp_path / "h.har")}),
+ )
+ monkeypatch.setattr(factory_module, "initialize_download_dir", lambda: str(tmp_path / "downloads"))
+ playwright = MagicMock()
+ playwright.chromium.launch_persistent_context = AsyncMock(return_value=MagicMock())
+
+ _, artifacts, _ = await factory_module._create_headless_chromium(
+ playwright,
+ browser_profile_id="bp_test",
+ organization_id="o_test",
+ )
+ assert artifacts.applied_browser_profile_id == "bp_test"
+
+ # Storage miss: the creator falls back to a temp dir and the field stays None.
+ monkeypatch.setattr(app.STORAGE, "retrieve_browser_profile", AsyncMock(return_value=None))
+ monkeypatch.setattr(factory_module, "make_temp_directory", lambda **_: str(tmp_path / "fresh"))
+ _, artifacts_no_profile, _ = await factory_module._create_headless_chromium(
+ playwright,
+ browser_profile_id="bp_test",
+ organization_id="o_test",
+ )
+ assert artifacts_no_profile.applied_browser_profile_id is None
diff --git a/tests/unit/test_copilot_agent_helpers.py b/tests/unit/test_copilot_agent_helpers.py
index 680fc73dd..8ee5821fd 100644
--- a/tests/unit/test_copilot_agent_helpers.py
+++ b/tests/unit/test_copilot_agent_helpers.py
@@ -4214,7 +4214,12 @@ workflow_definition:
monkeypatch.setattr(
policy_module.app,
"DATABASE",
- SimpleNamespace(credentials=SimpleNamespace(get_credentials_by_ids=AsyncMock(return_value=[]))),
+ SimpleNamespace(
+ credentials=SimpleNamespace(
+ get_credentials_by_ids=AsyncMock(return_value=[]),
+ get_credentials=AsyncMock(return_value=[]),
+ )
+ ),
)
captured: dict[str, str] = {}
@@ -4452,12 +4457,12 @@ workflow_definition:
handler=handler,
)
+ # Names are read by the model, so a turn with no classifier resolves none; the turn still
+ # proceeds and the credential stays reachable by ID or from the Credentials UI.
assert policy.classifier_status == "fallback"
assert policy.classifier_failure_kind == "provider_error"
assert policy.completion_contract_status == "present"
- assert policy.credential_input_kind == "credential_name"
- assert policy.credential_refs == ["mock-portal-login"]
- assert policy.resolved_credentials == [credential]
+ assert policy.resolved_credentials == []
@pytest.mark.asyncio
async def test_fallback_policy_does_not_substring_match_saved_credential_name(self, monkeypatch) -> None:
@@ -4484,7 +4489,7 @@ workflow_definition:
assert policy.completion_contract_status == "present"
assert policy.credential_input_kind == "none"
assert policy.credential_refs == []
- credentials.get_credentials.assert_not_called()
+ assert policy.resolved_credentials == []
@pytest.mark.asyncio
async def test_request_policy_url_matching_and_skip_draft(self, monkeypatch) -> None:
diff --git a/tests/unit/test_copilot_code_block_persist_seam.py b/tests/unit/test_copilot_code_block_persist_seam.py
index ba3ba28d5..2d48d0e8a 100644
--- a/tests/unit/test_copilot_code_block_persist_seam.py
+++ b/tests/unit/test_copilot_code_block_persist_seam.py
@@ -5541,6 +5541,26 @@ def _imposition_split_ctx() -> CopilotContext:
return ctx
+def _reaching_imposition_split_ctx() -> CopilotContext:
+ """The split fixture whose scouted pair carries indices, so it reads as an opening click and its commit."""
+ ctx = _imposition_split_ctx()
+ ctx.scout_trajectory = [
+ {
+ "tool_name": "click",
+ "selector": "#stage-a",
+ "source_url": "https://example.com/records",
+ "trajectory_index": 0,
+ },
+ {
+ "tool_name": "click",
+ "selector": "#stage-b",
+ "source_url": "https://example.com/records",
+ "trajectory_index": 1,
+ },
+ ]
+ return ctx
+
+
class TestSeparatedSpineImpositionRunEligibility:
def test_imposition_accepts_scouted_browser_stage_siblings(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(workflow_update_module, "synthesize_code_block", lambda *a, **k: _fake_spine_synthesized())
@@ -5805,7 +5825,15 @@ class TestCompiledAuthoringImposition:
"role": "textbox",
"accessible_name": "Provider Name",
"trajectory_index": 0,
- }
+ },
+ {
+ "tool_name": "click",
+ "selector": "#provider-search-submit",
+ "source_url": "https://example.com/find-care",
+ "role": "button",
+ "accessible_name": "Search",
+ "trajectory_index": 1,
+ },
]
return ctx
@@ -5865,7 +5893,7 @@ class TestCompiledAuthoringImposition:
]
assert result["data"]["imposed_substitutions"] == {
"block_label": "search_registry",
- "source_trajectory_count": 1,
+ "source_trajectory_count": 2,
"parameter_keys": ["provider_name"],
"credential_parameter_keys": [],
"selector_provenance": [
@@ -5874,7 +5902,13 @@ class TestCompiledAuthoringImposition:
"selector": "#provInput",
"emitted_literal": "#provInput",
"source": "selector",
- }
+ },
+ {
+ "trajectory_index": 1,
+ "selector": "#provider-search-submit",
+ "emitted_literal": "#provider-search-submit",
+ "source": "selector",
+ },
],
"prior_source": "workflow_yaml",
}
@@ -6571,7 +6605,15 @@ class TestCompiledAuthoringImposition:
"role": "textbox",
"accessible_name": "Address or postal code",
"trajectory_index": 0,
- }
+ },
+ {
+ "tool_name": "click",
+ "selector": "#directory-search-submit",
+ "source_url": "https://example.com/find-care",
+ "role": "button",
+ "accessible_name": "Search",
+ "trajectory_index": 1,
+ },
]
submitted = _yaml(
"""
@@ -6635,6 +6677,14 @@ class TestCompiledAuthoringImposition:
"accessible_name": "Provider First Name",
"trajectory_index": 1,
},
+ {
+ "tool_name": "click",
+ "selector": "#directory-search-submit",
+ "source_url": "https://example.com/find-care",
+ "role": "button",
+ "accessible_name": "Search",
+ "trajectory_index": 2,
+ },
]
submitted = _yaml(
"""
@@ -9229,7 +9279,14 @@ class TestBareDropSupersession:
def test_imposition_two_refiners_forgive_two_bare_drops(self) -> None:
ctx = _resale_ctx()
ctx.scout_trajectory = [
- {"tool_name": "click", "selector": "#start", "source_url": _RESALE_URL, "trajectory_index": 0},
+ {
+ "tool_name": "type_text",
+ "selector": "#order-id",
+ "source_url": _RESALE_URL,
+ "typed_length": 6,
+ "typed_value": "abc123",
+ "trajectory_index": 0,
+ },
{"tool_name": "click", "selector": "button", "source_url": _RESALE_URL, "trajectory_index": 1},
{"tool_name": "click", "selector": "button", "source_url": _RESALE_URL, "trajectory_index": 2},
{
@@ -9253,7 +9310,7 @@ class TestBareDropSupersession:
- block_type: code
label: order_status
code: |
- await page.locator("#start").click()
+ await page.locator("#order-id").fill(str(order_id))
await page.locator("button[data-action=\\"open\\"]").click()
await page.locator("button[data-action=\\"status\\"]").click()
"""
@@ -10618,7 +10675,7 @@ class TestScoutedSpinePersistSeamCoverage:
"synthesize_code_block",
lambda *a, **k: _fake_spine_synthesized(diagnostics=diagnostics),
)
- ctx = _imposition_split_ctx()
+ ctx = _reaching_imposition_split_ctx()
workflow_yaml = _records_block_yaml(
_SPINE_SYNTH_CODE
+ '\nvalue = await page.locator("#result").inner_text()\nreturn {"output": {"record_id": value}}'
@@ -14411,3 +14468,88 @@ def test_a_goal_path_stays_bare_when_the_block_returns_a_bare_mapping() -> None:
)
assert bare == {"records"}
+
+
+class TestUnlandedReplayNeverReplacesSubmission:
+ """SKY-13624: with nothing landed (no goal-reaching trajectory, no bound plan), the agent's
+ submitted code persists byte-equal; trajectory replay may only materialize an empty stub."""
+
+ def _probe_only_ctx(self) -> CopilotContext:
+ ctx = _code_only_ctx()
+ _enable_imposition(ctx)
+ ctx.scout_trajectory = [
+ {
+ "tool_name": "read_value",
+ "read_expression": "document.body.innerText.slice(0, 12000)",
+ "read_output_path": "output.scouted_read",
+ "read_output_path_source": "elimination",
+ "read_result_shape": "list",
+ "source_url": "https://example.com/dashboard",
+ "trajectory_index": 0,
+ }
+ ]
+ return ctx
+
+ def _submitted_program_yaml(self) -> str:
+ return _yaml(
+ """
+ title: Append visitors
+ workflow_definition:
+ blocks:
+ - block_type: code
+ label: collect_visitors
+ code: |
+ await page.goto("https://example.com/dashboard", wait_until="domcontentloaded")
+ card_text = await page.locator("body").inner_text()
+ visitor_count = card_text.split()[0]
+ return {"visitor_count": visitor_count}
+ """
+ )
+
+ def test_unlanded_replay_keeps_submitted_code_byte_equal(self) -> None:
+ ctx = self._probe_only_ctx()
+ submitted = self._submitted_program_yaml()
+
+ result = workflow_update_module._maybe_impose_synthesized_code_block(submitted, ctx)
+
+ assert result.violations == []
+ assert result.workflow_yaml == submitted
+
+ def test_unlanded_replay_keeps_submission_even_when_prior_version_differs(self) -> None:
+ ctx = self._probe_only_ctx()
+ ctx.last_workflow_yaml = _yaml(
+ """
+ title: Append visitors
+ workflow_definition:
+ blocks:
+ - block_type: code
+ label: collect_visitors
+ code: |
+ return {"visitor_count": "8.08K"}
+ """
+ )
+ submitted = self._submitted_program_yaml()
+
+ result = workflow_update_module._maybe_impose_synthesized_code_block(submitted, ctx)
+
+ assert result.violations == []
+ assert result.workflow_yaml == submitted
+
+ def test_empty_stub_carrier_is_still_materialized(self) -> None:
+ ctx = _code_only_ctx()
+ _enable_imposition(ctx)
+ submitted = _yaml(
+ """
+ title: Append visitors
+ workflow_definition:
+ blocks:
+ - block_type: code
+ label: collect_visitors
+ code: ""
+ """
+ )
+
+ result = workflow_update_module._maybe_impose_synthesized_code_block(submitted, ctx)
+
+ code = str(_single_code_block(parse_workflow_yaml(result.workflow_yaml))["code"])
+ assert code.strip()
diff --git a/tests/unit/test_copilot_code_block_synthesis.py b/tests/unit/test_copilot_code_block_synthesis.py
index facdfe46d..9bb93906b 100644
--- a/tests/unit/test_copilot_code_block_synthesis.py
+++ b/tests/unit/test_copilot_code_block_synthesis.py
@@ -51,6 +51,7 @@ from skyvern.forge.sdk.copilot.code_block_synthesis import (
LOCATOR_WITNESS_PARAM_SOURCE,
SCOUTED_SPINE_DROPPED_UNFORGIVEN_REASON_CODE,
SCOUTED_SPINE_TRUNCATED_REASON_CODE,
+ SYNTHESIZED_OFFER_SENTINEL,
TRUNCATED_FINDING,
UNFORGIVEN_DROP_FINDING,
ProducedStaticReturnEnvelope,
@@ -2555,7 +2556,7 @@ class TestRenderSynthesizedOfferText:
)
assert synthesized is not None
text = render_synthesized_offer_text(synthesized)
- assert text.startswith("SYNTHESIZED CODE BLOCK (offered once).")
+ assert text.startswith(SYNTHESIZED_OFFER_SENTINEL)
assert _SYNTHESIZED_BLOCK_LABEL in text
assert "```python" in text
assert 'await page.locator("#searcher_s").fill(str(search))' in text
diff --git a/tests/unit/test_copilot_diagnosis_repair_contract.py b/tests/unit/test_copilot_diagnosis_repair_contract.py
index 58f9d1bc5..bc88c4abe 100644
--- a/tests/unit/test_copilot_diagnosis_repair_contract.py
+++ b/tests/unit/test_copilot_diagnosis_repair_contract.py
@@ -2441,3 +2441,40 @@ def test_user_code_error_still_repairs_through_the_contract() -> None:
assert contract.diagnosis_result.suspected_failure_type != DiagnosisFailureType.UNRECOVERABLE_TOOL_ERROR
assert contract.repair_decision.next_action != RepairNextAction.STOP
+
+
+def test_sheets_missing_binding_failure_arms_repair_on_the_failed_block() -> None:
+ # SKY-13624 B2: the strict-render failure is run evidence that arms repair targeting the failed
+ # block, never an evidence-free fresh-build route.
+ run_result = {
+ "ok": False,
+ "error": "Run failed.",
+ "data": {
+ "workflow_run_id": "wr_failed",
+ "overall_status": "failed",
+ "blocks": [
+ {"label": "collect_visitors", "status": "completed"},
+ {
+ "label": "append_visitors_to_sheet",
+ "status": "failed",
+ "failure_reason": (
+ "Failed to format jinja template: block `append_visitors_to_sheet` field `values` "
+ "references a value no upstream block produced: 'dict object' has no attribute "
+ "'visitor_count'. Return that key from the producing block, or write an explicit "
+ "default (e.g. {{ block_label.field | default('') }}) if an empty cell is intended."
+ ),
+ },
+ ],
+ },
+ }
+
+ contract = build_diagnosis_repair_contract(
+ source_tool="update_and_run_blocks",
+ result=run_result,
+ ctx=_ctx(),
+ workflow_updated=True,
+ )
+
+ assert contract.repair_decision.next_action == RepairNextAction.REPAIR
+ assert contract.repair_decision.target_blocks == ["append_visitors_to_sheet"]
+ assert contract.diagnosis_input.failed_block_labels == ["append_visitors_to_sheet"]
diff --git a/tests/unit/test_copilot_discovery.py b/tests/unit/test_copilot_discovery.py
index 0daa198d9..8661ef565 100644
--- a/tests/unit/test_copilot_discovery.py
+++ b/tests/unit/test_copilot_discovery.py
@@ -17,7 +17,11 @@ from skyvern.forge.sdk.copilot.tools import (
_rank_discovery_entrypoint_candidates,
_resolve_discovery_entry_url,
)
-from skyvern.forge.sdk.copilot.tools.discovery import _discovery_build_result, _user_provided_entry_url
+from skyvern.forge.sdk.copilot.tools.discovery import (
+ _credential_entry_url,
+ _discovery_build_result,
+ _user_provided_entry_url,
+)
from skyvern.forge.sdk.copilot.turn_origin import TurnOrigin
from skyvern.forge.sdk.copilot.verification_evidence import WorkflowVerificationEvidence
@@ -816,3 +820,56 @@ class TestUserProvidedEntryUrl:
def test_nothing_the_user_provided_resolves_nothing(self) -> None:
assert _user_provided_entry_url(self._ctx("no addresses here")) is None
+
+
+class TestCredentialEntryUrl:
+ """The org already recorded where a credential signs in, so a site named in words whose
+ credential carries a login page is opened rather than answered with a request for its URL."""
+
+ @staticmethod
+ def _ctx(*tested_urls: str | None) -> SimpleNamespace:
+ policy = RequestPolicy()
+ policy.resolved_credentials = [
+ SimpleNamespace(credential_id=f"cred_{index}", name=f"credential {index}", tested_url=tested_url)
+ for index, tested_url in enumerate(tested_urls)
+ ]
+ return SimpleNamespace(request_policy=policy)
+
+ def test_the_credential_naming_the_requested_site_is_opened(self) -> None:
+ ctx = self._ctx("https://apps.hydroco.example/portal/Login.jsp")
+
+ assert _credential_entry_url(ctx, "hydroco") == "https://apps.hydroco.example/portal/Login.jsp"
+
+ def test_a_multi_word_site_name_resolves_its_credential(self) -> None:
+ ctx = self._ctx("https://apps.guelphhydro.example/portal/Login.jsp")
+
+ assert _credential_entry_url(ctx, "guelph hydro") == "https://apps.guelphhydro.example/portal/Login.jsp"
+
+ def test_the_credential_whose_host_names_the_site_wins_among_several(self) -> None:
+ ctx = self._ctx("https://us5.other.example.net/account/login", "https://us.chosen.example/login")
+
+ assert _credential_entry_url(ctx, "chosen") == "https://us.chosen.example/login"
+
+ def test_a_lone_credential_for_another_site_resolves_nothing(self) -> None:
+ """Approvals persist across turns, so a later request for a different site must not open this one."""
+ ctx = self._ctx("https://payroll.example.net/login")
+
+ assert _credential_entry_url(ctx, "zephyrmart") is None
+
+ def test_a_label_extending_the_requested_name_resolves(self) -> None:
+ ctx = self._ctx("https://us5.metricsdog.example/account/login")
+
+ assert _credential_entry_url(ctx, "metrics") == "https://us5.metricsdog.example/account/login"
+
+ def test_a_name_buried_inside_a_label_resolves_nothing(self) -> None:
+ ctx = self._ctx("https://groupsupport.example.net/login")
+
+ assert _credential_entry_url(ctx, "ups") is None
+
+ def test_several_unrelated_credentials_resolve_nothing(self) -> None:
+ ctx = self._ctx("https://a.example.net/login", "https://b.example.org/login")
+
+ assert _credential_entry_url(ctx, "unrelated") is None
+
+ def test_credentials_without_a_login_page_resolve_nothing(self) -> None:
+ assert _credential_entry_url(self._ctx(None), "example") is None
diff --git a/tests/unit/test_copilot_enforcement_pruning.py b/tests/unit/test_copilot_enforcement_pruning.py
index 3587d91a0..d265736ac 100644
--- a/tests/unit/test_copilot_enforcement_pruning.py
+++ b/tests/unit/test_copilot_enforcement_pruning.py
@@ -41,6 +41,7 @@ from skyvern.forge.sdk.copilot.config import (
)
from skyvern.forge.sdk.copilot.context import CodeAuthoringRepairContext
from skyvern.forge.sdk.copilot.enforcement import (
+ _RECENT_TOOL_OUTPUT_CHAR_CAP,
KEEP_RECENT_TOOL_OUTPUTS,
TOTAL_TIMEOUT_SECONDS,
_mark_copilot_total_timeout,
@@ -1640,6 +1641,35 @@ def test_recent_outputs_preserved_full() -> None:
assert pruned[i]["output"] == short
+def test_recent_code_sized_output_survives_untruncated() -> None:
+ # A code-bearing result in the recent window must reach the model whole; the cap
+ # is a pathological-payload tripwire, never a ration on legitimate code payloads.
+ code_sized = json.dumps({"ok": True, "data": {"code": "await page.click()\n" * 400}})
+ assert 2000 < len(code_sized) < _RECENT_TOOL_OUTPUT_CHAR_CAP
+ items = [_fco("c0", code_sized)]
+
+ pruned = _prune_input_list(items)
+ assert pruned[0]["output"] == code_sized
+
+
+def test_scout_budgets_stay_within_the_recent_window_cap() -> None:
+ from skyvern.forge.sdk.copilot.tools.scouting import _SCOUT_RECON_RESULT_CHAR_CAP, _SCOUT_RESULT_CHAR_CAP
+
+ # The shed-never-slice guarantee holds only while every scout budget fits inside
+ # the transcript's recent-window cap.
+ assert _SCOUT_RESULT_CHAR_CAP <= _SCOUT_RECON_RESULT_CHAR_CAP <= _RECENT_TOOL_OUTPUT_CHAR_CAP
+
+
+def test_old_code_output_synopsis_names_elided_code_size() -> None:
+ code = "await page.click()\n" * 300
+ old_output = json.dumps({"ok": True, "data": {"code": code}})
+ items = [_fco("c_old", old_output)] + [_fco(f"c{i}", '{"ok":true}') for i in range(KEEP_RECENT_TOOL_OUTPUTS)]
+
+ pruned = _prune_input_list(items)
+ synopsis = json.loads(pruned[0]["output"])
+ assert synopsis["code_chars_elided"] == len(code)
+
+
def test_old_large_output_is_summarized() -> None:
# An older, large JSON tool output gets compressed into a synopsis.
heavy_payload = {
@@ -1702,15 +1732,19 @@ def test_summarize_short_output_is_unchanged() -> None:
def test_recent_large_output_is_head_truncated_not_summarized() -> None:
- # Big JSON in the most-recent slot should be head-truncated at 2000 chars,
+ import structlog.testing
+
+ # Over-cap JSON in the most-recent slot should be head-truncated,
# NOT replaced with a summary.
- large = '{"ok":true,"data":{"value":"' + ("y" * 3000) + '"}}'
+ large = '{"ok":true,"data":{"value":"' + ("y" * (_RECENT_TOOL_OUTPUT_CHAR_CAP + 1000)) + '"}}'
items = [_fco("c_recent", large)]
- pruned = _prune_input_list(items)
+ with structlog.testing.capture_logs() as logs:
+ pruned = _prune_input_list(items)
out = pruned[0]["output"]
assert out.startswith('{"ok":true,')
assert out.endswith("\n... [truncated]")
- assert len(out) <= 2020
+ assert len(out) <= _RECENT_TOOL_OUTPUT_CHAR_CAP + 20
+ assert any(entry["event"] == "copilot_recent_tool_output_truncated" for entry in logs)
class TestEnforcement:
@@ -3505,3 +3539,102 @@ class TestUnboundOfferReopen:
# Without the latch this stays True for the rest of the turn and the whole offer is
# re-emitted on every prompt build.
assert synthesized_offer_reopened_for_extraction_plan(ctx, None) is False
+
+
+class TestCollapseSupersededSynthesizedOffers:
+ """Refreshed synthesized-block offers supersede rather than stack: the
+ collapse keeps at most the last offer and drops the superseded ones."""
+
+ @staticmethod
+ def _offer(body: str) -> dict[str, Any]:
+ from skyvern.forge.sdk.copilot.code_block_synthesis import SYNTHESIZED_OFFER_SENTINEL
+
+ return {"role": "user", "content": SYNTHESIZED_OFFER_SENTINEL + " " + body}
+
+ def test_keeps_only_the_last_offer_with_distinct_payloads(self) -> None:
+ from skyvern.forge.sdk.copilot.enforcement import collapse_superseded_synthesized_offers
+
+ goal = {"role": "user", "content": "build me a login workflow"}
+ # Distinct, non-monotonic payloads: a refresh can fire because the plan
+ # changed or the trajectory was evicted, so the newest body is not a
+ # superset of the older ones. Keep-last must not assume it is.
+ items = [
+ goal,
+ self._offer("await page.goto('https://a.example')"),
+ {"type": "function_call", "call_id": "c1", "name": "evaluate", "arguments": "{}"},
+ {"type": "function_call_output", "call_id": "c1", "output": '{"ok": true}'},
+ self._offer("await page.click('#submit')"),
+ {"role": "assistant", "content": "scouting"},
+ self._offer("await page.goto('https://b.example')"),
+ ]
+ result = collapse_superseded_synthesized_offers(items)
+
+ offers = [it for it in result if isinstance(it, dict) and "SYNTHESIZED CODE BLOCK" in str(it.get("content"))]
+ assert len(offers) == 1
+ assert "b.example" in offers[0]["content"]
+ # Everything that is not a superseded offer survives untouched, in order.
+ assert result[0] == goal
+ assert [it.get("call_id") for it in result if isinstance(it, dict) and it.get("call_id")] == ["c1", "c1"]
+
+ def test_idempotent_and_noop_without_stacking(self) -> None:
+ from skyvern.forge.sdk.copilot.enforcement import collapse_superseded_synthesized_offers
+
+ items = [
+ {"role": "user", "content": "build me a workflow"},
+ {"type": "function_call_output", "call_id": "c1", "output": "{}"},
+ self._offer("await page.goto('https://a.example')"),
+ ]
+ once = collapse_superseded_synthesized_offers(items)
+ assert once == items
+ assert collapse_superseded_synthesized_offers(once) == once
+
+ def test_never_drops_the_opening_item(self) -> None:
+ """A first transcript item that happens to start with the sentinel is the
+ user's goal, not a superseded offer; collapsing it would delete the turn's
+ anchor. It also must not count as the kept offer."""
+ from skyvern.forge.sdk.copilot.enforcement import collapse_superseded_synthesized_offers
+
+ opening = self._offer("verbatim text a user pasted")
+ items = [opening, self._offer("stale"), self._offer("fresh")]
+ result = collapse_superseded_synthesized_offers(items)
+ assert result[0] == opening
+ assert [it["content"] for it in result[1:]] == [items[2]["content"]]
+
+ def test_ignores_non_offer_user_messages_and_list_content(self) -> None:
+ from skyvern.forge.sdk.copilot.enforcement import collapse_superseded_synthesized_offers
+
+ items = [
+ {"role": "user", "content": "build me a workflow"},
+ {"role": "user", "content": "SYNTHESIZED but not the sentinel"},
+ {"role": "user", "content": [{"type": "input_text", "text": "screenshot-ish"}]},
+ self._offer("stale"),
+ self._offer("fresh"),
+ {"role": "user", "content": "one more real message"},
+ ]
+ result = collapse_superseded_synthesized_offers(items)
+ assert [it.get("content") for it in result] == [
+ "build me a workflow",
+ "SYNTHESIZED but not the sentinel",
+ [{"type": "input_text", "text": "screenshot-ish"}],
+ items[4]["content"],
+ "one more real message",
+ ]
+
+ def test_prune_input_list_collapses_offers(self) -> None:
+ items = [
+ {"role": "user", "content": "build me a workflow"},
+ self._offer("stale one"),
+ {"type": "function_call_output", "call_id": "c1", "output": "{}"},
+ self._offer("stale two"),
+ self._offer("fresh"),
+ ]
+ result = _prune_input_list(items)
+ offers = [it for it in result if isinstance(it, dict) and "SYNTHESIZED CODE BLOCK" in str(it.get("content"))]
+ assert len(offers) == 1
+ assert "fresh" in offers[0]["content"]
+
+ def test_offer_is_a_synthetic_user_message(self) -> None:
+ from skyvern.forge.sdk.copilot.enforcement import is_synthetic_user_message
+
+ assert is_synthetic_user_message(self._offer("await page.goto('https://a.example')"))
+ assert not is_synthetic_user_message({"role": "user", "content": "build me a workflow"})
diff --git a/tests/unit/test_copilot_request_policy_workflow_credentials.py b/tests/unit/test_copilot_request_policy_workflow_credentials.py
index 219a4ee2d..0b2fd01bd 100644
--- a/tests/unit/test_copilot_request_policy_workflow_credentials.py
+++ b/tests/unit/test_copilot_request_policy_workflow_credentials.py
@@ -32,8 +32,8 @@ from skyvern.forge.sdk.copilot.request_policy import (
_can_defer_unresolved_credential_name_for_draft,
_classification_from_raw,
_clean_email_list,
- _exact_credential_name_candidates,
_ground_user_provided_sites,
+ _has_resolvable_credential_scope,
_resolve_credentials,
_saved_credential_names_mentioned,
_seed_prior_approved_credentials,
@@ -352,10 +352,10 @@ async def test_fallback_code_block_credential_request_saves_draft_without_runnin
handler=None,
)
+ # Names are read by the model, so a turn with no classifier resolves none — the draft still
+ # saves without running, which is what this fallback owes the user.
assert policy.classifier_status == "fallback"
- assert policy.credential_input_kind == "credential_name"
- assert policy.credential_refs == ["mock-portal-login-email-otp"]
- assert policy.resolved_credentials == [credential]
+ assert policy.resolved_credentials == []
assert policy.allow_update_workflow is True
assert policy.allow_run_blocks is False
assert policy.allow_missing_credentials_in_draft is True
@@ -506,11 +506,17 @@ async def _build_with_forced_classifier(
"kind",
["none", "credential_id", "credential_name", "website_stored_credential", "placeholder"],
)
-async def test_success_empty_refs_resolves_named_credential(kind: str) -> None:
+async def test_an_extracted_name_resolves_under_any_credential_label(kind: str) -> None:
+ """Which label the classifier chose must not decide whether the name it read is looked up
+ (SKY-12917); the name itself is the signal. A name it never read resolves nothing — reading the
+ message is the model's job, and searching prose for each saved name matched things like a `/hex`
+ URL segment."""
credential = _cred("mock-portal-login", "cred_login")
policy = await _build_with_forced_classifier(
user_message="Sign in with the saved credential named mock-portal-login.",
- classifier_policy=RequestPolicy(credential_input_kind=kind, classifier_status="success"),
+ classifier_policy=RequestPolicy(
+ credential_input_kind=kind, credential_refs=["mock-portal-login"], classifier_status="success"
+ ),
org_credentials=[credential, _cred("other", "cred_other")],
)
@@ -528,6 +534,7 @@ async def test_success_website_kind_with_nonmatching_url_resolves_named_credenti
policy = await _build_with_forced_classifier(
user_message="Log in with the saved credential named mock-portal-login here.",
classifier_policy=RequestPolicy(
+ credential_refs=["mock-portal-login"],
credential_input_kind="website_stored_credential",
login_page_urls=["https://unrelated.example.com/login"],
classifier_status="success",
@@ -547,6 +554,7 @@ async def test_success_website_kind_prefers_named_credential_over_matching_url()
policy = await _build_with_forced_classifier(
user_message="Log in with the saved credential named other-login.",
classifier_policy=RequestPolicy(
+ credential_refs=["other-login"],
credential_input_kind="website_stored_credential",
login_page_urls=["https://portal.example.com/login"],
classifier_status="success",
@@ -576,13 +584,24 @@ async def _resolve_direct(
return load_mock, by_ids_mock
- "user_message",
- ["Use credential ID cred_current.", "Use the saved credential named current-login."],
-)
-def test_draft_deferral_recognizes_current_turn_credential_scope(user_message: str) -> None:
+_CURRENT_TURN_CREDENTIAL_SCOPES = [
+ pytest.param("Use credential ID cred_current.", "none", [], id="explicit_id"),
+ pytest.param(
+ "Use the saved credential named current-login.",
+ "credential_name",
+ ["current-login"],
+ id="corroborated_name",
+ ),
+]
+
+
[email protected](("user_message", "kind", "credential_refs"), _CURRENT_TURN_CREDENTIAL_SCOPES)
+def test_draft_deferral_recognizes_current_turn_credential_scope(
+ user_message: str, kind: str, credential_refs: list[str]
+) -> None:
policy = RequestPolicy(
- credential_input_kind="none",
+ credential_input_kind=kind,
+ credential_refs=credential_refs,
clarification_reason="credential_name_unresolved",
)
@@ -593,13 +612,13 @@ def test_draft_deferral_recognizes_current_turn_credential_scope(user_message: s
)
- "user_message",
- ["Use credential ID cred_current.", "Use the saved credential named current-login."],
-)
-def test_repeated_question_deferral_preserves_current_turn_credential_scope(user_message: str) -> None:
[email protected](("user_message", "kind", "credential_refs"), _CURRENT_TURN_CREDENTIAL_SCOPES)
+def test_repeated_question_deferral_preserves_current_turn_credential_scope(
+ user_message: str, kind: str, credential_refs: list[str]
+) -> None:
policy = RequestPolicy(
- credential_input_kind="none",
+ credential_input_kind=kind,
+ credential_refs=credential_refs,
clarification_reason="credential_name_unresolved",
)
history = [
@@ -640,6 +659,40 @@ async def test_name_miss_does_not_advance_to_classifier_only_url() -> None:
load_mock.assert_awaited_once()
[email protected]("kind", ["credential_name", "credential_id", "website_stored_credential"])
+async def test_an_unmatched_stated_name_asks_instead_of_binding_another_credential(kind: str) -> None:
+ """Naming a credential the org does not have must ask, not let the sole-credential rule hand
+ the turn a different one by elimination, whichever credential label carried the ref."""
+ unrelated = _cred("personal-bank", "cred_bank")
+ policy = RequestPolicy(credential_input_kind=kind, credential_refs=["acme-prd"], login_intent=True)
+
+ await _resolve_direct(
+ policy,
+ user_message="Log in to https://acme.example.com/login with credential acme-prd.",
+ org_credentials=[unrelated],
+ )
+
+ assert policy.resolved_credentials == []
+ assert policy.auto_bound_credentials == []
+
+
+async def test_a_vague_credential_reference_still_resolves_the_sole_credential() -> None:
+ """ "Use my credential" with exactly one saved credential has an obvious answer; asking which
+ one would be a question with one option."""
+ sole = _cred("personal-bank", "cred_bank")
+ policy = RequestPolicy(credential_input_kind="credential_name", login_intent=True)
+
+ await _resolve_direct(
+ policy,
+ user_message="Log in to https://acme.example.com/login with my credential.",
+ org_credentials=[sole],
+ )
+
+ assert [credential.credential_id for credential in policy.resolved_credentials] == ["cred_bank"]
+
+
@pytest.mark.asyncio
@pytest.mark.parametrize("name", ["abc", "spare-portal"], ids=["three_char", "twelve_char"])
async def test_url_less_ambiguity_is_independent_of_credential_name_length(name: str) -> None:
@@ -664,13 +717,9 @@ async def test_url_less_ambiguity_is_independent_of_credential_name_length(name:
"kind",
["none", "raw_secret", "credential_id", "credential_name", "website_stored_credential", "placeholder"],
)
[email protected]("credential_refs", [[], ["mock-portal-login"]], ids=["without_refs", "with_refs"])
-async def test_resolver_exact_name_is_independent_of_classifier_label(
- kind: str,
- credential_refs: list[str],
-) -> None:
+async def test_resolver_exact_name_is_independent_of_classifier_label(kind: str) -> None:
credential = _cred("mock-portal-login", "cred_login")
- policy = RequestPolicy(credential_input_kind=kind, credential_refs=credential_refs)
+ policy = RequestPolicy(credential_input_kind=kind, credential_refs=["mock-portal-login"])
await _resolve_direct(
policy,
@@ -686,7 +735,6 @@ async def test_resolver_exact_name_is_independent_of_classifier_label(
@pytest.mark.asyncio
async def test_explicit_id_wins_and_skips_name_and_url_scan() -> None:
by_ids = AsyncMock(return_value=[_cred("id-login", "cred_explicit")])
- load_mock = AsyncMock(side_effect=AssertionError("organization scan should not run after an ID hit"))
policy = RequestPolicy(
credential_input_kind="credential_id",
credential_refs=["cred_explicit", "named-login"],
@@ -697,13 +745,11 @@ async def test_explicit_id_wins_and_skips_name_and_url_scan() -> None:
policy,
user_message="Use credential ID cred_explicit and the saved credential named named-login.",
org_credentials=[],
- get_credentials=load_mock,
get_credentials_by_ids=by_ids,
)
assert [candidate.credential_id for candidate in policy.resolved_credentials] == ["cred_explicit"]
by_ids.assert_awaited_once_with(["cred_explicit"], organization_id="o_test")
- load_mock.assert_not_awaited()
@pytest.mark.asyncio
@@ -711,7 +757,6 @@ async def test_explicit_id_wins_for_non_id_label_and_competing_url_match() -> No
explicit = _cred("id-login", "cred_explicit")
competing = _cred("url-login", "cred_url", tested_url="https://portal.example.com/login")
by_ids = AsyncMock(return_value=[explicit])
- load_mock = AsyncMock(side_effect=AssertionError("organization scan should not run after an ID hit"))
policy = RequestPolicy(
credential_input_kind="website_stored_credential",
credential_refs=["cred_explicit"],
@@ -722,13 +767,11 @@ async def test_explicit_id_wins_for_non_id_label_and_competing_url_match() -> No
policy,
user_message="Use credential ID cred_explicit for https://portal.example.com/login.",
org_credentials=[competing],
- get_credentials=load_mock,
get_credentials_by_ids=by_ids,
)
assert [candidate.credential_id for candidate in policy.resolved_credentials] == ["cred_explicit"]
by_ids.assert_awaited_once_with(["cred_explicit"], organization_id="o_test")
- load_mock.assert_not_awaited()
@pytest.mark.asyncio
@@ -774,6 +817,7 @@ async def test_classifier_id_absent_from_current_message_does_not_resolve() -> N
"Avoid using cred_prod.",
"Exclude credential ID cred_prod.",
"Do not use the old workflow because it still mentions credential ID cred_prod.",
+ "Do not use the old workflow because it mentions cred_prod.",
"Do not use this credential, credential ID cred_prod.",
],
)
@@ -907,7 +951,7 @@ async def test_chained_replacement_uses_final_affirmative_id() -> None:
async def test_unrelated_replacement_does_not_grant_credential_id_authority() -> None:
alpha = _cred("alpha", "cred_alpha")
by_ids = AsyncMock(side_effect=AssertionError("documentation ID must not gain credential authority"))
- policy = RequestPolicy(credential_input_kind="credential_name")
+ policy = RequestPolicy(credential_input_kind="credential_name", credential_refs=["alpha"])
await _resolve_direct(
policy,
@@ -923,7 +967,7 @@ async def test_unrelated_replacement_does_not_grant_credential_id_authority() ->
@pytest.mark.asyncio
async def test_unrelated_replacement_does_not_grant_credential_name_authority() -> None:
unrelated = _cred("new-login", "cred_new")
- policy = RequestPolicy(credential_input_kind="credential_name")
+ policy = RequestPolicy(credential_input_kind="credential_name", credential_refs=["alpha", "new-login"])
await _resolve_direct(
policy,
@@ -939,7 +983,7 @@ async def test_unrelated_replacement_does_not_grant_credential_name_authority()
@pytest.mark.asyncio
async def test_unrelated_replacement_does_not_grant_postfix_credential_name_authority() -> None:
unrelated = _cred("new-login", "cred_new")
- policy = RequestPolicy(credential_input_kind="credential_name")
+ policy = RequestPolicy(credential_input_kind="credential_name", credential_refs=["alpha", "new-login"])
await _resolve_direct(
policy,
@@ -987,7 +1031,6 @@ async def test_bare_id_appositive_negation_revokes_earlier_authority() -> None:
@pytest.mark.asyncio
async def test_invalid_explicit_id_does_not_fall_through_to_name() -> None:
- load_mock = AsyncMock(side_effect=AssertionError("organization scan should not run after an explicit ID"))
policy = RequestPolicy(
credential_input_kind="credential_id",
credential_refs=["cred_missing", "named-login"],
@@ -997,20 +1040,18 @@ async def test_invalid_explicit_id_does_not_fall_through_to_name() -> None:
policy,
user_message="Use credential ID cred_missing and the saved credential named named-login.",
org_credentials=[],
- get_credentials=load_mock,
)
assert policy.invalid_credential_ids == ["cred_missing"]
assert policy.clarification_reason == "credential_name_unresolved"
assert policy.credential_ask_card_answerable is False
assert credential_pause_reason(SimpleNamespace(request_policy=policy)) is None
- load_mock.assert_not_awaited()
@pytest.mark.asyncio
async def test_duplicate_name_hits_are_deduplicated_by_credential_id() -> None:
credential = _cred("named-login", "cred_named")
- policy = RequestPolicy(credential_input_kind="placeholder")
+ policy = RequestPolicy(credential_input_kind="placeholder", credential_refs=["named-login"])
await _resolve_direct(
policy,
@@ -1025,6 +1066,7 @@ async def test_duplicate_name_hits_are_deduplicated_by_credential_id() -> None:
@pytest.mark.asyncio
async def test_multiple_name_hits_keep_the_existing_picker_and_skip_url() -> None:
policy = RequestPolicy(
+ credential_refs=["shared-login"],
credential_input_kind="website_stored_credential",
login_page_urls=["https://portal.example.com/login"],
login_intent=True,
@@ -1058,6 +1100,7 @@ async def test_name_collision_candidates_are_not_labelled_safe_matches() -> None
must not call them safe matches even when every one of them has been tested somewhere.
"""
policy = RequestPolicy(
+ credential_refs=["shared-login"],
credential_input_kind="website_stored_credential",
login_page_urls=["https://portal.example.com/login"],
login_intent=True,
@@ -1129,7 +1172,7 @@ async def test_zero_hit_policy_is_byte_identical_for_each_label(
) -> None:
policy = RequestPolicy(credential_input_kind=kind)
- load_mock, by_ids = await _resolve_direct(
+ _, by_ids = await _resolve_direct(
policy,
user_message="Continue with the workflow.",
org_credentials=[],
@@ -1143,7 +1186,6 @@ async def test_zero_hit_policy_is_byte_identical_for_each_label(
policy.clarification_reason,
policy.clarification_question,
) == expected
- load_mock.assert_not_awaited()
by_ids.assert_not_awaited()
@@ -1153,6 +1195,7 @@ async def test_build_path_named_candidate_overrides_only_credential_target_clari
user_message="Sign in with the saved credential named mock-portal-login.",
classifier_policy=RequestPolicy(
credential_input_kind="website_stored_credential",
+ credential_refs=["mock-portal-login"],
requires_user_clarification=True,
clarification_reason="missing_target_context",
classifier_status="success",
@@ -1171,7 +1214,6 @@ async def test_build_path_named_candidate_overrides_only_credential_target_clari
@pytest.mark.asyncio
async def test_classifier_stale_name_ref_without_current_message_anchor_does_not_resolve() -> None:
- load_mock = AsyncMock(side_effect=AssertionError("stale classifier refs must not scan credentials"))
policy = await _build_with_forced_classifier(
user_message="Continue with the workflow.",
classifier_policy=RequestPolicy(
@@ -1180,12 +1222,10 @@ async def test_classifier_stale_name_ref_without_current_message_anchor_does_not
classifier_status="success",
),
org_credentials=[],
- get_credentials=load_mock,
)
assert policy.resolved_credentials == []
assert policy.clarification_reason == "none"
- load_mock.assert_not_awaited()
@pytest.mark.asyncio
@@ -1203,7 +1243,6 @@ async def test_classifier_stale_name_ref_without_current_message_anchor_does_not
],
)
async def test_classifier_name_ref_requires_exact_affirmative_credential_context(user_message: str) -> None:
- load_mock = AsyncMock(side_effect=AssertionError("untrusted classifier refs must not scan credentials"))
policy = RequestPolicy(
credential_input_kind="credential_name",
credential_refs=["prod"],
@@ -1213,11 +1252,9 @@ async def test_classifier_name_ref_requires_exact_affirmative_credential_context
policy,
user_message=user_message,
org_credentials=[],
- get_credentials=load_mock,
)
assert policy.resolved_credentials == []
- load_mock.assert_not_awaited()
@pytest.mark.asyncio
@@ -1232,6 +1269,7 @@ async def test_later_affirmative_name_resolves_after_excluded_name(user_message:
policy = RequestPolicy(
credential_input_kind="credential_name",
credential_refs=["old", "new"],
+ login_intent=True,
)
await _resolve_direct(
@@ -1323,7 +1361,7 @@ async def test_credential_grammar_word_does_not_preempt_message_url_resolution()
async def test_replacement_name_discards_superseded_candidate() -> None:
old = _cred("old-login", "cred_old")
new = _cred("new-login", "cred_new")
- policy = RequestPolicy(credential_input_kind="credential_name")
+ policy = RequestPolicy(credential_input_kind="credential_name", credential_refs=["old-login", "new-login"])
await _resolve_direct(
policy,
@@ -1338,7 +1376,7 @@ async def test_replacement_name_discards_superseded_candidate() -> None:
async def test_postfix_replacement_name_discards_superseded_candidate() -> None:
old = _cred("old-login", "cred_old")
new = _cred("new-login", "cred_new")
- policy = RequestPolicy(credential_input_kind="credential_name")
+ policy = RequestPolicy(credential_input_kind="credential_name", credential_refs=["new-login"])
await _resolve_direct(
policy,
@@ -1354,7 +1392,9 @@ async def test_chained_replacement_name_uses_final_affirmative_candidate() -> No
old = _cred("old-login", "cred_old")
new = _cred("new-login", "cred_new")
final = _cred("final-login", "cred_final")
- policy = RequestPolicy(credential_input_kind="credential_name")
+ policy = RequestPolicy(
+ credential_input_kind="credential_name", credential_refs=["old-login", "new-login", "final-login"]
+ )
await _resolve_direct(
policy,
@@ -1371,7 +1411,7 @@ async def test_chained_replacement_name_uses_final_affirmative_candidate() -> No
@pytest.mark.asyncio
async def test_unrelated_switch_preserves_credential_name_authority() -> None:
credential = _cred("prod-login", "cred_prod")
- policy = RequestPolicy(credential_input_kind="credential_name")
+ policy = RequestPolicy(credential_input_kind="credential_name", credential_refs=["prod-login"])
await _resolve_direct(
policy,
@@ -1537,6 +1577,7 @@ async def test_website_kind_explicit_name_wins_over_url_matched_card() -> None:
),
classifier_policy=RequestPolicy(
credential_input_kind="website_stored_credential",
+ credential_refs=["analytics-login"],
login_page_urls=["https://portal.example.com/login"],
login_intent=True,
classifier_status="success",
@@ -1553,7 +1594,6 @@ async def test_website_kind_explicit_name_wins_over_url_matched_card() -> None:
)
assert policy.credential_input_kind == "website_stored_credential"
- assert policy.credential_refs == []
assert [c.credential_id for c in policy.resolved_credentials] == ["cred_named"]
assert policy.requires_user_clarification is False
assert policy.clarification_question is None
@@ -1562,7 +1602,6 @@ async def test_website_kind_explicit_name_wins_over_url_matched_card() -> None:
@pytest.mark.asyncio
async def test_success_credential_id_intent_is_not_overridden_by_name_scan() -> None:
by_ids = AsyncMock(return_value=[_cred("real", "cred_real")])
- load_mock = AsyncMock(return_value=[_cred("mock-portal-login", "cred_login")])
policy = await _build_with_forced_classifier(
user_message="Use cred_real, the saved credential named mock-portal-login.",
classifier_policy=RequestPolicy(
@@ -1571,13 +1610,11 @@ async def test_success_credential_id_intent_is_not_overridden_by_name_scan() ->
classifier_status="success",
),
org_credentials=[],
- get_credentials=load_mock,
get_credentials_by_ids=by_ids,
)
assert policy.credential_input_kind == "credential_id"
assert [c.credential_id for c in policy.resolved_credentials] == ["cred_real"]
- load_mock.assert_not_awaited()
@pytest.mark.asyncio
@@ -1618,7 +1655,9 @@ async def test_success_pre_resolution_clarification_is_preserved() -> None:
async def test_success_ambiguous_named_credentials_clarify_for_normal_request() -> None:
policy = await _build_with_forced_classifier(
user_message="Use the credential named alpha or the credential named beta to sign in.",
- classifier_policy=RequestPolicy(credential_input_kind="none", classifier_status="success"),
+ classifier_policy=RequestPolicy(
+ credential_input_kind="credential_name", credential_refs=["alpha", "beta"], classifier_status="success"
+ ),
org_credentials=[_cred("alpha", "cred_alpha"), _cred("beta", "cred_beta")],
)
@@ -1645,39 +1684,35 @@ async def test_success_ambiguous_named_credentials_draft_unbound_for_explicit_dr
@pytest.mark.asyncio
async def test_success_incidental_quote_out_of_credential_context_is_not_rewritten() -> None:
- load_mock = AsyncMock(return_value=[_cred("mock-portal-login", "cred_login")])
policy = await _build_with_forced_classifier(
user_message='Set the page title to "mock-portal-login" before saving.',
classifier_policy=RequestPolicy(credential_input_kind="none", classifier_status="success"),
org_credentials=[],
- get_credentials=load_mock,
)
assert policy.credential_input_kind == "none"
assert policy.resolved_credentials == []
- load_mock.assert_not_awaited()
@pytest.mark.asyncio
async def test_success_quote_after_word_containing_login_substring_is_not_context() -> None:
- load_mock = AsyncMock(return_value=[_cred("mock-portal-login", "cred_login")])
policy = await _build_with_forced_classifier(
user_message='Publish the blog in the section titled "mock-portal-login".',
classifier_policy=RequestPolicy(credential_input_kind="none", classifier_status="success"),
org_credentials=[],
- get_credentials=load_mock,
)
assert policy.credential_input_kind == "none"
assert policy.resolved_credentials == []
- load_mock.assert_not_awaited()
@pytest.mark.asyncio
async def test_success_quoted_name_in_credential_context_resolves() -> None:
policy = await _build_with_forced_classifier(
user_message='Sign in using the saved credential "mock-portal-login".',
- classifier_policy=RequestPolicy(credential_input_kind="credential_name", classifier_status="success"),
+ classifier_policy=RequestPolicy(
+ credential_input_kind="credential_name", credential_refs=["mock-portal-login"], classifier_status="success"
+ ),
org_credentials=[_cred("mock-portal-login", "cred_login")],
)
@@ -1699,7 +1734,7 @@ async def test_success_zero_name_match_falls_through_to_clarification() -> None:
@pytest.mark.asyncio
async def test_success_raw_secret_blocks_name_scan_before_loading_credentials() -> None:
- load_mock = AsyncMock(return_value=[_cred("mock-portal-login", "cred_login")])
+ load_mock = AsyncMock(side_effect=AssertionError("a raw-secret turn must not read the credential list"))
policy = await _build_with_forced_classifier(
user_message="Sign in with the saved credential named mock-portal-login.",
classifier_policy=RequestPolicy(
@@ -2040,21 +2075,17 @@ async def test_login_intent_none_kind_bare_host_message_does_not_trigger_url_res
@pytest.mark.asyncio
-async def test_login_intent_false_none_kind_never_loads_credentials_for_message_url() -> None:
- with patch(
- "skyvern.forge.sdk.copilot.request_policy._load_credentials",
- new=AsyncMock(side_effect=AssertionError("_load_credentials should not be called")),
- ):
- policy = await _build_with_forced_classifier(
- user_message="Go to http://localhost:8941/analytics_console/ and inspect the page.",
- classifier_policy=RequestPolicy(
- login_intent=False,
- credential_input_kind="none",
- login_page_urls=[],
- classifier_status="success",
- ),
- org_credentials=[],
- )
+async def test_login_intent_false_none_kind_binds_nothing_for_a_message_url() -> None:
+ policy = await _build_with_forced_classifier(
+ user_message="Go to http://localhost:8941/analytics_console/ and inspect the page.",
+ classifier_policy=RequestPolicy(
+ login_intent=False,
+ credential_input_kind="none",
+ login_page_urls=[],
+ classifier_status="success",
+ ),
+ org_credentials=[],
+ )
assert policy.user_response_policy == "proceed"
assert policy.resolved_credentials == []
@@ -2768,17 +2799,51 @@ def test_a_named_credential_resolves_without_the_word_credential() -> None:
""" "use skyvern-datadog to login" names the credential outright. The pattern-based candidates
only fire when the message also says "credential", which left this resolving nothing."""
credentials = [SimpleNamespace(name="skyvern-datadog"), SimpleNamespace(name="skyvern-posthog")]
+ policy = RequestPolicy(credential_input_kind="credential_name", credential_refs=["skyvern-datadog"])
+
+ assert _saved_credential_names_mentioned(policy, "use skyvern-datadog to login.", credentials) == [
+ "skyvern-datadog"
+ ]
+
+
+ "user_message",
+ [
+ "use the saved credential named prod-account",
+ "use credential prod-account",
+ "use the prod-account credential",
+ "use the credential called prod-account",
+ "log in with prod-account",
+ "sign in using prod-account",
+ "grab the invoice with prod-account",
+ "use `prod-account` to log in",
+ "I've connected the credential 'prod-account' — continue.",
+ ],
+)
+def test_a_saved_name_resolves_however_the_message_phrases_it(user_message: str) -> None:
+ """A saved name is recognised however the sentence is built, including without the word "credential"."""
+ credentials = [SimpleNamespace(name="prod-account"), SimpleNamespace(name="skyvern-posthog")]
+ policy = RequestPolicy(credential_input_kind="credential_name", credential_refs=["prod-account"])
- assert _saved_credential_names_mentioned("use skyvern-datadog to login.", credentials) == ["skyvern-datadog"]
+ assert _saved_credential_names_mentioned(policy, user_message, credentials) == ["prod-account"]
-def test_contraction_apostrophe_does_not_open_a_bogus_quoted_name() -> None:
- """A contraction's apostrophe once paired with a later quote and swallowed the span between them as
- a credential name (e.g. from "I've connected the credential 'hex'"). It must extract only 'hex'."""
- candidates = _exact_credential_name_candidates("I've connected the credential 'hex' — continue.")
+ "user_message",
+ [
+ "Use credential ID cred_x for the portal.",
+ "log in with the credential",
+ "use my credential to sign in",
+ "use the saved credential",
+ "just log in please",
+ ],
+)
+def test_prose_around_the_word_credential_is_never_a_candidate(user_message: str) -> None:
+ """Prose around the word "credential" names no credential: a candidate is a saved name or nothing."""
+ credentials = [SimpleNamespace(name="prod-account"), SimpleNamespace(name="skyvern-posthog")]
+ policy = RequestPolicy(credential_input_kind="none")
- assert "hex" in candidates
- assert not any("connected" in candidate for candidate in candidates)
+ assert _saved_credential_names_mentioned(policy, user_message, credentials) == []
@pytest.mark.asyncio
@@ -2787,7 +2852,9 @@ async def test_a_named_credential_resolves_when_the_classifier_misses_login_inte
it alone let the classifier veto a message that says "log in" outright — the very case this
resolves. The deterministic login phrasing must reach the scan on its own."""
named = SimpleNamespace(credential_id="cred_datadog", name="skyvern-datadog", credential_type="password")
- policy = RequestPolicy(credential_input_kind="skyvern_stored_credential", login_intent=False)
+ policy = RequestPolicy(
+ credential_input_kind="skyvern_stored_credential", login_intent=False, credential_refs=["skyvern-datadog"]
+ )
await _resolve_direct(policy, user_message="use skyvern-datadog to login.", org_credentials=[named])
@@ -2796,8 +2863,49 @@ async def test_a_named_credential_resolves_when_the_classifier_misses_login_inte
def test_a_negated_saved_name_is_not_treated_as_a_reference() -> None:
credentials = [SimpleNamespace(name="skyvern-datadog")]
+ policy = RequestPolicy(credential_input_kind="credential_name", credential_refs=["skyvern-datadog"])
+
+ assert _saved_credential_names_mentioned(policy, "do not use skyvern-datadog", credentials) == []
+
+
[email protected]("user_message", ["use hex to log in", "log in with qa"])
+def test_a_short_saved_name_is_still_a_reference(user_message: str) -> None:
+ """Orgs do keep two- and three-character credential names, and naming one is the whole point of
+ saying it out loud; only a model-authored ref needs a distinctiveness floor."""
+ credentials = [SimpleNamespace(name="hex"), SimpleNamespace(name="qa")]
+ policy = RequestPolicy(credential_input_kind="credential_name", credential_refs=["hex", "qa"])
+
+ assert _saved_credential_names_mentioned(policy, user_message, credentials) != []
+
+
+def test_a_short_saved_name_holds_off_the_clarification_question() -> None:
+ """A name the org actually has must reach resolution whatever its length; asking instead sends
+ the user back to retype the name they already gave."""
+ policy = RequestPolicy(
+ credential_input_kind="credential_name",
+ credential_refs=["qa"],
+ requires_user_clarification=True,
+ clarification_reason="credential_name_unresolved",
+ )
+
+ assert _has_resolvable_credential_scope(policy, "log in with qa") is True
+
+
+async def test_saved_names_differing_only_in_case_reach_the_picker() -> None:
+ """Credential names carry no unique index, so `Prod` and `prod` are two credentials and the turn
+ has to ask which is meant instead of binding whichever the org saved last."""
+ policy = RequestPolicy(credential_input_kind="credential_name", credential_refs=["Prod"], login_intent=True)
+
+ await _resolve_direct(
+ policy,
+ user_message="Use the saved credential Prod.",
+ org_credentials=[_cred("Prod", "cred_upper"), _cred("prod", "cred_lower")],
+ )
- assert _saved_credential_names_mentioned("do not use skyvern-datadog", credentials) == []
+ assert policy.resolved_credentials == []
+ assert policy.requires_user_clarification is True
+ assert policy.clarification_reason == "credential_name_unresolved"
async def _admit(
@@ -3272,30 +3380,30 @@ async def test_a_credential_labelled_turn_looks_a_written_name_up_by_name(kind:
@pytest.mark.asyncio
[email protected]("kind", ["none", "placeholder"])
-async def test_a_turn_the_classifier_never_called_a_credential_request_does_not_scan(kind: str) -> None:
[email protected]("kind", ["none", "placeholder", "credential_name"])
+async def test_a_ref_the_message_never_states_resolves_nothing(kind: str) -> None:
+ """What keeps an unrelated turn away from the credential list is the ref being absent from the
+ message, not which label the classifier chose — it labels a bare name a credential request."""
policy = RequestPolicy(credential_input_kind=kind, credential_refs=["analytics-login"])
- load_mock, _ = await _resolve_direct(
+ _, _ = await _resolve_direct(
policy,
- user_message="Use analytics-login here.",
+ user_message="Continue with the workflow.",
org_credentials=[_cred("analytics-login", "cred_analytics")],
- get_credentials=AsyncMock(side_effect=AssertionError("a non-credential turn must not scan")),
)
assert policy.resolved_credentials == []
- load_mock.assert_not_awaited()
@pytest.mark.asyncio
-async def test_a_degraded_classifier_never_enumerates_saved_credentials(kind: str = "credential_name") -> None:
- policy = RequestPolicy(
- credential_input_kind=kind, credential_refs=["analytics-login"], classifier_status="fallback"
- )
+async def test_a_degraded_classifier_never_enumerates_saved_credentials() -> None:
+ """A turn the classifier could not read reports no name, and a turn with no name to look up
+ never reads the org's credentials (GOTCHAS 23)."""
+ policy = RequestPolicy(credential_input_kind="none", classifier_status="fallback")
load_mock, _ = await _resolve_direct(
policy,
user_message="Use analytics-login here.",
org_credentials=[_cred("analytics-login", "cred_analytics")],
- get_credentials=AsyncMock(side_effect=AssertionError("a degraded turn must not scan")),
+ get_credentials=AsyncMock(side_effect=AssertionError("a degraded turn must not enumerate credentials")),
)
assert policy.resolved_credentials == []
@@ -3305,15 +3413,13 @@ async def test_a_degraded_classifier_never_enumerates_saved_credentials(kind: st
@pytest.mark.asyncio
async def test_a_stale_ref_absent_from_the_message_still_authorizes_nothing() -> None:
policy = RequestPolicy(credential_input_kind="website_stored_credential", credential_refs=["analytics-login"])
- load_mock, by_ids_mock = await _resolve_direct(
+ _, by_ids_mock = await _resolve_direct(
policy,
user_message="Continue with the workflow.",
org_credentials=[_cred("analytics-login", "cred_analytics")],
- get_credentials=AsyncMock(side_effect=AssertionError("a stale ref must not scan credentials")),
)
assert policy.resolved_credentials == []
- load_mock.assert_not_awaited()
def _passwordless_email_policy() -> RequestPolicy:
@@ -3401,7 +3507,7 @@ async def test_session_reuse_language_grants_no_authority_at_either_seam() -> No
(SKY-10131) — and the live-page seam must not convert it into authority either, because
the tool gate, not the wording, is what reaches admission."""
policy = RequestPolicy(login_intent=False, credential_input_kind="none")
- load_mock, _ = await _resolve_direct(
+ _, _ = await _resolve_direct(
policy,
user_message="Use the tab where I'm signed in at https://analytics.example.com/web and read the visitor count.",
org_credentials=[_cred("analytics", "cred_analytics", "https://analytics.example.com/login")],
diff --git a/tests/unit/test_copilot_scout_act_observe.py b/tests/unit/test_copilot_scout_act_observe.py
index 6a846bfe6..211c1ffb3 100644
--- a/tests/unit/test_copilot_scout_act_observe.py
+++ b/tests/unit/test_copilot_scout_act_observe.py
@@ -34,7 +34,6 @@ from skyvern.forge.sdk.copilot.context import (
finalize_discovery_counter_in_global_llm_context,
)
from skyvern.forge.sdk.copilot.enforcement import (
- _RECENT_TOOL_OUTPUT_CHAR_CAP,
record_scouted_output_coverage,
)
from skyvern.forge.sdk.copilot.output_extraction_plan import ShapeExpectation, ValueCardinality, ValueShape
@@ -44,6 +43,7 @@ from skyvern.forge.sdk.copilot.runtime import AgentContext
from skyvern.forge.sdk.copilot.tools import _click_post_hook
from skyvern.forge.sdk.copilot.tools import scouting as scouting_module
from skyvern.forge.sdk.copilot.tools.scouting import (
+ _SCOUT_RESULT_CHAR_CAP,
_capture_scout_source_url,
_consume_pending_browser_interaction_observation,
_mark_pending_browser_interaction_observation,
@@ -547,7 +547,7 @@ class TestActObserveSuccess:
assert page["result_container_count"] == 1
assert page["challenge_detected"] is False
assert page["modal_dismiss_controls"] == ["Accept"]
- assert len(json.dumps(result)) <= _RECENT_TOOL_OUTPUT_CHAR_CAP
+ assert len(json.dumps(result)) <= _SCOUT_RESULT_CHAR_CAP
@pytest.mark.asyncio
async def test_content_witnessed_kv_reveal_admitted_without_bounded_schema(self) -> None:
@@ -1157,7 +1157,7 @@ class TestActObserveSummaryBound:
assert result["ok"] is True
serialized = json.dumps(result)
- assert len(serialized) <= _RECENT_TOOL_OUTPUT_CHAR_CAP
+ assert len(serialized) <= _SCOUT_RESULT_CHAR_CAP
assert json.loads(serialized) == result
# The flow-evidence packet keeps the full schema; only the tool result is compact.
assert ctx.flow_evidence[0]["had_bounded_schema"] is True
diff --git a/tests/unit/test_copilot_session_factory.py b/tests/unit/test_copilot_session_factory.py
index 46c6c8358..b7488ce17 100644
--- a/tests/unit/test_copilot_session_factory.py
+++ b/tests/unit/test_copilot_session_factory.py
@@ -56,6 +56,71 @@ class TestFirstTurnCompaction:
for recent in recent_three:
assert small_summary_marker not in recent["output"]
+ def test_recent_code_sized_output_survives_session_compaction(self) -> None:
+ from skyvern.forge.sdk.copilot.enforcement import _RECENT_TOOL_OUTPUT_CHAR_CAP
+ from skyvern.forge.sdk.copilot.session_factory import copilot_call_model_input_filter
+
+ code_sized = json.dumps({"ok": True, "data": {"code": "await page.click()\n" * 400}})
+ assert 2000 < len(code_sized) < _RECENT_TOOL_OUTPUT_CHAR_CAP
+ items: list[dict[str, Any]] = [
+ {"role": "user", "content": "please build me a workflow"},
+ {"type": "function_call_output", "call_id": "call-code", "output": code_sized},
+ ]
+ result = copilot_call_model_input_filter(_mk_input_data(items))
+ outputs = [it for it in result.input if it.get("type") == "function_call_output"]
+ assert outputs[0]["output"] == code_sized
+
+ def test_recent_overcap_output_truncates_and_warns_on_session_path(self) -> None:
+ import structlog.testing
+
+ from skyvern.forge.sdk.copilot.enforcement import _RECENT_TOOL_OUTPUT_CHAR_CAP
+ from skyvern.forge.sdk.copilot.session_factory import copilot_call_model_input_filter
+
+ oversized = "x" * (_RECENT_TOOL_OUTPUT_CHAR_CAP + 1000)
+ items: list[dict[str, Any]] = [
+ {"role": "user", "content": "please build me a workflow"},
+ {"type": "function_call_output", "call_id": "call-big", "output": oversized},
+ ]
+ with structlog.testing.capture_logs() as logs:
+ result = copilot_call_model_input_filter(_mk_input_data(items))
+ outputs = [it for it in result.input if it.get("type") == "function_call_output"]
+ assert outputs[0]["output"].endswith("... [truncated]")
+ assert any(entry["event"] == "copilot_recent_tool_output_truncated" for entry in logs)
+
+ def test_emergency_truncation_logs_distinct_event_with_count(self) -> None:
+ import structlog.testing
+
+ from skyvern.forge.sdk.copilot.session_factory import make_copilot_call_model_input_filter
+
+ items: list[dict[str, Any]] = [
+ {"role": "user", "content": "please build me a workflow"},
+ *({"type": "function_call_output", "call_id": f"call-{i}", "output": "x" * 5000} for i in range(3)),
+ {"type": "function_call_output", "call_id": "call-small", "output": "ok"},
+ ]
+ tight_filter = make_copilot_call_model_input_filter(token_budget=200)
+ with structlog.testing.capture_logs() as logs:
+ tight_filter(_mk_input_data(items))
+ emergency = [entry for entry in logs if entry["event"] == "copilot_tool_output_emergency_truncated"]
+ assert [entry["cap"] for entry in emergency] == [2000, 300]
+ assert all(entry["truncated_count"] >= 2 for entry in emergency)
+
+ def test_soft_emergency_rung_spares_code_when_it_fits(self) -> None:
+ import structlog.testing
+
+ from skyvern.forge.sdk.copilot.session_factory import make_copilot_call_model_input_filter
+
+ items: list[dict[str, Any]] = [
+ {"role": "user", "content": "please build me a workflow"},
+ {"type": "function_call_output", "call_id": "call-big", "output": "x" * 40_000},
+ ]
+ soft_filter = make_copilot_call_model_input_filter(token_budget=800)
+ with structlog.testing.capture_logs() as logs:
+ result = soft_filter(_mk_input_data(items))
+ emergency = [entry for entry in logs if entry["event"] == "copilot_tool_output_emergency_truncated"]
+ assert [entry["cap"] for entry in emergency] == [2000]
+ outputs = [it for it in result.input if it.get("type") == "function_call_output"]
+ assert 2000 <= len(outputs[0]["output"]) <= 2020
+
def test_filter_summarizes_older_function_call_args_on_first_turn(self) -> None:
"""F3/CORR-2 guard: older `function_call` items get their bulky
``arguments`` payload (e.g. a full workflow YAML) compacted, exactly
@@ -200,3 +265,164 @@ class TestModelInputCapture:
copilot_call_model_input_filter(_mk_input_data(items, context=RunContextWrapper(context=SimpleNamespace())))
assert len(sorted(tmp_path.glob("call-*.json"))) == 2
+
+
+class TestSynthesizedOfferCollapse:
+ """The model-input filter keeps at most one synthesized-block offer, and
+ offers do not count as real user turns for the session callback's
+ recent-turn boundary."""
+
+ @staticmethod
+ def _offer(body: str) -> dict[str, Any]:
+ from skyvern.forge.sdk.copilot.code_block_synthesis import SYNTHESIZED_OFFER_SENTINEL
+
+ return {"role": "user", "content": SYNTHESIZED_OFFER_SENTINEL + " " + body}
+
+ def test_filter_keeps_at_most_one_offer(self) -> None:
+ from skyvern.forge.sdk.copilot.session_factory import copilot_call_model_input_filter
+
+ code = "await page.goto('https://a.example')\n" * 200
+ items: list[dict[str, Any]] = [{"role": "user", "content": "build me a workflow"}]
+ for i in range(6):
+ items.append(self._offer(f"v{i} " + code))
+ items.append({"type": "function_call", "call_id": f"c-{i}", "name": "evaluate", "arguments": "{}"})
+ items.append({"type": "function_call_output", "call_id": f"c-{i}", "output": '{"ok": true}'})
+
+ result = copilot_call_model_input_filter(_mk_input_data(items))
+ offers = [
+ it for it in result.input if isinstance(it, dict) and "SYNTHESIZED CODE BLOCK" in str(it.get("content"))
+ ]
+ assert len(offers) == 1
+ assert offers[0]["content"].startswith("SYNTHESIZED CODE BLOCK")
+ assert "v5 " in offers[0]["content"]
+
+ def test_production_shape_stale_offers_in_history_fresh_offer_in_new_items(self) -> None:
+ """The shape the enforcement loop actually emits: prior offers sit in session
+ history while the refreshed offer arrives beside the screenshot and nudge, and
+ only the refreshed one may reach the model."""
+ from skyvern.forge.sdk.copilot.enforcement import NUDGE_SENTINEL
+ from skyvern.forge.sdk.copilot.session_factory import (
+ copilot_call_model_input_filter,
+ copilot_session_input_callback,
+ )
+
+ history: list[dict[str, Any]] = [
+ {"role": "user", "content": "please build me a workflow"},
+ self._offer("stale one"),
+ {"type": "function_call", "call_id": "c-1", "name": "evaluate", "arguments": "{}"},
+ {"type": "function_call_output", "call_id": "c-1", "output": '{"ok": true}'},
+ self._offer("stale two"),
+ ]
+ new_items: list[dict[str, Any]] = [
+ self._offer("fresh"),
+ self._screenshot("live frame"),
+ {"role": "user", "content": NUDGE_SENTINEL + "keep going"},
+ ]
+
+ merged = copilot_session_input_callback(history, new_items)
+ result = copilot_call_model_input_filter(_mk_input_data(merged))
+
+ offers = [
+ it for it in result.input if isinstance(it, dict) and "SYNTHESIZED CODE BLOCK" in str(it.get("content"))
+ ]
+ assert len(offers) == 1
+ assert "fresh" in offers[0]["content"]
+
+ def test_offers_do_not_hold_recent_real_turn_slots(self) -> None:
+ """Offers interleaved with the tool chain used to hold the recent-real-turn
+ slots and shield everything after them from middle-region compaction; as
+ synthetic messages the older tool outputs compact while pairing survives."""
+ from skyvern.forge.sdk.copilot.enforcement import KEEP_RECENT_TOOL_OUTPUTS
+ from skyvern.forge.sdk.copilot.session_factory import copilot_session_input_callback
+
+ goal = {"role": "user", "content": "please build me a workflow"}
+
+ def pair(i: int) -> list[dict[str, Any]]:
+ return [
+ {
+ "type": "function_call",
+ "call_id": f"c-{i}",
+ "name": "evaluate",
+ "arguments": json.dumps({"blob": "z" * 4000}),
+ },
+ {
+ "type": "function_call_output",
+ "call_id": f"c-{i}",
+ "output": json.dumps({"ok": True, "data": {"blob": "z" * 4000}}),
+ },
+ ]
+
+ history: list[dict[str, Any]] = [goal, *pair(0), *pair(1), self._offer("stale")]
+ history += [*pair(2), *pair(3), self._offer("fresh"), *pair(4), *pair(5)]
+
+ combined = copilot_session_input_callback(history, [{"role": "user", "content": "[copilot:nudge] finish"}])
+
+ outputs = [it for it in combined if isinstance(it, dict) and it.get("type") == "function_call_output"]
+ assert len(outputs) == 6
+ for older in outputs[:-KEEP_RECENT_TOOL_OUTPUTS]:
+ assert "_summarized" in older["output"]
+ for recent in outputs[-KEEP_RECENT_TOOL_OUTPUTS:]:
+ assert "_summarized" not in recent["output"]
+ # Pairing survives: every retained output still has its call.
+ call_ids = {it["call_id"] for it in combined if isinstance(it, dict) and it.get("type") == "function_call"}
+ assert {out["call_id"] for out in outputs} <= call_ids
+
+ @staticmethod
+ def _screenshot(label: str) -> dict[str, Any]:
+ from skyvern.forge.sdk.copilot.enforcement import SCREENSHOT_SENTINEL
+
+ return {
+ "role": "user",
+ "content": [
+ {"type": "input_text", "text": SCREENSHOT_SENTINEL + " " + label},
+ {"type": "input_image", "image_url": "data:image/png;base64,AAAA"},
+ ],
+ }
+
+ def test_newest_history_screenshot_survives_middle_compaction(self) -> None:
+ """With every injected message synthetic the whole post-goal history is
+ middle, and the newest frame must not collapse to a placeholder."""
+ from skyvern.forge.sdk.copilot.enforcement import is_screenshot_message
+ from skyvern.forge.sdk.copilot.session_factory import copilot_session_input_callback
+
+ history = [
+ {"role": "user", "content": "please build me a workflow"},
+ self._screenshot("old frame"),
+ {"type": "function_call", "call_id": "c-1", "name": "evaluate", "arguments": "{}"},
+ {"type": "function_call_output", "call_id": "c-1", "output": "{}"},
+ self._screenshot("newest frame"),
+ self._offer("fresh"),
+ ]
+ combined = copilot_session_input_callback(history, [{"role": "user", "content": "[copilot:nudge] finish"}])
+ frames = [it for it in combined if is_screenshot_message(it) and isinstance(it.get("content"), list)]
+ assert len(frames) == 1
+ assert "newest frame" in frames[0]["content"][0]["text"]
+
+ def test_text_only_screenshot_lookalike_cannot_shadow_the_real_frame(self) -> None:
+ from skyvern.forge.sdk.copilot.enforcement import SCREENSHOT_SENTINEL, is_screenshot_message
+ from skyvern.forge.sdk.copilot.session_factory import copilot_session_input_callback
+
+ history = [
+ {"role": "user", "content": "please build me a workflow"},
+ self._screenshot("real frame"),
+ {"role": "user", "content": SCREENSHOT_SENTINEL + " text-only lookalike"},
+ self._offer("fresh"),
+ ]
+ combined = copilot_session_input_callback(history, [{"role": "user", "content": "[copilot:nudge] finish"}])
+ frames = [it for it in combined if is_screenshot_message(it) and isinstance(it.get("content"), list)]
+ assert len(frames) == 1
+ assert "real frame" in frames[0]["content"][0]["text"]
+
+ def test_history_screenshot_yields_to_newer_frame_in_new_items(self) -> None:
+ from skyvern.forge.sdk.copilot.enforcement import is_screenshot_message
+ from skyvern.forge.sdk.copilot.session_factory import copilot_session_input_callback
+
+ history = [
+ {"role": "user", "content": "please build me a workflow"},
+ self._screenshot("old frame"),
+ self._offer("fresh"),
+ ]
+ combined = copilot_session_input_callback(history, [self._screenshot("live frame")])
+ list_frames = [it for it in combined if is_screenshot_message(it) and isinstance(it.get("content"), list)]
+ assert len(list_frames) == 1
+ assert "live frame" in list_frames[0]["content"][0]["text"]
diff --git a/tests/unit/test_evaluate_actionable_target.py b/tests/unit/test_evaluate_actionable_target.py
index 7e073167e..14632338f 100644
--- a/tests/unit/test_evaluate_actionable_target.py
+++ b/tests/unit/test_evaluate_actionable_target.py
@@ -373,6 +373,18 @@ async def test_hash_route_change_not_steered(monkeypatch) -> None:
assert "next_action" not in second["data"]
+def test_recon_result_keeps_raw_payload_within_its_own_ceiling() -> None:
+ moderate = {"ok": True, "data": {"result": {"bodyText": "y" * 10_000}, "next_action_reason": "recon"}}
+ scouting._fit_evaluate_steer_under_cap(moderate, moderate["data"], keep_raw_page_payload=True)
+ assert moderate["data"]["result"]["bodyText"] == "y" * 10_000
+
+
+def test_recon_result_over_ceiling_sheds_down_to_recon_cap() -> None:
+ huge = {"ok": True, "data": {"result": "z" * 60_000, "raw_page_text": "w" * 30_000}}
+ scouting._fit_evaluate_steer_under_cap(huge, huge["data"], keep_raw_page_payload=True)
+ assert len(json.dumps(huge, default=str)) <= scouting._SCOUT_RECON_RESULT_CHAR_CAP
+
+
@pytest.mark.asyncio
async def test_repeat_scalar_result_sheds_under_cap(monkeypatch) -> None:
ctx = _ctx()
@@ -392,8 +404,8 @@ async def test_repeat_scalar_result_sheds_under_cap(monkeypatch) -> None:
serialized = json.dumps(second, default=str)
assert second["data"].get("next_action") == "click"
assert second["data"].get("actionable_targets")
- assert len(serialized) <= scouting._RECENT_TOOL_OUTPUT_CHAR_CAP
- assert '"next_action"' in serialized[: scouting._RECENT_TOOL_OUTPUT_CHAR_CAP]
+ assert len(serialized) <= scouting._SCOUT_RESULT_CHAR_CAP
+ assert '"next_action"' in serialized[: scouting._SCOUT_RESULT_CHAR_CAP]
@pytest.mark.asyncio
@@ -417,8 +429,8 @@ async def test_first_loaded_result_table_sheds_large_payload_but_keeps_compositi
assert ctx.latest_recorded_build_test_outcome.phase == "scout_evaluate"
assert ctx.latest_recorded_build_test_outcome.reason_code == "loaded_result_targets_observed"
assert ctx.latest_recorded_build_test_outcome.structural_key is not None
- assert len(serialized) <= scouting._RECENT_TOOL_OUTPUT_CHAR_CAP
- assert '"next_action"' in serialized[: scouting._RECENT_TOOL_OUTPUT_CHAR_CAP]
+ assert len(serialized) <= scouting._SCOUT_RESULT_CHAR_CAP
+ assert '"next_action"' in serialized[: scouting._SCOUT_RESULT_CHAR_CAP]
assert "compose_extraction" in serialized
@@ -448,7 +460,7 @@ async def test_loaded_result_composition_targets_are_cap_aware_for_max_shaped_ev
serialized = json.dumps(result, default=str)
composition_targets = result["data"]["composition_targets"]
target = composition_targets["targets"][0]
- assert len(serialized) <= scouting._RECENT_TOOL_OUTPUT_CHAR_CAP
+ assert len(serialized) <= scouting._SCOUT_RESULT_CHAR_CAP
assert result["data"]["next_action"] == "compose_extraction"
assert composition_targets["result_container_count"] == 8
assert composition_targets["table_result_container_count"] == 8
@@ -490,7 +502,7 @@ async def test_single_loaded_result_structural_target_fits_evaluate_cap_with_url
serialized = json.dumps(result, default=str)
composition_targets = result["data"]["composition_targets"]
target = composition_targets["targets"][0]
- assert len(serialized) <= scouting._RECENT_TOOL_OUTPUT_CHAR_CAP
+ assert len(serialized) <= scouting._SCOUT_RESULT_CHAR_CAP
assert result["data"]["next_action"] == "compose_extraction"
assert composition_targets["result_container_count"] == 1
assert composition_targets["table_result_container_count"] == 1
@@ -537,8 +549,8 @@ async def test_repeat_nested_result_sheds_under_cap(monkeypatch) -> None:
serialized = json.dumps(second, default=str)
assert second["data"].get("next_action") == "click"
assert second["data"].get("actionable_targets")
- assert len(serialized) <= scouting._RECENT_TOOL_OUTPUT_CHAR_CAP
- assert '"next_action"' in serialized[: scouting._RECENT_TOOL_OUTPUT_CHAR_CAP]
+ assert len(serialized) <= scouting._SCOUT_RESULT_CHAR_CAP
+ assert '"next_action"' in serialized[: scouting._SCOUT_RESULT_CHAR_CAP]
@pytest.mark.asyncio
@@ -901,7 +913,7 @@ async def test_single_low_tier_link_auto_acts(monkeypatch) -> None:
assert second["data"]["page"]["page_title"] == "Statement"
assert "next_action" not in second["data"]
assert "actionable_targets" not in second["data"]
- assert len(json.dumps(second, default=str)) <= scouting._RECENT_TOOL_OUTPUT_CHAR_CAP
+ assert len(json.dumps(second, default=str)) <= scouting._SCOUT_RESULT_CHAR_CAP
@pytest.mark.asyncio
@@ -929,7 +941,7 @@ async def test_auto_act_post_evidence_none_sheds_bulky_result_under_cap(monkeypa
assert second["data"]["auto_acted"]["selector"] == "#stmt"
assert second["data"]["auto_acted"].get("note")
assert "page" not in second["data"]
- assert len(json.dumps(second, default=str)) <= scouting._RECENT_TOOL_OUTPUT_CHAR_CAP
+ assert len(json.dumps(second, default=str)) <= scouting._SCOUT_RESULT_CHAR_CAP
@pytest.mark.asyncio
@@ -975,7 +987,7 @@ async def test_auto_act_success_branch_sheds_oversized_page_under_cap(monkeypatc
assert [call[0] for call in server.calls].count("skyvern_click") == 1
assert "auto_acted" in second["data"]
assert "page" in second["data"]
- assert len(json.dumps(second, default=str)) <= scouting._RECENT_TOOL_OUTPUT_CHAR_CAP
+ assert len(json.dumps(second, default=str)) <= scouting._SCOUT_RESULT_CHAR_CAP
@pytest.mark.asyncio
diff --git a/tests/unit/test_finally_block_dag.py b/tests/unit/test_finally_block_dag.py
index 3de8a4dd8..b778063cf 100644
--- a/tests/unit/test_finally_block_dag.py
+++ b/tests/unit/test_finally_block_dag.py
@@ -100,6 +100,31 @@ class TestStripFinallyBlockReferences:
"Branch pointing to finally_block should have been nullified"
)
+ def test_conditional_own_next_block_label_pointing_to_finally_is_nullified(self):
+ block_1 = _make_task_block("block_1")
+ cond_block = ConditionalBlock(
+ label="cond_block",
+ output_parameter=_make_output_parameter("cond_block"),
+ next_block_label="finally_block",
+ branch_conditions=[
+ BranchCondition(next_block_label="block_1", is_default=True),
+ BranchCondition(
+ next_block_label="finally_block",
+ criteria={"criteria_type": "jinja2_template", "expression": "{{ true }}"},
+ ),
+ ],
+ )
+ finally_block = _make_http_block("finally_block")
+
+ result = WorkflowService._strip_finally_block_references(
+ [block_1, cond_block, finally_block],
+ "finally_block",
+ )
+
+ cond = next(b for b in result if b.label == "cond_block")
+ assert cond.next_block_label is None
+ assert [branch.next_block_label for branch in cond.branch_conditions] == ["block_1", None]
+
def test_noop_when_no_finally_block(self):
block_1 = _make_task_block("block_1", next_block_label="block_2")
block_2 = _make_task_block("block_2")
@@ -160,6 +185,45 @@ class TestBuildWorkflowGraphWithFinallyBlock:
assert start_label == "cond_block"
assert set(label_to_block.keys()) == {"cond_block", "block_1"}
+ def test_dag_validation_with_conditional_whose_merge_point_is_the_finally_block(self):
+ """A conditional whose own next_block_label is the finally block must still build.
+
+ The stale merge point used to be re-pointed onto every branch terminal by
+ resolve_conditional_merge_edges, so the graph build blamed a terminal block
+ the user never edited.
+ """
+ cond_block = ConditionalBlock(
+ label="cond_block",
+ output_parameter=_make_output_parameter("cond_block"),
+ next_block_label="finally_block",
+ branch_conditions=[
+ BranchCondition(next_block_label="branch_a_terminal", is_default=True),
+ BranchCondition(
+ next_block_label="branch_b_terminal",
+ criteria={"criteria_type": "jinja2_template", "expression": "{{ true }}"},
+ ),
+ ],
+ )
+ branch_a_terminal = _make_task_block("branch_a_terminal")
+ branch_b_terminal = _make_task_block("branch_b_terminal")
+ finally_block = _make_http_block("finally_block")
+
+ dag_blocks = WorkflowService._strip_finally_block_references(
+ [cond_block, branch_a_terminal, branch_b_terminal, finally_block],
+ "finally_block",
+ )
+
+ svc = WorkflowService()
+ start_label, label_to_block, default_next_map = svc._build_workflow_graph(dag_blocks)
+
+ assert start_label == "cond_block"
+ assert set(label_to_block.keys()) == {"cond_block", "branch_a_terminal", "branch_b_terminal"}
+ # No merge point survives the strip, so both branch terminals end the traversal
+ # and the finally block runs out-of-band.
+ assert default_next_map["cond_block"] is None
+ assert default_next_map["branch_a_terminal"] is None
+ assert default_next_map["branch_b_terminal"] is None
+
def test_dag_validation_without_finally_block(self):
block_1 = _make_task_block("block_1", next_block_label="block_2")
block_2 = _make_task_block("block_2")
diff --git a/tests/unit/test_quickstart_command.py b/tests/unit/test_quickstart_command.py
index 0b434abef..51127157a 100644
--- a/tests/unit/test_quickstart_command.py
+++ b/tests/unit/test_quickstart_command.py
@@ -1,6 +1,7 @@
from __future__ import annotations
import json
+import logging
import os
import re
import signal
@@ -368,7 +369,7 @@ def test_run_mcp_prepares_cloud_env_before_starting_mcp(tmp_path, monkeypatch) -
@pytest.mark.parametrize("failure_stage", ["import", "call"])
-def test_run_mcp_logs_sweep_failure_and_continues(monkeypatch, failure_stage: str) -> None:
+def test_run_mcp_logs_sweep_failure_and_continues(monkeypatch, caplog, failure_stage: str) -> None:
events: list[str] = []
run_commands = _patch_minimal_run_mcp_dependencies(monkeypatch, events, lambda: events.append("run"))
@@ -389,11 +390,20 @@ def test_run_mcp_logs_sweep_failure_and_continues(monkeypatch, failure_stage: st
fake_local_browser_profile.sweep_local_browser_profiles_with_budget = failing_sweep
monkeypatch.setitem(sys.modules, "skyvern.library.local_browser_profile", fake_local_browser_profile)
- with capture_logs() as logs:
+ with capture_logs() as logs, caplog.at_level(logging.WARNING, logger="skyvern.cli.run_commands"):
run_commands.run_mcp()
assert events == ["prepare", "run", "cleanup"]
- assert any(log.get("log_level") == "warning" for log in logs)
+ structlog_warning = any(
+ log.get("event") == "local_browser_profile_startup_sweep_failed"
+ and (log.get("log_level") or log.get("level")) == "warning"
+ for log in logs
+ )
+ stdlib_warning = any(
+ record.levelno == logging.WARNING and "local_browser_profile_startup_sweep_failed" in record.getMessage()
+ for record in caplog.records
+ )
+ assert structlog_warning or stdlib_warning
def test_run_mcp_starts_within_sweep_budget_and_stops_blocking_child(tmp_path: Path, monkeypatch) -> None:
diff --git a/tests/unit/test_seed_precedence_engine.py b/tests/unit/test_seed_precedence_engine.py
index 36cd3c5e3..26a99b5fc 100644
--- a/tests/unit/test_seed_precedence_engine.py
+++ b/tests/unit/test_seed_precedence_engine.py
@@ -19,6 +19,7 @@ from skyvern.forge.sdk.workflow.models.parameter import (
from skyvern.forge.sdk.workflow.service import WorkflowService
from skyvern.schemas.runs import ProxyLocation
from skyvern.services.workflow_service import workflow_request_body_from_existing_run
+from skyvern.webeye.browser_artifacts import BrowserArtifacts
def _workflow(*, persist: bool = False, pick: str | None = None, key: str | None = None) -> SimpleNamespace:
@@ -1164,7 +1165,9 @@ async def test_login_block_cached_browser_degrades_instead_of_credential_seeding
AsyncMock(return_value=SimpleNamespace(incompatible_reason=None, attach_browser_session_id=None)),
)
monkeypatch.setattr(app.AGENT_FUNCTION, "is_browser_memory_engine_enabled", AsyncMock(return_value=True))
- monkeypatch.setattr(app.BROWSER_MANAGER, "get_for_workflow_run", lambda *a, **k: object()) # browser exists
+ # Browser exists but was NOT booted from the resolved profile (applied id is None).
+ open_state = SimpleNamespace(browser_artifacts=BrowserArtifacts())
+ monkeypatch.setattr(app.BROWSER_MANAGER, "get_for_workflow_run", lambda *a, **k: open_state)
get_or_create = AsyncMock()
monkeypatch.setattr(app.BROWSER_MANAGER, "get_or_create_for_workflow_run", get_or_create)
update = AsyncMock()
@@ -1236,6 +1239,157 @@ async def test_login_block_no_cached_browser_still_credential_seeds(monkeypatch:
)
+async def test_login_block_open_browser_booted_from_profile_keeps_credential_seed(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ # A script/setup boot already applied the setup-stamped credential profile; the loader must keep
+ # credential provenance instead of degrading a genuinely-loaded seed.
+ svc = WorkflowService()
+ monkeypatch.setattr(svc, "_apply_login_block_credential_proxy_pin", AsyncMock())
+ monkeypatch.setattr(svc, "_resolve_login_block_browser_profile_id", AsyncMock(return_value="cred_profile_x"))
+ monkeypatch.setattr(
+ svc,
+ "_evaluate_debug_session_profile_decision",
+ AsyncMock(return_value=SimpleNamespace(incompatible_reason=None, attach_browser_session_id=None)),
+ )
+ monkeypatch.setattr(app.AGENT_FUNCTION, "is_browser_memory_engine_enabled", AsyncMock(return_value=True))
+ open_state = SimpleNamespace(browser_artifacts=BrowserArtifacts(applied_browser_profile_id="cred_profile_x"))
+ monkeypatch.setattr(app.BROWSER_MANAGER, "get_for_workflow_run", lambda *a, **k: open_state)
+ get_or_create = AsyncMock()
+ monkeypatch.setattr(app.BROWSER_MANAGER, "get_or_create_for_workflow_run", get_or_create)
+ update = AsyncMock()
+ monkeypatch.setattr(app.DATABASE.workflow_runs, "update_workflow_run", update)
+ reloaded = SimpleNamespace(
+ browser_seed_source=BrowserSeedSource.credential,
+ browser_profile_id="cred_profile_x",
+ start_fresh_browser=False,
+ workflow_permanent_id="wpid",
+ )
+ monkeypatch.setattr(app.DATABASE.workflow_runs, "get_workflow_run", AsyncMock(return_value=reloaded))
+
+ block = SimpleNamespace(navigation_goal="log in", url="https://site.example/home", label="login")
+ run = SimpleNamespace(
+ browser_seed_source=BrowserSeedSource.credential, start_fresh_browser=False, workflow_permanent_id="wpid"
+ )
+ result = await svc._prepare_login_block_browser_profile(
+ block=block, # type: ignore[arg-type]
+ workflow_run=run, # type: ignore[arg-type]
+ workflow_run_id="wr_test",
+ organization_id="o_test",
+ browser_session_id=None,
+ )
+
+ update.assert_awaited_once_with(
+ workflow_run_id="wr_test",
+ browser_profile_id="cred_profile_x",
+ browser_seed_source=BrowserSeedSource.credential,
+ )
+ get_or_create.assert_not_awaited() # the open browser already holds the profile
+ assert result.browser_seed_source == BrowserSeedSource.credential
+ assert block.navigation_goal.startswith("A saved browser session has been loaded.")
+
+
+async def test_login_block_boot_degrades_when_profile_not_applied(monkeypatch: pytest.MonkeyPatch) -> None:
+ # The created browser (e.g. a remote/vendor type) accepted the profile id but never loaded it —
+ # the run must degrade instead of staying credential-stamped over an unrelated context.
+ get_or_create = AsyncMock(
+ return_value=SimpleNamespace(
+ get_working_page=AsyncMock(return_value=None),
+ browser_artifacts=BrowserArtifacts(),
+ )
+ )
+ degraded_run = SimpleNamespace(browser_seed_source=BrowserSeedSource.degraded_fresh, browser_profile_id=None)
+
+ result, update, _ = await _prepare_login_block_profile_boot(
+ monkeypatch,
+ url="https://login.example/session",
+ values={},
+ get_or_create=get_or_create,
+ get_workflow_run=AsyncMock(side_effect=[None, degraded_run]),
+ )
+
+ assert update.await_args_list[-1].kwargs == {
+ "workflow_run_id": "wr_test",
+ "browser_profile_id": None,
+ "browser_seed_source": BrowserSeedSource.degraded_fresh,
+ }
+ assert result is degraded_run
+
+
+async def test_login_block_boot_keeps_credential_when_profile_applied(monkeypatch: pytest.MonkeyPatch) -> None:
+ page = SimpleNamespace(url="https://login.example/session", wait_for_load_state=AsyncMock())
+ get_or_create = AsyncMock(
+ return_value=SimpleNamespace(
+ get_working_page=AsyncMock(return_value=page),
+ browser_artifacts=BrowserArtifacts(applied_browser_profile_id="cred_profile_x"),
+ )
+ )
+
+ _, update, block = await _prepare_login_block_profile_boot(
+ monkeypatch,
+ url="https://login.example/session",
+ values={},
+ get_or_create=get_or_create,
+ )
+
+ update.assert_awaited_once_with(
+ workflow_run_id="wr_test",
+ browser_profile_id="cred_profile_x",
+ browser_seed_source=BrowserSeedSource.credential,
+ )
+ assert block.navigation_goal.startswith("A saved browser session has been loaded.")
+
+
+async def test_login_block_boot_skips_not_applied_check_for_debug_session_attach(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ # A compatible debug-session attach returns the PBS browser, whose artifacts do not carry the
+ # profile id; compat was already evaluated, so the not-applied degrade must not fire.
+ svc = WorkflowService()
+ monkeypatch.setattr(svc, "_apply_login_block_credential_proxy_pin", AsyncMock())
+ monkeypatch.setattr(svc, "_resolve_login_block_browser_profile_id", AsyncMock(return_value="cred_profile_x"))
+ monkeypatch.setattr(
+ svc,
+ "_evaluate_debug_session_profile_decision",
+ AsyncMock(return_value=SimpleNamespace(incompatible_reason=None, attach_browser_session_id="pbs_1")),
+ )
+ monkeypatch.setattr(app.AGENT_FUNCTION, "is_browser_memory_engine_enabled", AsyncMock(return_value=True))
+ monkeypatch.setattr(app.BROWSER_MANAGER, "get_for_workflow_run", lambda *a, **k: None)
+ page = SimpleNamespace(url="https://site.example/home", wait_for_load_state=AsyncMock())
+ get_or_create = AsyncMock(
+ return_value=SimpleNamespace(
+ get_working_page=AsyncMock(return_value=page),
+ browser_artifacts=BrowserArtifacts(),
+ )
+ )
+ monkeypatch.setattr(app.BROWSER_MANAGER, "get_or_create_for_workflow_run", get_or_create)
+ update = AsyncMock()
+ monkeypatch.setattr(app.DATABASE.workflow_runs, "update_workflow_run", update)
+ monkeypatch.setattr(app.DATABASE.workflow_runs, "get_workflow_run", AsyncMock(return_value=None))
+
+ block = SimpleNamespace(navigation_goal="log in", url="https://site.example/home", label="login")
+ run = SimpleNamespace(
+ browser_seed_source=BrowserSeedSource.credential, start_fresh_browser=False, workflow_permanent_id="wpid"
+ )
+ await svc._prepare_login_block_browser_profile(
+ block=block, # type: ignore[arg-type]
+ workflow_run=run, # type: ignore[arg-type]
+ workflow_run_id="wr_test",
+ organization_id="o_test",
+ browser_session_id=None,
+ )
+
+ update.assert_awaited_once_with(
+ workflow_run_id="wr_test",
+ browser_profile_id="cred_profile_x",
+ browser_seed_source=BrowserSeedSource.credential,
+ )
+
+
async def _prepare_login_block_profile_boot(
monkeypatch: pytest.MonkeyPatch,
*,
@@ -1357,7 +1511,12 @@ async def test_login_block_profile_boot_resolves_url(
expected_url: str,
) -> None:
page = SimpleNamespace(url=expected_url, wait_for_load_state=AsyncMock())
- get_or_create = AsyncMock(return_value=SimpleNamespace(get_working_page=AsyncMock(return_value=page)))
+ get_or_create = AsyncMock(
+ return_value=SimpleNamespace(
+ get_working_page=AsyncMock(return_value=page),
+ browser_artifacts=BrowserArtifacts(applied_browser_profile_id="cred_profile_x"),
+ )
+ )
_, _, block = await _prepare_login_block_profile_boot(
monkeypatch,
diff --git a/tests/unit/test_session_cookies.py b/tests/unit/test_session_cookies.py
index a3d1ac4c4..6cfccfdb4 100644
--- a/tests/unit/test_session_cookies.py
+++ b/tests/unit/test_session_cookies.py
@@ -9,9 +9,11 @@ from typing import cast
import pytest
from playwright.async_api import BrowserContext
+from skyvern.webeye.profile_cookie_merge import BANKED_COOKIES_FILENAME
from skyvern.webeye.session_cookies import (
SESSION_COOKIES_FILENAME,
persist_session_cookies,
+ refresh_banked_cookies,
restore_session_cookies,
)
@@ -120,3 +122,28 @@ async def test_restore_tolerates_corrupt_sidecar(tmp_path: Path) -> None:
fake = FakeContext()
await restore_session_cookies(_ctx(fake), str(tmp_path))
assert fake.added == []
+
+
+def _banked(tmp_path: Path) -> Path:
+ return tmp_path / BANKED_COOKIES_FILENAME
+
+
+async def test_refresh_banked_cookies_replaces_the_sidecar_with_the_live_jar(tmp_path: Path) -> None:
+ _banked(tmp_path).write_text(json.dumps([{**_SESSION, "name": "seed_era"}]))
+
+ await refresh_banked_cookies(_ctx(FakeContext([_PERSISTENT])), str(tmp_path))
+
+ assert json.loads(_banked(tmp_path).read_text()) == [_PERSISTENT]
+
+
+async def test_refresh_banked_cookies_drops_the_sidecar_without_a_live_context(tmp_path: Path) -> None:
+ # A closed browser has flushed its Cookies database, so a seed-era sidecar must not ride along.
+ _banked(tmp_path).write_text(json.dumps([_PERSISTENT]))
+ await refresh_banked_cookies(None, str(tmp_path))
+ assert not _banked(tmp_path).exists()
+
+ _banked(tmp_path).write_text(json.dumps([_PERSISTENT]))
+ await refresh_banked_cookies(_ctx(RaisingContext()), str(tmp_path))
+ assert not _banked(tmp_path).exists()
diff --git a/tests/unit/test_workflow_service_profile_writeback.py b/tests/unit/test_workflow_service_profile_writeback.py
index 30d8a159d..84852910d 100644
--- a/tests/unit/test_workflow_service_profile_writeback.py
+++ b/tests/unit/test_workflow_service_profile_writeback.py
@@ -7,6 +7,8 @@ overwrite the shared S3 profile with their dirty state.
from __future__ import annotations
+import json
+from pathlib import Path
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
@@ -17,6 +19,7 @@ from skyvern.forge.sdk.workflow.browser_profile_key import build_workflow_browse
from skyvern.forge.sdk.workflow.models.workflow import WorkflowRunStatus
from skyvern.forge.sdk.workflow.service import WorkflowService
from skyvern.webeye.browser_manager import BrowserCleanupResult
+from skyvern.webeye.profile_cookie_merge import BANKED_COOKIES_FILENAME
def _make_workflow(persist: bool = True) -> MagicMock:
@@ -49,11 +52,12 @@ def _make_workflow_run(
return wr
-def _make_browser_state() -> MagicMock:
+def _make_browser_state(applied_browser_profile_id: str | None = None) -> MagicMock:
bs = MagicMock()
bs.browser_artifacts.browser_session_dir = "/tmp/fake_profile"
bs.browser_artifacts._seed_load_failed = False
bs.browser_artifacts._seed_capture_failed = False
+ bs.browser_artifacts.applied_browser_profile_id = applied_browser_profile_id
return bs
@@ -526,7 +530,7 @@ async def test_engine_on_writes_resolved_sink_profile(monkeypatch: pytest.Monkey
workflow_run = _make_workflow_run(
WorkflowRunStatus.completed, browser_profile_id="bp_seed", browser_sink_profile_id="bp_sink"
)
- store_session_mock = _patch_clean_up_deps(monkeypatch, _make_browser_state())
+ store_session_mock = _patch_clean_up_deps(monkeypatch, _make_browser_state(applied_browser_profile_id="bp_seed"))
monkeypatch.setattr(app.AGENT_FUNCTION, "is_browser_memory_engine_enabled", AsyncMock(return_value=True))
monkeypatch.setattr(app.AGENT_FUNCTION, "bank_credential_profile_on_healthy_run", AsyncMock())
@@ -543,6 +547,29 @@ async def test_engine_on_writes_resolved_sink_profile(monkeypatch: pytest.Monkey
store_session_mock.assert_not_awaited()
+async def test_engine_on_skips_sink_write_when_stamped_seed_not_applied(monkeypatch: pytest.MonkeyPatch) -> None:
+ """A stamped seed the browser never applied (vendor-routed boot) must not be overwritten by an
+ unrelated directory; seedless accumulate rows are unaffected (browser_profile_id None)."""
+ from skyvern.forge.sdk.workflow.service import WorkflowService
+
+ workflow = _make_workflow(persist=True)
+ workflow_run = _make_workflow_run(
+ WorkflowRunStatus.completed, browser_profile_id="bp_seed", browser_sink_profile_id="bp_sink"
+ )
+ _patch_clean_up_deps(monkeypatch, _make_browser_state(applied_browser_profile_id=None))
+ monkeypatch.setattr(app.AGENT_FUNCTION, "is_browser_memory_engine_enabled", AsyncMock(return_value=True))
+ monkeypatch.setattr(app.AGENT_FUNCTION, "bank_credential_profile_on_healthy_run", AsyncMock())
+
+ svc = WorkflowService()
+ monkeypatch.setattr(svc, "persist_video_data", AsyncMock())
+ monkeypatch.setattr(svc, "get_tasks_by_workflow_run_id", AsyncMock(return_value=[]))
+
+ await svc.clean_up_workflow(workflow=workflow, workflow_run=workflow_run, need_call_webhook=False)
+
+ app.STORAGE.store_browser_profile.assert_not_awaited()
+
+
@pytest.mark.asyncio
async def test_flag_off_start_fresh_run_suppresses_own_memory_write(monkeypatch: pytest.MonkeyPatch) -> None:
"""A run that opted into a fresh browser writes no own-memory back, even flag-off + persist on."""
@@ -834,6 +861,33 @@ async def test_sink_writeback_suppressed_when_seed_profile_failed_to_load(monkey
store.assert_awaited_once()
+async def test_sink_full_write_drops_the_seed_era_banked_sidecar(
+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+ # The sidecar restored from the seed archive rides this dir into the sink profile, where every later
+ # boot would replay it over a fresher Cookies database. The closed browser's database is authoritative.
+ from skyvern.forge.sdk.workflow.service import WorkflowService
+
+ banked = tmp_path / BANKED_COOKIES_FILENAME
+ banked.write_text(json.dumps([{"name": "seed_era", "value": "x", "domain": "example.test", "path": "/"}]))
+ store = AsyncMock()
+ monkeypatch.setattr(app.STORAGE, "store_browser_profile", store)
+ wr = _make_workflow_run(WorkflowRunStatus.completed, browser_sink_profile_id="bp")
+ bs = _make_browser_state_b2(seed_cookies=[], seed_etag=None, fresh_login=False)
+ bs.browser_artifacts.browser_session_dir = str(tmp_path)
+
+ await WorkflowService()._persist_run_sink_profile_if_needed(
+ workflow_run=wr,
+ browser_state=bs,
+ close_browser_on_completion=True,
+ effective_workflow_run_status=WorkflowRunStatus.completed,
+ )
+
+ store.assert_awaited_once()
+ assert not banked.exists()
+
+
@pytest.mark.asyncio
async def test_sink_writeback_no_full_overwrite_when_seed_capture_failed(monkeypatch: pytest.MonkeyPatch) -> None:
# Lawy: an uncaptured seed fingerprint is UNKNOWN — the guard must never full-overwrite (a None seed
diff --git a/tests/unit/webeye/test_browser_session_response.py b/tests/unit/webeye/test_browser_session_response.py
index dfec9f625..5ed9f38ff 100644
--- a/tests/unit/webeye/test_browser_session_response.py
+++ b/tests/unit/webeye/test_browser_session_response.py
@@ -29,6 +29,7 @@ PINNED_CLIENT_FIELDS = frozenset(
"browser_profile_id",
"generate_browser_profile",
"vnc_streaming_supported",
+ "stream_transport",
"download_path",
"downloaded_files",
"recordings",
@@ -336,3 +337,73 @@ async def test_base_agent_function_preserves_the_existing_browser_session_addres
)
assert resolved_address == direct_address
+
+
+async def test_browser_session_response_carries_per_session_stream_transport() -> None:
+ now = datetime.now(timezone.utc)
+ session = PersistentBrowserSession(
+ persistent_browser_session_id="pbs_123",
+ organization_id="org_123",
+ status="active",
+ created_at=now,
+ modified_at=now,
+ )
+
+ with (
+ patch.object(app.AGENT_FUNCTION, "resolve_browser_session_connect_url", AsyncMock(return_value=None)),
+ patch.object(
+ app.AGENT_FUNCTION, "resolve_stream_transport", AsyncMock(return_value="cdp")
+ ) as transport_resolver,
+ ):
+ response = await BrowserSessionResponse.from_browser_session(session, include_stream_transport=True)
+
+ assert response.stream_transport == "cdp"
+ transport_resolver.assert_awaited_once_with(
+ browser_session_id="pbs_123", organization_id="org_123", ip_address=None
+ )
+
+
+async def test_browser_session_response_leaves_the_transport_unresolved_by_default() -> None:
+ """The list endpoints serialize an unpaginated set concurrently, so they must not each pay a
+ per-session infrastructure lookup — nor publish which sessions are hosted elsewhere."""
+ now = datetime.now(timezone.utc)
+ session = PersistentBrowserSession(
+ persistent_browser_session_id="pbs_123",
+ organization_id="org_123",
+ status="active",
+ created_at=now,
+ modified_at=now,
+ )
+
+ with (
+ patch.object(app.AGENT_FUNCTION, "resolve_browser_session_connect_url", AsyncMock(return_value=None)),
+ patch.object(
+ app.AGENT_FUNCTION, "resolve_stream_transport", AsyncMock(return_value="cdp")
+ ) as transport_resolver,
+ ):
+ response = await BrowserSessionResponse.from_browser_session(session)
+
+ assert response.stream_transport is None
+ transport_resolver.assert_not_awaited()
+
+
+async def test_browser_session_response_withholds_a_transport_outside_the_contract() -> None:
+ now = datetime.now(timezone.utc)
+ session = PersistentBrowserSession(
+ persistent_browser_session_id="pbs_123",
+ organization_id="org_123",
+ status="active",
+ created_at=now,
+ modified_at=now,
+ )
+
+ with (
+ patch.object(app.AGENT_FUNCTION, "resolve_browser_session_connect_url", AsyncMock(return_value=None)),
+ patch.object(app.AGENT_FUNCTION, "resolve_stream_transport", AsyncMock(return_value="webrtc")),
+ ):
+ response = await BrowserSessionResponse.from_browser_session(session, include_stream_transport=True)
+
+ assert response.stream_transport is None
diff --git a/tests/unit/workflow/test_execute_single_block_phases.py b/tests/unit/workflow/test_execute_single_block_phases.py
index cb491d522..b6d986175 100644
--- a/tests/unit/workflow/test_execute_single_block_phases.py
+++ b/tests/unit/workflow/test_execute_single_block_phases.py
@@ -28,6 +28,7 @@ from skyvern.forge.sdk.workflow.models.workflow import WorkflowRunStatus
from skyvern.forge.sdk.workflow.service import DebugSessionProfileDecision, WorkflowService
from skyvern.schemas.scripts import ScriptBlock
from skyvern.schemas.workflows import BlockResult, BlockStatus
+from skyvern.webeye.browser_artifacts import BrowserArtifacts
def _output_parameter(key: str) -> OutputParameter:
@@ -368,6 +369,7 @@ async def test_login_block_with_saved_profile_rewrites_goal_and_persists_profile
page.url = "https://example.com/home"
browser_state = AsyncMock()
browser_state.get_working_page = AsyncMock(return_value=page)
+ browser_state.browser_artifacts = BrowserArtifacts(applied_browser_profile_id="bp_123")
# No browser open yet: the credential profile loads into a fresh browser (the seed path this test
# covers). A pre-existing browser would instead degrade to fresh (see the cached-browser guard).
monkeypatch.setattr(app.BROWSER_MANAGER, "get_for_workflow_run", lambda *a, **k: None)
diff --git a/tests/unit_tests/test_streaming_screencast.py b/tests/unit_tests/test_streaming_screencast.py
index c78ec613a..dc6082776 100644
--- a/tests/unit_tests/test_streaming_screencast.py
+++ b/tests/unit_tests/test_streaming_screencast.py
@@ -26,7 +26,7 @@ async def test_resolve_browser_state_for_workflow_run(monkeypatch: pytest.Monkey
expected_state = object()
fake_app = _make_app(
browser_manager=SimpleNamespace(get_for_workflow_run=Mock(return_value=expected_state), get_for_task=Mock()),
- persistent_sessions_manager=SimpleNamespace(get_browser_state=AsyncMock()),
+ persistent_sessions_manager=SimpleNamespace(get_observer_browser_state=AsyncMock()),
)
monkeypatch.setattr(screencast, "app", fake_app)
@@ -35,7 +35,7 @@ async def test_resolve_browser_state_for_workflow_run(monkeypatch: pytest.Monkey
assert result is expected_state
fake_app.BROWSER_MANAGER.get_for_workflow_run.assert_called_once_with("wr_123")
fake_app.BROWSER_MANAGER.get_for_task.assert_not_called()
- fake_app.PERSISTENT_SESSIONS_MANAGER.get_browser_state.assert_not_awaited()
+ fake_app.PERSISTENT_SESSIONS_MANAGER.get_observer_browser_state.assert_not_awaited()
@pytest.mark.asyncio
@@ -43,7 +43,7 @@ async def test_resolve_browser_state_for_task(monkeypatch: pytest.MonkeyPatch) -
expected_state = object()
fake_app = _make_app(
browser_manager=SimpleNamespace(get_for_workflow_run=Mock(), get_for_task=Mock(return_value=expected_state)),
- persistent_sessions_manager=SimpleNamespace(get_browser_state=AsyncMock()),
+ persistent_sessions_manager=SimpleNamespace(get_observer_browser_state=AsyncMock()),
)
monkeypatch.setattr(screencast, "app", fake_app)
@@ -52,7 +52,7 @@ async def test_resolve_browser_state_for_task(monkeypatch: pytest.MonkeyPatch) -
assert result is expected_state
fake_app.BROWSER_MANAGER.get_for_task.assert_called_once_with("task_123", "wr_123")
fake_app.BROWSER_MANAGER.get_for_workflow_run.assert_not_called()
- fake_app.PERSISTENT_SESSIONS_MANAGER.get_browser_state.assert_not_awaited()
+ fake_app.PERSISTENT_SESSIONS_MANAGER.get_observer_browser_state.assert_not_awaited()
@pytest.mark.asyncio
@@ -60,14 +60,14 @@ async def test_resolve_browser_state_for_browser_session(monkeypatch: pytest.Mon
expected_state = object()
fake_app = _make_app(
browser_manager=SimpleNamespace(get_for_workflow_run=Mock(), get_for_task=Mock()),
- persistent_sessions_manager=SimpleNamespace(get_browser_state=AsyncMock(return_value=expected_state)),
+ persistent_sessions_manager=SimpleNamespace(get_observer_browser_state=AsyncMock(return_value=expected_state)),
)
monkeypatch.setattr(screencast, "app", fake_app)
result = await screencast._resolve_browser_state("bs_123", "browser_session", organization_id="o_123")
assert result is expected_state
- fake_app.PERSISTENT_SESSIONS_MANAGER.get_browser_state.assert_awaited_once_with("bs_123", "o_123")
+ fake_app.PERSISTENT_SESSIONS_MANAGER.get_observer_browser_state.assert_awaited_once_with("bs_123", "o_123")
fake_app.BROWSER_MANAGER.get_for_workflow_run.assert_not_called()
fake_app.BROWSER_MANAGER.get_for_task.assert_not_called()
@@ -76,7 +76,7 @@ async def test_resolve_browser_state_for_browser_session(monkeypatch: pytest.Mon
async def test_resolve_browser_state_unknown_entity_type(monkeypatch: pytest.MonkeyPatch) -> None:
fake_app = _make_app(
browser_manager=SimpleNamespace(get_for_workflow_run=Mock(), get_for_task=Mock()),
- persistent_sessions_manager=SimpleNamespace(get_browser_state=AsyncMock()),
+ persistent_sessions_manager=SimpleNamespace(get_observer_browser_state=AsyncMock()),
)
monkeypatch.setattr(screencast, "app", fake_app)
@@ -85,7 +85,7 @@ async def test_resolve_browser_state_unknown_entity_type(monkeypatch: pytest.Mon
assert result is None
fake_app.BROWSER_MANAGER.get_for_workflow_run.assert_not_called()
fake_app.BROWSER_MANAGER.get_for_task.assert_not_called()
- fake_app.PERSISTENT_SESSIONS_MANAGER.get_browser_state.assert_not_awaited()
+ fake_app.PERSISTENT_SESSIONS_MANAGER.get_observer_browser_state.assert_not_awaited()
@pytest.mark.asyncio
@@ -115,8 +115,8 @@ async def test_wait_for_browser_state_returns_none_on_timeout(monkeypatch: pytes
monkeypatch.setattr(screencast.asyncio, "sleep", sleep_mock)
result = await screencast.wait_for_browser_state(
- "bs_123",
- "browser_session",
+ "wr_123",
+ "workflow_run",
timeout=0.3,
poll_interval=0.1,
)
@@ -124,3 +124,91 @@ async def test_wait_for_browser_state_returns_none_on_timeout(monkeypatch: pytes
assert result is None
assert resolve_mock.await_count == 3
assert browser_state.get_working_page.await_count == 3
+
+
+ ("entity_id", "entity_type", "kwargs"),
+ [
+ ("wr_1", "workflow_run", {}),
+ ("tsk_1", "task", {"workflow_run_id": "wr_1"}),
+ ],
+)
+async def test_a_run_without_in_process_state_never_adopts_the_session(
+ monkeypatch: pytest.MonkeyPatch, entity_id: str, entity_type: str, kwargs: dict
+) -> None:
+ """A run's frames come from the worker that drives it, so the API process resolves a run only
+ against its own manager. Reaching into the session instead would adopt a browser the worker is
+ driving — and would miss anyway, since a task inside a workflow leases under the run's id."""
+ fake_app = _make_app(
+ browser_manager=SimpleNamespace(
+ get_for_workflow_run=Mock(return_value=None), get_for_task=Mock(return_value=None)
+ ),
+ persistent_sessions_manager=SimpleNamespace(
+ get_session_by_runnable_id=AsyncMock(),
+ get_browser_state=AsyncMock(),
+ get_observer_browser_state=AsyncMock(),
+ ),
+ )
+ monkeypatch.setattr(screencast, "app", fake_app)
+
+ assert await screencast._resolve_browser_state(entity_id, entity_type, organization_id="o_1", **kwargs) is None
+
+ fake_app.PERSISTENT_SESSIONS_MANAGER.get_session_by_runnable_id.assert_not_awaited()
+ fake_app.PERSISTENT_SESSIONS_MANAGER.get_browser_state.assert_not_awaited()
+ fake_app.PERSISTENT_SESSIONS_MANAGER.get_observer_browser_state.assert_not_awaited()
+
+
+async def test_waiting_out_a_session_holds_one_connection_and_gives_it_back(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """Each resolve mints its own connection, so re-resolving per poll would leave one adopted
+ browser per tick behind."""
+ browser_state = SimpleNamespace(get_working_page=AsyncMock(return_value=None))
+ release = AsyncMock()
+ fake_app = _make_app(
+ persistent_sessions_manager=SimpleNamespace(
+ get_observer_browser_state=AsyncMock(return_value=browser_state),
+ release_observer_browser_state=release,
+ ),
+ )
+ monkeypatch.setattr(screencast, "app", fake_app)
+ monkeypatch.setattr(screencast.asyncio, "sleep", AsyncMock())
+
+ result = await screencast.wait_for_browser_state(
+ "bs_123", "browser_session", organization_id="o_1", timeout=0.3, poll_interval=0.1
+ )
+
+ assert result is None
+ assert fake_app.PERSISTENT_SESSIONS_MANAGER.get_observer_browser_state.await_count == 1
+ assert browser_state.get_working_page.await_count == 3
+ release.assert_awaited_once_with("bs_123", browser_state)
+
+
+async def test_following_the_active_page_does_not_reconnect(monkeypatch: pytest.MonkeyPatch) -> None:
+ page = object()
+ browser_state = SimpleNamespace(get_working_page=AsyncMock(return_value=page))
+ fake_app = _make_app(
+ persistent_sessions_manager=SimpleNamespace(get_observer_browser_state=AsyncMock()),
+ )
+ monkeypatch.setattr(screencast, "app", fake_app)
+
+ result = await screencast._resolve_working_page(browser_state, "bs_123", "browser_session", organization_id="o_1")
+
+ assert result is page
+ fake_app.PERSISTENT_SESSIONS_MANAGER.get_observer_browser_state.assert_not_awaited()
+
+
+async def test_a_run_state_is_not_the_viewers_to_release(monkeypatch: pytest.MonkeyPatch) -> None:
+ release = AsyncMock()
+ fake_app = _make_app(
+ persistent_sessions_manager=SimpleNamespace(release_observer_browser_state=release),
+ )
+ monkeypatch.setattr(screencast, "app", fake_app)
+
+ await screencast.release_browser_state(SimpleNamespace(), "workflow_run", "wr_1")
+
+ release.assert_not_awaited()
diff --git a/tests/unit_tests/test_streaming_screenshot_local.py b/tests/unit_tests/test_streaming_screenshot_local.py
index 688eee287..b990e65a0 100644
--- a/tests/unit_tests/test_streaming_screenshot_local.py
+++ b/tests/unit_tests/test_streaming_screenshot_local.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+from types import SimpleNamespace
from unittest.mock import AsyncMock
import pytest
@@ -30,12 +31,13 @@ async def test_run_local_screencast_happy_path(monkeypatch: pytest.MonkeyPatch)
wait_for_running=wait_for_running,
check_finalized=check_finalized,
get_current_status=get_current_status,
+ organization_id="org_123",
get_workflow_run_id=lambda: "wr_123",
)
wait_for_running.assert_awaited_once()
wait_for_browser_state_mock.assert_awaited_once_with(
- "task_123", "task", workflow_run_id="wr_123", organization_id=None
+ "task_123", "task", workflow_run_id="wr_123", organization_id="org_123"
)
start_screencast_loop_mock.assert_awaited_once_with(
websocket=websocket,
@@ -44,7 +46,7 @@ async def test_run_local_screencast_happy_path(monkeypatch: pytest.MonkeyPatch)
entity_type="task",
check_finalized=check_finalized,
workflow_run_id="wr_123",
- organization_id=None,
+ organization_id="org_123",
)
get_current_status.assert_awaited_once()
send_status_mock.assert_awaited_once_with(websocket, "task_id", "task_123", "completed")
@@ -72,6 +74,7 @@ async def test_run_local_screencast_timeout_when_browser_state_not_available(
wait_for_running=wait_for_running,
check_finalized=check_finalized,
get_current_status=get_current_status,
+ organization_id="org_123",
)
wait_for_running.assert_awaited_once()
@@ -79,8 +82,37 @@ async def test_run_local_screencast_timeout_when_browser_state_not_available(
"bs_123",
"browser_session",
workflow_run_id=None,
- organization_id=None,
+ organization_id="org_123",
)
start_screencast_loop_mock.assert_not_awaited()
get_current_status.assert_not_awaited()
send_status_mock.assert_awaited_once_with(websocket, "browser_session_id", "bs_123", "timeout")
+
+
+ ("entrypoint", "entity_id", "database_attr", "getter"),
+ [
+ ("_local_screencast_for_task", "tsk_123", "tasks", "get_task"),
+ ("_local_screencast_for_workflow_run", "wr_123", "workflow_runs", "get_workflow_run"),
+ ],
+)
+async def test_local_screencast_forwards_the_organization(
+ monkeypatch: pytest.MonkeyPatch, entrypoint: str, entity_id: str, database_attr: str, getter: str
+) -> None:
+ """A browser held out of process is reachable only through an organization-scoped session
+ lookup, so an entrypoint that drops the organization streams nothing."""
+ finished = SimpleNamespace(status=SimpleNamespace(is_final=lambda: True), organization_id="org_123")
+ monkeypatch.setattr(
+ screenshot,
+ "app",
+ SimpleNamespace(
+ DATABASE=SimpleNamespace(**{database_attr: SimpleNamespace(**{getter: AsyncMock(return_value=finished)})})
+ ),
+ )
+ run_local_screencast_mock = AsyncMock()
+ monkeypatch.setattr(screenshot, "_run_local_screencast", run_local_screencast_mock)
+
+ await getattr(screenshot, entrypoint)(object(), entity_id, "org_123")
+
+ assert run_local_screencast_mock.await_args.kwargs["organization_id"] == "org_123"