Skip to content

Automate browser based workflows with AI

fix(SKY-9285): feasibility-gate resolves repair utterances against session context (#5681)

Andrew Neilsonbd9b58ad67f6
diff --git a/skyvern/forge/prompts/skyvern/feasibility-gate.j2 b/skyvern/forge/prompts/skyvern/feasibility-gate.j2
index 45731f357..3dc4892b7 100644
--- a/skyvern/forge/prompts/skyvern/feasibility-gate.j2
+++ b/skyvern/forge/prompts/skyvern/feasibility-gate.j2
@@ -1,4 +1,4 @@
-You are a feasibility classifier for a browser-automation workflow copilot. The copilot builds Skyvern workflows that navigate websites and extract or submit data on the user's behalf. Your job is to inspect a single user request and decide whether it is specific enough to build a workflow immediately, or whether the copilot should ask one clarifying question first.
+You are a feasibility classifier for a browser-automation workflow copilot. The copilot builds Skyvern workflows that navigate websites and extract or submit data on the user's behalf. Your job is to inspect the user's latest message — interpreted in the context of the prior conversation and the current workflow — and decide whether the request is specific enough to build (or update) a workflow immediately, or whether the copilot should ask one clarifying question first.
Return ONLY a JSON object. No prose, no markdown fences.
@@ -7,22 +7,31 @@ Required fields:
- question: required iff verdict == "ask_clarification". A single, specific, one-sentence question for the user.
- rationale: short internal reasoning.
-Your default is `proceed`. The main copilot can inspect the live page, iterate, and ask its own clarifying question later if needed. Your job is only to catch requests that are structurally broken before any navigation happens — not to disambiguate words, choose between plausible interpretations, or route the user to a "better" URL.
+Your default is `proceed`. The main copilot can inspect the live page, iterate, and ask its own clarifying question later if needed. Your job is only to catch requests that are impossible to fulfill on the resolved target — not to disambiguate words, choose between plausible interpretations, or route the user to a "better" URL.
-When a concrete URL is present in the request, default even harder toward `proceed`. Do NOT return `ask_clarification` for any of these reasons when a URL is given:
+Refinements and corrections:
+
+The user's latest message may be a refinement, correction, or continuation of an earlier request rather than a fresh standalone ask. Short utterances like "I meant Fulham", "use Z instead", "change the URL to Y", "actually 5 of them", or "it's the away game" inherit the website, the action, and the surrounding framing from the prior turn. Resolve the latest message against `chat_history` and `workflow_yaml` first, then evaluate feasibility on the resolved request. NEVER return `ask_clarification` to re-ask for information already established earlier in the session — the website, the action, and the framing are all valid context to inherit.
+
+When a concrete URL is present in the latest message OR in `workflow_yaml`, default even harder toward `proceed`. Do NOT return `ask_clarification` for any of these reasons when a URL is established:
- A word or phrase in the goal is ambiguous ("level", "latest", "best", "right one", "most popular") — the main copilot will resolve it by reading the page.
- The request has multiple parts and one part is unclear — the copilot will handle each part as it navigates.
- The URL is a portal / search engine / marketplace / social platform / catalog / aggregator and you think a sibling sub-property would be a "better" fit — the copilot can navigate to the sibling itself, or run the query on the given URL.
- You would like more specificity about quantities, filters, rankings, or categories — those are runtime decisions.
+Mid-session pivots:
+
+A follow-up message that introduces a NEW target — a different website, a different domain, or a different broad task than the prior turn established — is itself a fresh request and must be evaluated on its own merits even though prior chat_history exists. If the user pivots to a target structurally incapable of the new task ("actually do this on [a video streaming site]" for a download-filings task), return `ask_clarification` exactly as you would on a first turn. Do not let the existence of prior chat_history bias you toward `proceed` when the pivot itself is impossible.
+
Return `ask_clarification` only when ALL of these are true:
-1. The request cannot plausibly be satisfied on the given URL by any means — no on-site search, no category browse, no sibling property, no interpretation of the user's goal makes it feasible.
+1. The resolved request cannot plausibly be satisfied on the resolved target by any means — no on-site search, no category browse, no sibling property, no interpretation of the user's goal makes it feasible.
2. A single one-sentence question would resolve the impasse (if the fix is "tell me a completely different goal", don't bother asking).
3. The same question has not already been asked-and-answered in chat_history or global_llm_context.
Canonical `ask_clarification` examples (rare):
- User gave a URL that is structurally incapable of the task (read-only page for a submit task, a video streaming site for a "download filings" task).
-- No URL at all AND the request names multiple distinct external sources (e.g. "pull the latest release notes" without saying which project on which host).
+- No URL at all in the message OR in the workflow AND the request names multiple distinct external sources (e.g. "pull the latest release notes" without saying which project on which host).
+- A mid-session pivot to a NEW target that is structurally incapable of the new task.
If you are on the fence, return `proceed`. The cost of a wrong `proceed` is one wasted iteration; the cost of a wrong `ask_clarification` is blocking an otherwise-solvable request.
diff --git a/skyvern/forge/sdk/copilot/agent.py b/skyvern/forge/sdk/copilot/agent.py
index 303e7e670..908a4f849 100644
--- a/skyvern/forge/sdk/copilot/agent.py
+++ b/skyvern/forge/sdk/copilot/agent.py
@@ -321,8 +321,8 @@ async def run_copilot_agent(
api_key: str | None = None,
security_rules: str = "",
) -> AgentResult:
- # Preflight feasibility classifier. Never raises (errors fall through to
- # proceed). Off by default; enable via settings.ENABLE_COPILOT_FEASIBILITY_GATE.
+ # Preflight feasibility classifier — fires on every turn so mid-session pivots
+ # to impossible targets are caught the same as first-turn structural mismatches.
from skyvern.forge.sdk.copilot.feasibility_gate import run_feasibility_gate
feasibility_verdict = await run_feasibility_gate(
diff --git a/skyvern/forge/sdk/copilot/feasibility_gate.py b/skyvern/forge/sdk/copilot/feasibility_gate.py
index 74fde047a..585654f32 100644
--- a/skyvern/forge/sdk/copilot/feasibility_gate.py
+++ b/skyvern/forge/sdk/copilot/feasibility_gate.py
@@ -163,18 +163,18 @@ async def run_feasibility_gate(
return _PROCEED
verdict = _coerce_verdict(response)
+ # INFO not DEBUG so verdict-rate dashboards work without trace-bisection.
+ # `question` is UI-displayed so safe at INFO; `rationale` stays at DEBUG below
+ # because untrusted LLM output can echo back user content under prompt injection.
+ log_extras: dict[str, Any] = {
+ "verdict": verdict.verdict,
+ "user_message_len": len(user_message),
+ "chat_history_len": len(chat_history or ""),
+ "workflow_yaml_len": len(workflow_yaml or ""),
+ }
+ if verdict.question is not None:
+ log_extras["question"] = verdict.question
+ LOG.info("feasibility-gate verdict", **log_extras)
if verdict.verdict == "ask_clarification":
- # `question` is user-facing (displayed in the UI) so logging it at
- # INFO is fine. `rationale` is the LLM's internal reasoning and can
- # echo back user content under prompt injection -- drop it to DEBUG
- # so untrusted model output doesn't ship to every log aggregator.
- LOG.info(
- "feasibility-gate classifier asked for clarification",
- question=verdict.question,
- )
LOG.debug("feasibility-gate clarification rationale", rationale=verdict.rationale)
- else:
- # Debug-level so latency regressions and skip-rate anomalies are
- # traceable without adding INFO noise on every copilot message.
- LOG.debug("feasibility-gate classifier returned proceed")
return verdict
diff --git a/tests/unit/test_copilot_feasibility_gate.py b/tests/unit/test_copilot_feasibility_gate.py
index cb3b14965..336900cbc 100644
--- a/tests/unit/test_copilot_feasibility_gate.py
+++ b/tests/unit/test_copilot_feasibility_gate.py
@@ -359,3 +359,33 @@ def test_feasibility_verdict_dataclass() -> None:
assert v.verdict == "proceed"
assert v.question is None
assert v.rationale is None
+
+
+# ---------------------------------------------------------------------------
+# Rendered-prompt snapshot — guards against accidental prompt regressions.
+# ---------------------------------------------------------------------------
+
+
+def _render_feasibility_prompt() -> str:
+ from skyvern.forge.prompts import prompt_engine
+
+ return prompt_engine.load_prompt(
+ template="feasibility-gate",
+ user_message="I meant the other one",
+ workflow_yaml="name: example_workflow",
+ chat_history="USER: place an order\nAI: tested, no result",
+ global_llm_context="",
+ )
+
+
+def test_prompt_carries_context_aware_framing() -> None:
+ prompt = _render_feasibility_prompt()
+ assert "refinement, correction, or continuation" in prompt
+ assert "Refinements and corrections:" in prompt
+ assert "Mid-session pivots:" in prompt
+
+
+def test_prompt_does_not_revert_to_single_request_framing() -> None:
+ prompt = _render_feasibility_prompt()
+ assert "single user request" not in prompt
+ assert "before any navigation happens" not in prompt