Skip to content

Automate browser based workflows with AI

fix(SKY-12545): keep credentialed terminal actions past login (#7507)

Andrew Neilsone06e65ed5835
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2
index d86e489f6..3dca42cc1 100644
--- a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2
+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2
@@ -1,6 +1,37 @@
+{% if terminal_action_reconciliation %}
+TERMINAL ACTION RECONCILIATION MODE
+
+The primary RequestPolicy is credential-bearing but omitted a non-method-mandated terminal_action. Return only JSON in
+this exact shape:
+{"version":"1","criterion_id":"c0","terminal_action_family":"request"}
+
+- criterion_id must select exactly one existing criterion ID from PRIMARY COMPLETION CRITERIA, or be null to abstain.
+- terminal_action_family must be request|application|form|order when criterion_id is non-null, or null when abstaining.
+- Select a criterion only when the latest request requires authentication followed by creating or submitting that
+ criterion's request, application, or form outcome, or placing an order.
+- Abstain when the requested end state is successful authentication itself, including MFA, or when no single existing
+ criterion unambiguously represents the post-login terminal action.
+- Preserve criterion identity: do not invent, remove, duplicate, paraphrase, or re-order criteria. Do not select a
+ method-mandated criterion. The server changes only the selected criterion's kind and terminal_action_family.
+
+Credential input kind: {{ reconciliation_credential_input_kind }}
+Latest user message:
+```{{ user_message }}```
+PRIMARY COMPLETION CRITERIA:
+```{{ reconciliation_criteria }}```
+{% else %}
Classify the latest Workflow Copilot request. Return only JSON with keys:
testing_intent=require_test|skip_test|unspecified; authoring_intent=author_now|defer_authoring; credential_input_kind=none|raw_secret|credential_id|credential_name|website_stored_credential|placeholder; credential_refs; login_page_urls; requires_user_clarification; clarification_reason; completion_contract; completion_criteria; raw_secret_evidence; raw_secret_handling=none|block|redacted_draft.
+terminal_action priority:
+- Separate authentication from the requested end state before writing completion_criteria. Authentication is a prefix
+ when the credentialed request continues to create or submit a request, application, or form, or to place an order.
+- In that case, include the post-login end state exactly once as kind=terminal_action with the matching family. Do not
+ replace that post-login end state with successful authentication. Example: "sign in and place an order" produces an
+ order terminal_action criterion, not a sign-in criterion.
+- Emit no terminal_action for a pure-login or MFA request whose only requested end state is authentication. Do not
+ invent a post-login action that the user did not request.
+
testing_intent:
- skip_test: the user asks for an untested draft, says they will test it themselves, says draft only, just sketch it, no validation run, or skip testing.
- require_test: the user asks to build/create and test/run/verify/end-to-end it, or asks for a complete workflow that should be validated.
@@ -48,6 +79,7 @@ completion_criteria:
- Example: if the user asks to return the selected or highest-priority document name, emit the criterion with output_path=output.document_name and requested_output_evidence_source=independent_run_evidence.
- kind=outcome|terminal_action|validation_classification. Use validation_classification only when the user's no-side-effect goal is itself a judgment/classification returned by the workflow, such as whether a path is login-gated. Use terminal_action only when the criterion's end state is a terminal action in the external site for one of the supported families below. Use outcome for all other outcomes.
- terminal_action_family=request|application|form|order|null. Set the matching family when kind=terminal_action; set null otherwise.
+- When a credentialed goal requires the workflow to authenticate and then create or submit a request, application, or form, or place an order, emit that post-login end state as kind=terminal_action with the matching terminal_action_family. Authentication is only the prefix: credential fill and login submit do not satisfy the terminal action. Do not emit terminal_action for a pure-login goal whose requested end state is successful authentication itself.
- classification_output_key: null unless kind=validation_classification. Set it to the exact top-level block output field/key that will carry the classification scalar. Do not use JSONPath, regex, transcript paths, screenshots, prose conditions, or nested paths.
- expected_classification: null unless kind=validation_classification. Set it to the exact expected classification target as a boolean or canonical enum/string value. Do not hide this target in outcome prose.
- For validation-only utility/path checks whose goal is to classify whether the reachable path is login-only/login-gated, prefer kind=validation_classification with classification_output_key=login_only and expected_classification=true. Do not represent that classification target as a requested-output criterion such as output.login_only.
@@ -114,3 +146,4 @@ Current workflow YAML:
{% if active_completion_criteria %}ACTIVE COMPLETION CRITERIA (canonical phrasing for the current goal):
```{{ active_completion_criteria }}```
{% endif %}Global context: ```{{ global_llm_context }}```
+{% endif %}
diff --git a/skyvern/forge/sdk/copilot/completion_criteria_store.py b/skyvern/forge/sdk/copilot/completion_criteria_store.py
index 79b1ab296..dff3f788e 100644
--- a/skyvern/forge/sdk/copilot/completion_criteria_store.py
+++ b/skyvern/forge/sdk/copilot/completion_criteria_store.py
@@ -27,6 +27,7 @@ from skyvern.forge.sdk.copilot.request_policy import (
RequestedOutputEvidenceSource,
RequestedOutputPathMintSource,
TerminalActionFamily,
+ TerminalActionVerificationMode,
_canonical_bool_string,
_coerce_classification_output_key,
_coerce_expected_classification,
@@ -57,6 +58,7 @@ TRIPWIRE_CONSECUTIVE_ALL_NO_EVIDENCE = 2
_CRITERION_LEVELS = ("definition", "run")
_CRITERION_KINDS = ("outcome", "terminal_action", "validation_classification")
_TERMINAL_ACTION_FAMILIES = ("request", "application", "form", "order")
+_TERMINAL_ACTION_VERIFICATION_MODES = frozenset(get_args(TerminalActionVerificationMode))
_MINT_DISPOSITIONS = frozenset(get_args(MintDisposition))
_PINABILITIES = frozenset(get_args(Pinability))
@@ -158,6 +160,8 @@ def criteria_to_json(criteria: tuple[CompletionCriterion, ...] | list[Completion
}
if criterion.deliverable_confirmation_criterion_id is not None:
item["deliverable_confirmation_criterion_id"] = criterion.deliverable_confirmation_criterion_id
+ if criterion.kind == "terminal_action":
+ item["terminal_action_verification_mode"] = criterion.terminal_action_verification_mode
if criterion.requested_output_corroborator:
item["requested_output_corroborator"] = True
if criterion.requested_output_path_mint_source is not None:
@@ -210,6 +214,12 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:
terminal_action_family = (
family_raw if kind == "terminal_action" and family_raw in _TERMINAL_ACTION_FAMILIES else None
)
+ verification_mode_raw = item.get("terminal_action_verification_mode")
+ terminal_action_verification_mode = (
+ verification_mode_raw
+ if kind == "terminal_action" and verification_mode_raw in _TERMINAL_ACTION_VERIFICATION_MODES
+ else "family_record_v1"
+ )
mint_source_raw = item.get("requested_output_path_mint_source")
requested_output_path_mint_source = cast(
RequestedOutputPathMintSource | None,
@@ -287,6 +297,9 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:
requested_output_path_mint_source=requested_output_path_mint_source,
kind=cast(CriterionKind, kind),
terminal_action_family=cast(TerminalActionFamily | None, terminal_action_family),
+ terminal_action_verification_mode=cast(
+ TerminalActionVerificationMode, terminal_action_verification_mode
+ ),
classification_output_key=classification_output_key,
expected_classification=expected_classification,
requested_output_corroborator=bool(item.get("requested_output_corroborator")),
@@ -315,6 +328,9 @@ def _criterion_reconcile_key(criterion: CompletionCriterion) -> str:
expected_output_value_key = typed_expected_output_value_key(criterion.expected_output_value)
expected_output_shape_key = criterion.expected_output_shape or ""
requested_output_evidence_source_key = criterion.requested_output_evidence_source
+ terminal_action_verification_mode_key = (
+ criterion.terminal_action_verification_mode if criterion.kind == "terminal_action" else ""
+ )
classification_output_key = criterion.classification_output_key or ""
expected_classification_key = (
str(criterion.expected_classification) if criterion.expected_classification is not None else ""
@@ -328,6 +344,7 @@ def _criterion_reconcile_key(criterion: CompletionCriterion) -> str:
f"\x1fexpected_output_shape:{expected_output_shape_key}"
f"\x1frequested_output_evidence_source:{requested_output_evidence_source_key}"
f"\x1fkind:{criterion.kind}"
+ f"\x1fterminal_action_verification_mode:{terminal_action_verification_mode_key}"
f"\x1fclassification_output_key:{classification_output_key}"
f"\x1fexpected_classification:{expected_classification_key}"
)
@@ -343,6 +360,7 @@ def _criterion_reconcile_key(criterion: CompletionCriterion) -> str:
f"contingent:{contingent_key}\x1fantecedent_path:{contingent_path_key}"
f"\x1fdeliverable_kind:{deliverable_kind_key}"
f"\x1fkind:{criterion.kind}"
+ f"\x1fterminal_action_verification_mode:{terminal_action_verification_mode_key}"
f"\x1foutcome:{normalized_criterion_outcome_key(criterion.outcome)}"
)
@@ -351,6 +369,35 @@ def _outcome_key_set(criteria: tuple[CompletionCriterion, ...] | list[Completion
return {_criterion_reconcile_key(criterion) for criterion in criteria}
+def _preserve_stored_terminal_action_authority(
+ fresh: list[CompletionCriterion], stored: tuple[CompletionCriterion, ...]
+) -> list[CompletionCriterion]:
+ """Carry semantic terminal-action authority into a fresh set before deciding its epoch."""
+ stored_semantic_terminal_actions = [
+ criterion
+ for criterion in stored
+ if criterion.kind == "terminal_action" and criterion.terminal_action_verification_mode == "semantic_outcome_v1"
+ ]
+ preserved: list[CompletionCriterion] = []
+ for criterion in fresh:
+ matches = []
+ for stored_criterion in stored_semantic_terminal_actions:
+ if criterion.kind not in {"outcome", "terminal_action"}:
+ continue
+ promoted = replace(
+ criterion,
+ kind="terminal_action",
+ terminal_action_family=stored_criterion.terminal_action_family,
+ terminal_action_verification_mode="semantic_outcome_v1",
+ )
+ if _criterion_reconcile_key(promoted) == _criterion_reconcile_key(stored_criterion):
+ matches.append(promoted)
+ if len(matches) == 1:
+ criterion = matches[0]
+ preserved.append(criterion)
+ return preserved
+
+
def _word_tokens(text: str) -> list[str]:
return "".join(char if char.isalnum() else " " for char in text.casefold()).split()
@@ -521,6 +568,7 @@ def reconcile_completion_criteria(
epoch=stored.goal_epoch,
criteria=stored.criteria,
)
+ fresh = _preserve_stored_terminal_action_authority(fresh, stored.criteria)
if _outcome_key_set(fresh) <= _outcome_key_set(stored.criteria):
return ReconcileDecision(
action="adopt_stored",
diff --git a/skyvern/forge/sdk/copilot/completion_verification.py b/skyvern/forge/sdk/copilot/completion_verification.py
index 6757ee855..cb29a91ca 100644
--- a/skyvern/forge/sdk/copilot/completion_verification.py
+++ b/skyvern/forge/sdk/copilot/completion_verification.py
@@ -1321,7 +1321,9 @@ def grade_terminal_goal_record_criteria(
eligible_criteria = [
criterion
for criterion in criteria
- if criterion.kind == "terminal_action" and criterion.terminal_action_family in _TERMINAL_RECORD_FAMILY_ARTIFACTS
+ if criterion.kind == "terminal_action"
+ and criterion.terminal_action_verification_mode == "family_record_v1"
+ and criterion.terminal_action_family in _TERMINAL_RECORD_FAMILY_ARTIFACTS
]
if not eligible_criteria:
return []
diff --git a/skyvern/forge/sdk/copilot/enforcement.py b/skyvern/forge/sdk/copilot/enforcement.py
index 95ace04c8..34574ea6d 100644
--- a/skyvern/forge/sdk/copilot/enforcement.py
+++ b/skyvern/forge/sdk/copilot/enforcement.py
@@ -7,7 +7,7 @@ import copy
import json
import re
import time
-from collections.abc import Mapping
+from collections.abc import Mapping, Sequence
from dataclasses import replace
from typing import TYPE_CHECKING, Any
from urllib.parse import urlparse
@@ -249,6 +249,15 @@ _SYNTHESIZED_BLOCK_PERSISTENCE_MUTATING_TOOLS = frozenset(
# for either or an update_workflow re-author silently spends the one-shot rescout.
_SYNTHESIZED_BLOCK_REAUTHORING_TOOLS = frozenset({SYNTHESIZED_BLOCK_PERSISTENCE_TOOL, "update_workflow"})
_SYNTHESIZED_BLOCK_COMMIT_TOOLS = frozenset({"click", "press_key"})
+_LOGIN_SUBMIT_NAME_PATTERN = re.compile(
+ r"^(?:log in|login|sign in|authenticate)(?: now| securely| to continue)?$",
+ re.I,
+)
+_LOGIN_SUBMIT_SELECTOR_PATTERN = re.compile(
+ r"^(?:(?:log in|login|sign in|authenticate)(?: submit| button| btn)?|"
+ r"(?:submit|button|btn) (?:log in|login|sign in|authenticate))$",
+ re.I,
+)
# Evidence sources confirmable only after a run — excluded from the pre-run scout-coverage gate.
_PRE_RUN_UNGATED_EVIDENCE_SOURCES = frozenset(
{"independent_run_evidence", "registered_output_parameter", "registered_artifact_content"}
@@ -2312,7 +2321,7 @@ def _canonical_output_path(path: str) -> str:
def _active_completion_criteria(ctx: AgentContext) -> tuple[CompletionCriterion, ...]:
- turn_state = ctx.completion_criteria_turn_state
+ turn_state = getattr(ctx, "completion_criteria_turn_state", None)
if turn_state is None or turn_state.decision is None:
return ()
return turn_state.decision.criteria
@@ -2583,6 +2592,20 @@ def _credential_password_demand_holds(ctx: Any, interactions: list[dict[str, Any
return bool(getattr(ctx, "last_scout_observation_has_password_control", False))
+def _first_stable_login_submit_index(interactions: Sequence[Mapping[str, Any]], credential_index: int) -> int | None:
+ for index, interaction in enumerate(interactions[credential_index + 1 :], start=credential_index + 1):
+ tool_name = str(interaction.get("tool_name") or "").strip()
+ if tool_name == "press_key" and str(interaction.get("key") or "").strip() == "Enter":
+ return index
+ if tool_name != "click":
+ continue
+ accessible_name = re.sub(r"[^a-z0-9]+", " ", str(interaction.get("accessible_name") or "").lower()).strip()
+ selector = re.sub(r"[^a-z0-9]+", " ", str(interaction.get("selector") or "").lower()).strip()
+ if _LOGIN_SUBMIT_NAME_PATTERN.fullmatch(accessible_name) or _LOGIN_SUBMIT_SELECTOR_PATTERN.fullmatch(selector):
+ return index
+ return None
+
+
def _credential_flow_scout_gap_incomplete(ctx: Any, trajectory: list[Any]) -> bool:
"""Trajectory- and inventory-scoped mirror of the persist seam's credential scout gate: engaged
credentials (username/password fills) must have every required field filled plus a post-fill
@@ -2604,31 +2627,63 @@ def _credential_flow_scout_gap_incomplete(ctx: Any, trajectory: list[Any]) -> bo
# requires_submit is always True here: the predicate is deliberately stricter than the persist
# gate, which demands a submit only when the block's code itself performs one.
gap = credential_scout_gap(interactions, requirements, requires_submit=True)
+ if gap.missing_submit and _active_non_method_mandated_terminal_actions(ctx):
+ credential_index = _last_scout_credential_fill_index(interactions)
+ if (
+ credential_index is not None
+ and _first_stable_login_submit_index(interactions, credential_index) is not None
+ ):
+ return bool(gap.missing_fields)
return bool(gap.missing_fields) or gap.missing_submit
+def _active_non_method_mandated_terminal_actions(ctx: AgentContext) -> tuple[CompletionCriterion, ...]:
+ return tuple(
+ criterion
+ for criterion in _active_completion_criteria(ctx)
+ if criterion.kind == "terminal_action" and not criterion.method_mandated
+ )
+
+
def synthesized_trajectory_reaches_goal(ctx: AgentContext) -> bool:
"""The scout trajectory covers an opening click followed by a commit, a durable entry followed by a commit,
or a reached download target with a selector. Monotone in what the scout captured."""
trajectory = ctx.scout_trajectory
if not trajectory:
return False
- download = ctx.reached_download_target
- if download is not None and download.selector:
+ if _active_non_method_mandated_terminal_actions(ctx):
+ return _trajectory_reaches_post_credential_commit(ctx)
+ return _trajectory_reaches_generic_goal(ctx, trajectory, include_download=True)
+
+
+def _trajectory_reaches_generic_goal(
+ ctx: AgentContext,
+ trajectory: list[Any],
+ *,
+ include_download: bool,
+ allow_intermediate_interactions: bool = False,
+) -> bool:
+ """Apply the established download, open-to-commit, and durable-entry reach shapes to one trajectory slice."""
+ download = getattr(ctx, "reached_download_target", None)
+ if include_download and download is not None and download.selector:
return True
- if len(trajectory) == 2:
- opening, commit = trajectory
+ opening_trajectory_index: int | None = None
+ ordered_pair_candidates = trajectory if allow_intermediate_interactions or len(trajectory) == 2 else []
+ for interaction in ordered_pair_candidates:
+ if not isinstance(interaction, dict):
+ continue
+ trajectory_index = interaction.get("trajectory_index")
+ if not isinstance(trajectory_index, int):
+ continue
if (
- isinstance(opening, dict)
- and str(opening.get("tool_name") or "") == "click"
- and isinstance(opening.get("trajectory_index"), int)
- and isinstance(commit, dict)
- and str(commit.get("tool_name") or "") in _SYNTHESIZED_BLOCK_COMMIT_TOOLS
- and isinstance(commit.get("trajectory_index"), int)
- and opening["trajectory_index"] < commit["trajectory_index"]
- and not is_generic_entry_opener_click(commit)
+ opening_trajectory_index is not None
+ and trajectory_index > opening_trajectory_index
+ and str(interaction.get("tool_name") or "") in _SYNTHESIZED_BLOCK_COMMIT_TOOLS
+ and not is_generic_entry_opener_click(interaction)
):
return True
+ if opening_trajectory_index is None and str(interaction.get("tool_name") or "") == "click":
+ opening_trajectory_index = trajectory_index
last_entry_index: int | None = None
for index, item in enumerate(trajectory):
if isinstance(item, dict) and is_durable_fallback_entry_target(item):
@@ -2665,40 +2720,44 @@ def _last_scout_credential_fill_index(trajectory: list[Any]) -> int | None:
def _trajectory_reaches_post_credential_commit(ctx: AgentContext) -> bool:
- """Deterministic floor for the terminal-action release: the scout captured a durable entry followed by a
- commit that both sit past the last live credential fill, so a bare login prefix (whose only commit is the
- login submit) never satisfies it while a business spine continued past login does."""
+ """Apply the ordinary reach shapes only to the business spine after the credential submit."""
trajectory = ctx.scout_trajectory
if not trajectory:
return False
- credential_index = _last_scout_credential_fill_index(trajectory)
- start = 0 if credential_index is None else credential_index + 1
- last_entry_index: int | None = None
- for index in range(start, len(trajectory)):
- item = trajectory[index]
- if isinstance(item, dict) and is_durable_fallback_entry_target(item):
- last_entry_index = index
- if last_entry_index is None:
+ interactions = [item for item in trajectory if isinstance(item, dict)]
+ credential_index = _last_scout_credential_fill_index(interactions)
+ if credential_index is None:
+ return _trajectory_reaches_generic_goal(
+ ctx,
+ interactions,
+ include_download=False,
+ allow_intermediate_interactions=True,
+ )
+ credential_submit_index = _first_stable_login_submit_index(interactions, credential_index)
+ latest_fill_source_url = str(interactions[credential_index].get("source_url") or "").strip()
+ if credential_submit_index is None and latest_fill_source_url:
+ credential_submit_index = first_matched_post_fill_submit_index(
+ interactions,
+ credential_index,
+ {latest_fill_source_url},
+ )
+ if credential_submit_index is None:
return False
- return any(
- isinstance(item, dict)
- and str(item.get("tool_name") or "") in _SYNTHESIZED_BLOCK_COMMIT_TOOLS
- and not is_generic_entry_opener_click(item)
- for item in trajectory[last_entry_index + 1 :]
+ return _trajectory_reaches_generic_goal(
+ ctx,
+ interactions[credential_submit_index + 1 :],
+ include_download=False,
+ allow_intermediate_interactions=True,
)
def reached_terminal_action_criterion_ids(ctx: AgentContext) -> set[str]:
"""Active, non-method-mandated terminal_action criterion ids the scout has structurally reached: empty
- until the trajectory shows a durable entry->commit past the credential flow. The method_mandated synthetic
- durable-fill criterion is excluded so a login-only turn never self-releases through it."""
+ until the post-credential trajectory shows an ordered open->commit pair or durable entry->commit. The
+ method_mandated synthetic durable-fill criterion is excluded so a login-only turn never self-releases."""
if not _trajectory_reaches_post_credential_commit(ctx):
return set()
- return {
- criterion.id
- for criterion in _active_completion_criteria(ctx)
- if criterion.kind == "terminal_action" and not criterion.method_mandated
- }
+ return {criterion.id for criterion in _active_non_method_mandated_terminal_actions(ctx)}
def record_reached_terminal_action_observation(ctx: AgentContext) -> None:
@@ -2716,9 +2775,7 @@ def _request_expects_unreached_terminal_action(ctx: AgentContext) -> bool:
# A terminal_action criterion is reached only once the scout observes its downstream page, which no
# pre-run page scalar evidences; a goal-reaching login prefix would otherwise read goal-complete before
# the scout crosses into the business spine and land the latch on a login-only trajectory.
- for criterion in _active_completion_criteria(ctx):
- if criterion.kind != "terminal_action" or criterion.method_mandated:
- continue
+ for criterion in _active_non_method_mandated_terminal_actions(ctx):
if criterion.id not in ctx.scout_observed_terminal_criterion_ids:
return True
return False
@@ -2801,10 +2858,14 @@ def _should_force_synthesized_block_persistence(ctx: Any) -> bool:
def _should_block_mutating_tool_after_synthesized_offer(ctx: Any, tool_name: str) -> bool:
if tool_name not in _SYNTHESIZED_BLOCK_PERSISTENCE_MUTATING_TOOLS:
return False
- if uncovered_requested_output_paths(ctx):
- return False
- if _credential_flow_scout_gap_incomplete(ctx, getattr(ctx, "scout_trajectory", None) or []):
- return False
+ if _active_non_method_mandated_terminal_actions(ctx):
+ if not synthesized_trajectory_is_goal_complete(ctx):
+ return False
+ else:
+ if uncovered_requested_output_paths(ctx):
+ return False
+ if _credential_flow_scout_gap_incomplete(ctx, getattr(ctx, "scout_trajectory", None) or []):
+ return False
if getattr(ctx, "update_workflow_called", False) and not synthesized_persistence_reopened(ctx):
return False
if not _turn_intent_can_update_and_run_without_user_input(getattr(ctx, "turn_intent", None)):
diff --git a/skyvern/forge/sdk/copilot/request_policy.py b/skyvern/forge/sdk/copilot/request_policy.py
index da289ca36..929848d92 100644
--- a/skyvern/forge/sdk/copilot/request_policy.py
+++ b/skyvern/forge/sdk/copilot/request_policy.py
@@ -13,6 +13,7 @@ import structlog
from skyvern.config import settings
from skyvern.forge import app
from skyvern.forge.prompts import prompt_engine
+from skyvern.forge.sdk.api.llm.api_handler import LLMAPIHandler
from skyvern.forge.sdk.copilot.config import CopilotConfig
from skyvern.forge.sdk.copilot.context import StructuredContext, sanitize_global_llm_context_for_prompt
from skyvern.forge.sdk.copilot.llm_errors import is_retriable_llm_error
@@ -70,6 +71,10 @@ _CLASSIFICATION_RESPONSE_FIELDS = {
"raw_secret_handling",
"clarification_reason",
}
+_TERMINAL_ACTION_RECONCILIATION_RESPONSE_FIELDS = frozenset({"version", "criterion_id", "terminal_action_family"})
+_TERMINAL_ACTION_RECONCILABLE_CREDENTIAL_KINDS = frozenset(
+ {"credential_id", "credential_name", "website_stored_credential"}
+)
_LONE_REGISTERED_DOWNLOAD_OUTPUT_PATH = "output.downloaded_files"
REGISTERED_DOWNLOAD_COMPLETION_CRITERION_ID = "__copilot_registered_download__downloaded_files_non_empty"
ClarificationReason = Literal[
@@ -193,6 +198,7 @@ CriterionLevel = Literal["definition", "run"]
_CRITERION_LEVELS: frozenset[str] = frozenset({"definition", "run"})
CriterionKind = Literal["outcome", "terminal_action", "validation_classification"]
TerminalActionFamily = Literal["request", "application", "form", "order"]
+TerminalActionVerificationMode = Literal["family_record_v1", "semantic_outcome_v1"]
ClassificationTarget = str | bool
ExpectedOutputValue = str | bool
ExpectedOutputShape = Literal[
@@ -314,6 +320,9 @@ class CompletionCriterion:
requested_output_path_mint_source: RequestedOutputPathMintSource | None = None
kind: CriterionKind = "outcome"
terminal_action_family: TerminalActionFamily | None = None
+ # Native terminal actions retain the historical structured-family grader. A criterion
+ # promoted from a generic outcome must opt into semantic verification instead.
+ terminal_action_verification_mode: TerminalActionVerificationMode = "family_record_v1"
classification_output_key: str | None = None
expected_classification: ClassificationTarget | None = None
requested_output_corroborator: bool = False
@@ -328,6 +337,13 @@ class CompletionCriterion:
mint_disposition: MintDisposition = "decidable"
+@dataclass(frozen=True)
+class TerminalActionReconciliationV1:
+ version: Literal["1"]
+ criterion_id: str | None
+ terminal_action_family: TerminalActionFamily | None
+
+
@dataclass
class RequestPolicy:
testing_intent: str = "unspecified"
@@ -388,6 +404,11 @@ class RequestPolicy:
"completion_criteria_method_mandated_count": sum(
1 for criterion in self.completion_criteria if criterion.method_mandated
),
+ "completion_criteria_terminal_action_count": sum(
+ 1
+ for criterion in self.completion_criteria
+ if criterion.kind == "terminal_action" and not criterion.method_mandated
+ ),
"raw_secret_detected": self.raw_secret_detected,
"has_raw_secret_evidence": self.raw_secret_evidence is not None,
"raw_secret_handling": self.raw_secret_handling,
@@ -829,6 +850,28 @@ def _coerce_classifier_payload(raw: Any) -> dict[str, Any] | None:
return raw
+def _parse_terminal_action_reconciliation(raw: Any) -> TerminalActionReconciliationV1 | None:
+ if isinstance(raw, str):
+ raw = parse_final_response(raw)
+ if not isinstance(raw, dict) or set(raw) != _TERMINAL_ACTION_RECONCILIATION_RESPONSE_FIELDS:
+ return None
+ if raw.get("version") != "1":
+ return None
+ criterion_id = raw.get("criterion_id")
+ family = raw.get("terminal_action_family")
+ if criterion_id is None and family is None:
+ return TerminalActionReconciliationV1(version="1", criterion_id=None, terminal_action_family=None)
+ if not isinstance(criterion_id, str) or not criterion_id.strip():
+ return None
+ if not isinstance(family, str) or family not in _TERMINAL_ACTION_FAMILIES:
+ return None
+ return TerminalActionReconciliationV1(
+ version="1",
+ criterion_id=criterion_id,
+ terminal_action_family=cast(TerminalActionFamily, family),
+ )
+
+
def _request_slot_anchor(item: dict[str, Any]) -> tuple[str, str] | None:
source_id = item.get("request_slot_source_id")
source_quote = item.get("request_slot_source_quote")
@@ -2582,10 +2625,16 @@ def _apply_explicit_code_block_credential_draft_policy(policy: RequestPolicy, us
policy.clarification_question = None
-async def _run_request_policy_classifier(handler: Any, prompt: str) -> tuple[Any | None, str, int]:
+async def _run_request_policy_classifier(
+ handler: LLMAPIHandler,
+ prompt: str,
+ *,
+ deadline: float | None = None,
+) -> tuple[Any | None, str, int]:
# Diverges from turn-intent on purpose: a retriable provider error (429/5xx) retries once
# within the budget, while a timeout is never retried (retrying cannot beat the budget).
- deadline = time.monotonic() + settings.COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDS
+ if deadline is None:
+ deadline = time.monotonic() + settings.COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDS
retry_count = 0
while True:
remaining = deadline - time.monotonic()
@@ -2607,12 +2656,153 @@ async def _run_request_policy_classifier(handler: Any, prompt: str) -> tuple[Any
return None, "provider_error", retry_count
+def _terminal_action_reconciliation_candidates(policy: RequestPolicy) -> list[CompletionCriterion]:
+ return [
+ criterion
+ for criterion in policy.completion_criteria
+ if criterion.kind == "outcome" and not criterion.method_mandated
+ ]
+
+
+def _render_terminal_action_reconciliation_criteria(criteria: list[CompletionCriterion]) -> str:
+ return json.dumps(
+ [
+ {
+ "id": criterion.id,
+ "outcome": criterion.outcome,
+ "contingent_on": criterion.contingent_on,
+ "implicit": criterion.implicit,
+ "method_mandated": criterion.method_mandated,
+ "level": criterion.level,
+ "output_path": criterion.output_path,
+ "kind": criterion.kind,
+ }
+ for criterion in criteria
+ ],
+ ensure_ascii=True,
+ separators=(",", ":"),
+ )
+
+
+def _policy_needs_terminal_action_reconciliation(policy: RequestPolicy) -> bool:
+ if policy.requires_user_clarification:
+ return False
+ if policy.credential_input_kind not in _TERMINAL_ACTION_RECONCILABLE_CREDENTIAL_KINDS:
+ return False
+ return not any(
+ criterion.kind == "terminal_action" and not criterion.method_mandated
+ for criterion in policy.completion_criteria
+ )
+
+
+def _log_terminal_action_reconciliation_omitted(
+ policy: RequestPolicy,
+ *,
+ reason: str,
+ failure_kind: str = "none",
+ retry_count: int = 0,
+) -> None:
+ LOG.info(
+ "copilot_terminal_action_reconciliation_omitted",
+ reason=reason,
+ failure_kind=failure_kind,
+ retry_count=retry_count,
+ credential_input_kind=policy.credential_input_kind,
+ completion_criteria_count=len(policy.completion_criteria),
+ )
+
+
+async def _reconcile_missing_terminal_action(
+ policy: RequestPolicy,
+ *,
+ user_message: str,
+ handler: LLMAPIHandler,
+) -> None:
+ if not _policy_needs_terminal_action_reconciliation(policy):
+ return
+ candidates = _terminal_action_reconciliation_candidates(policy)
+ if not candidates:
+ _log_terminal_action_reconciliation_omitted(policy, reason="no_eligible_criteria")
+ return
+
+ prompt = prompt_engine.load_prompt(
+ template=PROMPT_NAME,
+ terminal_action_reconciliation=True,
+ user_message=escape_code_fences(redact_raw_secrets_for_prompt(user_message)),
+ reconciliation_credential_input_kind=policy.credential_input_kind,
+ reconciliation_criteria=escape_code_fences(_render_terminal_action_reconciliation_criteria(candidates)),
+ )
+ raw, failure_kind, retry_count = await _run_request_policy_classifier(
+ handler,
+ prompt,
+ )
+ if raw is None:
+ _log_terminal_action_reconciliation_omitted(
+ policy,
+ reason="classifier_failed",
+ failure_kind=failure_kind,
+ retry_count=retry_count,
+ )
+ return
+ reconciliation = _parse_terminal_action_reconciliation(raw)
+ if reconciliation is None:
+ _log_terminal_action_reconciliation_omitted(
+ policy,
+ reason="malformed_response",
+ failure_kind=failure_kind,
+ retry_count=retry_count,
+ )
+ return
+ if reconciliation.criterion_id is None:
+ _log_terminal_action_reconciliation_omitted(
+ policy,
+ reason="model_abstained",
+ failure_kind=failure_kind,
+ retry_count=retry_count,
+ )
+ return
+
+ matching_indexes = [
+ index for index, criterion in enumerate(candidates) if criterion.id == reconciliation.criterion_id
+ ]
+ if len(matching_indexes) != 1:
+ _log_terminal_action_reconciliation_omitted(
+ policy,
+ reason="unknown_or_ambiguous_criterion",
+ failure_kind=failure_kind,
+ retry_count=retry_count,
+ )
+ return
+ selected = candidates[matching_indexes[0]]
+ policy_index = next(
+ (index for index, criterion in enumerate(policy.completion_criteria) if criterion is selected),
+ None,
+ )
+ if policy_index is None:
+ _log_terminal_action_reconciliation_omitted(policy, reason="criterion_not_in_policy")
+ return
+ policy.completion_criteria[policy_index] = replace(
+ selected,
+ kind="terminal_action",
+ terminal_action_family=reconciliation.terminal_action_family,
+ terminal_action_verification_mode="semantic_outcome_v1",
+ )
+ LOG.info(
+ "copilot_terminal_action_reconciled",
+ criterion_id=selected.id,
+ terminal_action_family=reconciliation.terminal_action_family,
+ credential_input_kind=policy.credential_input_kind,
+ completion_criteria_count=len(policy.completion_criteria),
+ retry_count=retry_count,
+ )
+
+
async def _classify_request(
user_message: str,
workflow_yaml: str,
chat_history: list[WorkflowCopilotChatHistoryMessage],
global_llm_context: str,
- handler: Any,
+ handler: LLMAPIHandler | None,
*,
active_criteria: list[CompletionCriterion] | None = None,
config: CopilotConfig | None = None,
@@ -2671,6 +2861,7 @@ async def _classify_request(
)
prompt = prompt_engine.load_prompt(
template=PROMPT_NAME,
+ terminal_action_reconciliation=False,
user_message=escape_code_fences(safe_user_message),
raw_secret_present=str(raw_secret_present).lower(),
workflow_yaml=escape_code_fences(redact_raw_secrets_for_prompt(workflow_yaml)[:2048]),
@@ -2691,7 +2882,12 @@ async def _classify_request(
separators=(",", ":"),
),
)
- raw, failure_kind, retry_count = await _run_request_policy_classifier(handler, prompt)
+ classifier_deadline = time.monotonic() + settings.COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDS
+ raw, failure_kind, retry_count = await _run_request_policy_classifier(
+ handler,
+ prompt,
+ deadline=classifier_deadline,
+ )
if raw is None:
LOG.warning("request-policy classifier failed", failure_kind=failure_kind, retry_count=retry_count)
return _classifier_fallback_policy(
@@ -2835,6 +3031,11 @@ async def _classify_request(
policy.requires_user_clarification = False
policy.raw_secret_evidence = None
_degrade_pathless_contingent_criteria(policy)
+ await _reconcile_missing_terminal_action(
+ policy,
+ user_message=safe_user_message,
+ handler=handler,
+ )
policy.completion_contract_status = (
"present" if policy.completion_contract or policy.graded_completion_criteria() else "absent"
)
@@ -3325,7 +3526,7 @@ async def build_request_policy(
chat_history: list[WorkflowCopilotChatHistoryMessage],
global_llm_context: str,
organization_id: str,
- handler: Any,
+ handler: LLMAPIHandler | None,
active_criteria: list[CompletionCriterion] | None = None,
config: CopilotConfig | None = None,
) -> RequestPolicy:
diff --git a/tests/unit/test_copilot_code_block_persist_seam.py b/tests/unit/test_copilot_code_block_persist_seam.py
index ed8247c2c..5eddfe89e 100644
--- a/tests/unit/test_copilot_code_block_persist_seam.py
+++ b/tests/unit/test_copilot_code_block_persist_seam.py
@@ -16458,7 +16458,7 @@ class TestTerminalActionScoutGate:
def test_login_prefix_with_unreached_terminal_action_is_not_goal_complete(self) -> None:
ctx = self._login_prefix_ctx(self._terminal_action_criterion())
- assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is False
assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is False
def test_login_is_the_whole_goal_stays_goal_complete(self) -> None:
@@ -16469,10 +16469,10 @@ class TestTerminalActionScoutGate:
ctx = self._login_prefix_ctx(self._terminal_action_criterion(method_mandated=True))
assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True
- def test_scout_observed_terminal_action_releases_goal_complete(self) -> None:
+ def test_observed_terminal_action_does_not_bypass_post_credential_floor(self) -> None:
ctx = self._login_prefix_ctx(self._terminal_action_criterion())
ctx.scout_observed_terminal_criterion_ids = {"start_service_request"}
- assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True
+ assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is False
def test_post_credential_business_spine_records_terminal_action_observation(self) -> None:
ctx = self._login_prefix_ctx(self._terminal_action_criterion())
@@ -16482,6 +16482,176 @@ class TestTerminalActionScoutGate:
assert ctx.scout_observed_terminal_criterion_ids == {"start_service_request"}
assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True
+ def test_post_login_open_then_submit_reaches_terminal_action(self) -> None:
+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())
+ ctx.scout_trajectory = list(ctx.scout_trajectory) + [
+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 4},
+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 5},
+ ]
+
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True
+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}
+ enforcement_module.record_reached_terminal_action_observation(ctx)
+ assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True
+
+ def test_post_login_three_click_business_spine_reaches_terminal_action(self) -> None:
+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())
+ ctx.scout_trajectory = list(ctx.scout_trajectory) + [
+ {"tool_name": "click", "selector": "#open-item", "trajectory_index": 4},
+ {"tool_name": "click", "selector": "#add-to-cart", "trajectory_index": 5},
+ {"tool_name": "click", "selector": "#place-order", "trajectory_index": 6},
+ ]
+
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True
+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}
+
+ def test_post_login_non_committing_click_sequence_stays_gated(self) -> None:
+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())
+ ctx.scout_trajectory = list(ctx.scout_trajectory) + [
+ {"tool_name": "click", "selector": "button", "role": "button", "trajectory_index": 4},
+ {"tool_name": "click", "selector": "button", "role": "button", "trajectory_index": 5},
+ {"tool_name": "click", "selector": "button", "role": "button", "trajectory_index": 6},
+ ]
+
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is False
+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == set()
+
+ def test_sourceless_enter_login_submit_establishes_business_boundary(self) -> None:
+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())
+ ctx.scout_trajectory = [
+ {
+ "tool_name": "fill_credential_field",
+ "credential_id": "cred_1",
+ "credential_field": "username",
+ "trajectory_index": 0,
+ },
+ {"tool_name": "press_key", "key": "Enter", "trajectory_index": 1},
+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 2},
+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 3},
+ ]
+ ctx.scouted_credential_field_inventory_by_credential_id = {"cred_1": frozenset({"username"})}
+
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True
+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}
+
+ def test_sourceless_non_enter_key_does_not_establish_login_boundary(self) -> None:
+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())
+ ctx.scout_trajectory = [
+ {
+ "tool_name": "fill_credential_field",
+ "credential_id": "cred_1",
+ "credential_field": "username",
+ "trajectory_index": 0,
+ },
+ {"tool_name": "press_key", "key": "Tab", "trajectory_index": 1},
+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 2},
+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 3},
+ ]
+ ctx.scouted_credential_field_inventory_by_credential_id = {"cred_1": frozenset({"username"})}
+
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is False
+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == set()
+
+ def test_post_login_boundary_ignores_divergent_sourced_click(self) -> None:
+ login_url = "https://portal.example.test/login"
+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())
+ ctx.scout_trajectory = [
+ {
+ "tool_name": "fill_credential_field",
+ "credential_id": "cred_1",
+ "credential_field": "username",
+ "source_url": login_url,
+ "trajectory_index": 0,
+ },
+ {
+ "tool_name": "click",
+ "accessible_name": "Learn more",
+ "source_url": "https://portal.example.test/help",
+ "trajectory_index": 1,
+ },
+ {
+ "tool_name": "click",
+ "selector": "button[type='submit']",
+ "source_url": login_url,
+ "trajectory_index": 2,
+ },
+ {
+ "tool_name": "click",
+ "selector": "#open-request",
+ "source_url": self._BUSINESS_URL,
+ "trajectory_index": 3,
+ },
+ {
+ "tool_name": "click",
+ "selector": "#submit-request",
+ "source_url": self._BUSINESS_URL,
+ "trajectory_index": 4,
+ },
+ ]
+ ctx.scouted_credential_field_inventory_by_credential_id = {"cred_1": frozenset({"username"})}
+
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True
+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}
+
+ def test_sourceless_divergent_click_uses_stable_login_submit_identity(self) -> None:
+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())
+ ctx.scout_trajectory = [
+ {
+ "tool_name": "fill_credential_field",
+ "credential_id": "cred_1",
+ "credential_field": "username",
+ "trajectory_index": 0,
+ },
+ {
+ "tool_name": "click",
+ "selector": "#login-help",
+ "accessible_name": "Trouble signing in?",
+ "trajectory_index": 1,
+ },
+ {"tool_name": "click", "accessible_name": "Sign in", "trajectory_index": 2},
+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 3},
+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 4},
+ ]
+ ctx.scouted_credential_field_inventory_by_credential_id = {"cred_1": frozenset({"username"})}
+
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True
+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}
+
+ def test_mixed_fill_sources_use_latest_sourceless_login_identity(self) -> None:
+ helper = TestCredentialScoutGapMatcher
+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())
+ ctx.scout_trajectory = [
+ helper._fill("cred_1", "username", helper._PAGE_ONE),
+ helper._fill("cred_1", "password", ""),
+ {"tool_name": "click", "accessible_name": "Sign in", "trajectory_index": 2},
+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 3},
+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 4},
+ ]
+
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True
+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}
+ enforcement_module.record_reached_terminal_action_observation(ctx)
+ assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True
+
+ def test_sourceless_login_submit_without_stable_identity_fails_closed(self) -> None:
+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())
+ ctx.scout_trajectory = [
+ {
+ "tool_name": "fill_credential_field",
+ "credential_id": "cred_1",
+ "credential_field": "username",
+ "trajectory_index": 0,
+ },
+ {"tool_name": "click", "accessible_name": "Help", "trajectory_index": 1},
+ {"tool_name": "click", "selector": "button", "trajectory_index": 2},
+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 3},
+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 4},
+ ]
+ ctx.scouted_credential_field_inventory_by_credential_id = {"cred_1": frozenset({"username"})}
+
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is False
+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == set()
+
def test_login_only_trajectory_records_no_terminal_action_observation(self) -> None:
ctx = self._login_prefix_ctx(self._terminal_action_criterion())
enforcement_module.record_reached_terminal_action_observation(ctx)
diff --git a/tests/unit/test_copilot_completion_criteria_lifecycle.py b/tests/unit/test_copilot_completion_criteria_lifecycle.py
index 628db0951..cc2882859 100644
--- a/tests/unit/test_copilot_completion_criteria_lifecycle.py
+++ b/tests/unit/test_copilot_completion_criteria_lifecycle.py
@@ -5,6 +5,7 @@ of the judge-unavailable success bypass."""
from __future__ import annotations
import asyncio
+from dataclasses import replace
from unittest.mock import MagicMock
import pytest
@@ -316,6 +317,143 @@ def test_criteria_json_round_trip_preserves_fields(criteria: tuple[CompletionCri
assert criteria_from_json(criteria_to_json(criteria)) == criteria
+def test_terminal_action_verification_mode_round_trips() -> None:
+ criterion = CompletionCriterion(
+ id="c0",
+ outcome="The service request is created.",
+ kind="terminal_action",
+ terminal_action_family="request",
+ terminal_action_verification_mode="semantic_outcome_v1",
+ )
+
+ raw = criteria_to_json([criterion])
+
+ assert raw[0]["terminal_action_verification_mode"] == "semantic_outcome_v1"
+ assert criteria_from_json(raw) == (criterion,)
+ legacy = {key: value for key, value in raw[0].items() if key != "terminal_action_verification_mode"}
+ assert criteria_from_json([legacy])[0].terminal_action_verification_mode == "family_record_v1"
+ stored = StoredCriteriaSnapshot(
+ active=StoredCriteriaSet(
+ set_id="wccs_1",
+ goal_epoch=1,
+ criteria=(replace(criterion, terminal_action_verification_mode="family_record_v1"),),
+ ),
+ next_epoch=2,
+ )
+ decision = reconcile_completion_criteria(stored, [criterion], actionable=True)
+ assert decision.action == "create"
+ assert decision.criteria[0].terminal_action_verification_mode == "semantic_outcome_v1"
+
+
+def test_terminal_action_reconciliation_cannot_regress_semantic_authority() -> None:
+ semantic = CompletionCriterion(
+ id="c0",
+ outcome="The service request is created.",
+ kind="terminal_action",
+ terminal_action_family="request",
+ terminal_action_verification_mode="semantic_outcome_v1",
+ )
+ stored = StoredCriteriaSnapshot(
+ active=StoredCriteriaSet(set_id="wccs_1", goal_epoch=1, criteria=(semantic,)),
+ next_epoch=2,
+ )
+
+ decision = reconcile_completion_criteria(
+ stored,
+ [replace(semantic, terminal_action_verification_mode="family_record_v1")],
+ actionable=True,
+ )
+
+ assert decision.action == "adopt_stored"
+ assert decision.criteria == (semantic,)
+
+
+def test_terminal_action_reconciliation_abstention_cannot_drop_stored_semantic_authority() -> None:
+ semantic = CompletionCriterion(
+ id="c0",
+ outcome="The service request is created.",
+ kind="terminal_action",
+ terminal_action_family="request",
+ terminal_action_verification_mode="semantic_outcome_v1",
+ )
+ stored = StoredCriteriaSnapshot(
+ active=StoredCriteriaSet(set_id="wccs_1", goal_epoch=1, criteria=(semantic,)),
+ next_epoch=2,
+ )
+
+ decision = reconcile_completion_criteria(
+ stored,
+ [
+ replace(
+ semantic,
+ kind="outcome",
+ terminal_action_family=None,
+ terminal_action_verification_mode="family_record_v1",
+ )
+ ],
+ actionable=True,
+ )
+
+ assert decision.action == "adopt_stored"
+ assert decision.criteria == (semantic,)
+
+
+def test_terminal_action_semantic_authority_survives_fresh_criterion_reindexing() -> None:
+ semantic = CompletionCriterion(
+ id="c0",
+ outcome="The service request is created.",
+ kind="terminal_action",
+ terminal_action_family="request",
+ terminal_action_verification_mode="semantic_outcome_v1",
+ )
+ stored = StoredCriteriaSnapshot(
+ active=StoredCriteriaSet(set_id="wccs_1", goal_epoch=1, criteria=(semantic,)),
+ next_epoch=2,
+ )
+ authenticated = CompletionCriterion(id="c0", outcome="The user is authenticated.")
+ reindexed = replace(
+ semantic,
+ id="c1",
+ kind="outcome",
+ terminal_action_family=None,
+ terminal_action_verification_mode="family_record_v1",
+ )
+
+ decision = reconcile_completion_criteria(stored, [authenticated, reindexed], actionable=True)
+
+ assert decision.action == "create"
+ assert [
+ (criterion.id, criterion.kind, criterion.terminal_action_verification_mode) for criterion in decision.criteria
+ ] == [
+ ("c0", "outcome", "family_record_v1"),
+ ("c1", "terminal_action", "semantic_outcome_v1"),
+ ]
+
+
+def test_expanded_criteria_preserve_stored_terminal_action_semantic_authority() -> None:
+ semantic = CompletionCriterion(
+ id="c0",
+ outcome="The service request is created.",
+ kind="terminal_action",
+ terminal_action_family="request",
+ terminal_action_verification_mode="semantic_outcome_v1",
+ )
+ added = CompletionCriterion(id="c1", outcome="The confirmation number is returned.")
+ stored = StoredCriteriaSnapshot(
+ active=StoredCriteriaSet(set_id="wccs_1", goal_epoch=1, criteria=(semantic,)),
+ next_epoch=2,
+ )
+
+ decision = reconcile_completion_criteria(
+ stored,
+ [replace(semantic, terminal_action_verification_mode="family_record_v1"), added],
+ actionable=True,
+ )
+
+ assert decision.action == "create"
+ assert decision.criteria == (semantic, added)
+
+
def test_typed_criteria_list_round_trip_preserves_grading_metadata() -> None:
criterion = CompletionCriterion(
id="c0",
diff --git a/tests/unit/test_copilot_completion_verification.py b/tests/unit/test_copilot_completion_verification.py
index 0651ba91d..798625580 100644
--- a/tests/unit/test_copilot_completion_verification.py
+++ b/tests/unit/test_copilot_completion_verification.py
@@ -1588,6 +1588,49 @@ def test_terminal_goal_record_satisfies_flat_submit_payload() -> None:
]
+async def test_reconciled_terminal_action_requires_semantic_verification_authority() -> None:
+ criterion = CompletionCriterion(
+ id="c0",
+ outcome="The requested service setup is created.",
+ kind="terminal_action",
+ terminal_action_family="request",
+ terminal_action_verification_mode="semantic_outcome_v1",
+ )
+ snapshot = RunEvidenceSnapshot(
+ block_outputs={
+ "terminal_result": {
+ "request_id": "QC-2002-DEMO",
+ "submitted": True,
+ "submission_result": "created",
+ }
+ }
+ )
+
+ assert grade_terminal_goal_record_criteria([criterion], snapshot) == []
+ captured_prompt = ""
+
+ async def handler(*, prompt: str, prompt_name: str) -> dict[str, object]:
+ nonlocal captured_prompt
+ captured_prompt = prompt
+ return {
+ "verdicts": [
+ {
+ "criterion_id": "c0",
+ "satisfied": True,
+ "reason_code": "evidence_confirms",
+ "evidence_ref": "block_outputs:terminal_result",
+ }
+ ]
+ }
+
+ result = await evaluate_completion_criteria([criterion], snapshot, handler)
+
+ assert result.is_fully_satisfied() is True
+ assert criterion.outcome in captured_prompt
+ assert "QC-2002-DEMO" in captured_prompt
+
+
def test_terminal_goal_record_accepts_family_artifact_without_self_asserted_boolean() -> None:
payload = _terminal_goal_payload(submitted=None)
snapshot = RunEvidenceSnapshot(block_outputs={"submit_water_request": payload})
diff --git a/tests/unit/test_copilot_request_policy_completion_criteria.py b/tests/unit/test_copilot_request_policy_completion_criteria.py
index 9ce8820ed..94e1154aa 100644
--- a/tests/unit/test_copilot_request_policy_completion_criteria.py
+++ b/tests/unit/test_copilot_request_policy_completion_criteria.py
@@ -2,11 +2,16 @@ from __future__ import annotations
import hashlib
import json
+from collections.abc import Awaitable
from dataclasses import replace
+from types import SimpleNamespace
from typing import Any
import pytest
+from structlog.testing import capture_logs
+from skyvern.forge.sdk.copilot import enforcement as enforcement_module
+from skyvern.forge.sdk.copilot import request_policy as request_policy_module
from skyvern.forge.sdk.copilot.completion_criteria_store import (
StoredCriteriaSet,
StoredCriteriaSnapshot,
@@ -15,7 +20,8 @@ from skyvern.forge.sdk.copilot.completion_criteria_store import (
criteria_to_json,
reconcile_completion_criteria,
)
-from skyvern.forge.sdk.copilot.config import CopilotConfig
+from skyvern.forge.sdk.copilot.config import BlockAuthoringPolicy, CopilotConfig
+from skyvern.forge.sdk.copilot.context import CopilotContext
from skyvern.forge.sdk.copilot.request_policy import (
REGISTERED_DOWNLOAD_COMPLETION_CRITERION_ID,
CompletionCriterion,
@@ -33,8 +39,79 @@ from skyvern.forge.sdk.copilot.request_policy import (
is_fallback_floor_base_criterion,
request_policy_has_present_completion_contract,
)
+from skyvern.forge.sdk.copilot.tools import workflow_update as workflow_update_module
+from skyvern.forge.sdk.copilot.turn_intent import TurnIntent, TurnIntentAuthority, TurnIntentMode
+from tests.unit.conftest import make_copilot_context
from tests.unit.copilot_test_helpers import make_completion_criterion as _criterion
+_TERMINAL_ACTION_CREATE_OUTCOMES = (
+ "A new service request is submitted.",
+ "The requested service setup is created.",
+ "The account's start-service request is created.",
+)
+_TERMINAL_ACTION_ABSTAIN = {"version": "1", "criterion_id": None, "terminal_action_family": None}
+
+
+def _terminal_action_packet(*outcomes: str) -> dict[str, object]:
+ return {
+ "testing_intent": "require_test",
+ "credential_input_kind": "credential_name",
+ "credential_refs": ["Portal Login"],
+ "requires_user_clarification": False,
+ "completion_criteria": [
+ {
+ "outcome": outcome,
+ "implicit": False,
+ "method_mandated": False,
+ "level": "run",
+ "kind": "outcome",
+ "terminal_action_family": None,
+ }
+ for outcome in outcomes
+ ],
+ }
+
+
+async def _classify_with_terminal_action_reconciliation(
+ user_message: str,
+ packet: dict[str, object],
+ reconciliation: dict[str, object],
+) -> tuple[RequestPolicy, list[str]]:
+ prompts: list[str] = []
+
+ async def handler(prompt: str, prompt_name: str) -> dict[str, object]:
+ assert prompt_name == request_policy_module.PROMPT_NAME
+ prompts.append(prompt)
+ if "TERMINAL ACTION RECONCILIATION MODE" in prompt:
+ return reconciliation
+ return packet
+
+ policy = await request_policy_module._classify_request(user_message, "", [], "", handler)
+ return policy, prompts
+
+
+def _terminal_action_enforcement_ctx(criteria: tuple[CompletionCriterion, ...]) -> CopilotContext:
+ ctx = make_copilot_context()
+ ctx.turn_intent = TurnIntent(
+ mode=TurnIntentMode.BUILD,
+ authority=TurnIntentAuthority(may_update_workflow=True, may_run_blocks=True),
+ )
+ ctx.block_authoring_policy = BlockAuthoringPolicy.CODE_ONLY_BROWSER
+ ctx.completion_criteria_turn_state = SimpleNamespace(decision=SimpleNamespace(criteria=criteria))
+ ctx.scout_trajectory = [
+ {
+ "tool_name": "fill_credential_field",
+ "credential_id": "credential_ref",
+ "credential_field": "username",
+ "trajectory_index": 0,
+ },
+ {"tool_name": "click", "accessible_name": "Sign in", "trajectory_index": 1},
+ ]
+ ctx.synthesized_block_offered = True
+ ctx.synthesized_block_offered_trajectory_len = len(ctx.scout_trajectory)
+ ctx.synthesized_block_offered_goal_complete = enforcement_module.synthesized_trajectory_is_goal_complete(ctx)
+ return ctx
+
async def _policy_for_message(
user_message: str,
@@ -86,6 +163,115 @@ def _criteria_fingerprint(criteria: list[CompletionCriterion]) -> str:
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
[email protected]("outcome", _TERMINAL_ACTION_CREATE_OUTCOMES)
+async def test_credentialed_create_omission_is_reconciled_before_custody_and_foreclosure(outcome: str) -> None:
+ policy, prompts = await _classify_with_terminal_action_reconciliation(
+ "Sign in with the saved credential and create the requested service setup.",
+ _terminal_action_packet(outcome),
+ {"version": "1", "criterion_id": "c0", "terminal_action_family": "request"},
+ )
+
+ assert len(prompts) == 2
+ assert '"id":"c0"' in prompts[1] and '"kind":"outcome"' in prompts[1]
+ stored = criteria_from_json(criteria_to_json(policy.completion_criteria))
+ assert [
+ (
+ item.id,
+ item.outcome,
+ item.kind,
+ item.terminal_action_family,
+ item.terminal_action_verification_mode,
+ )
+ for item in stored
+ ] == [("c0", outcome, "terminal_action", "request", "semantic_outcome_v1")]
+ assert stored[0].method_mandated is False
+ ctx = _terminal_action_enforcement_ctx(stored)
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is False
+ assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is False
+ assert enforcement_module._should_block_mutating_tool_after_synthesized_offer(ctx, "click") is False
+
+ ctx.update_workflow_called = True
+ with capture_logs() as logs:
+ workflow_update_module._log_imposition_skipped_after_update(ctx)
+ (event,) = (entry for entry in logs if entry["event"] == "copilot_imposition_skipped_after_update")
+ assert (event["reaches_goal"], event["goal_complete"]) == (False, False)
+
+
+ ("user_message", "outcome"),
+ [
+ ("Sign in with the saved credential named Portal Login.", "The user is authenticated."),
+ ("Sign in with the saved credential and complete MFA.", "Authentication including MFA completes."),
+ ],
+)
+async def test_login_only_reconciliation_abstains_and_keeps_generic_goal_floor(user_message: str, outcome: str) -> None:
+ with capture_logs() as logs:
+ policy, prompts = await _classify_with_terminal_action_reconciliation(
+ user_message, _terminal_action_packet(outcome), _TERMINAL_ACTION_ABSTAIN
+ )
+
+ assert len(prompts) == 2
+ assert [(item.id, item.outcome, item.kind) for item in policy.completion_criteria] == [("c0", outcome, "outcome")]
+ assert [
+ entry["reason"] for entry in logs if entry["event"] == "copilot_terminal_action_reconciliation_omitted"
+ ] == ["model_abstained"]
+ ctx = _terminal_action_enforcement_ctx(tuple(policy.completion_criteria))
+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True
+ assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True
+
+
+async def test_ambiguous_terminal_action_reconciliation_abstains_without_rekeying() -> None:
+ malformed = {"version": "1", "criterion_id": ["c0", "c1"], "terminal_action_family": "request"}
+ with capture_logs() as logs:
+ policy, _ = await _classify_with_terminal_action_reconciliation(
+ "Sign in and create a service request.",
+ _terminal_action_packet("A service request is created.", "The created request is visible."),
+ malformed,
+ )
+
+ assert [(item.id, item.kind) for item in policy.completion_criteria] == [("c0", "outcome"), ("c1", "outcome")]
+ assert [
+ entry["reason"] for entry in logs if entry["event"] == "copilot_terminal_action_reconciliation_omitted"
+ ] == ["malformed_response"]
+
+
+async def test_terminal_action_reconciliation_gets_its_own_classifier_deadline(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ timeouts: list[float] = []
+ # The initial classification may be followed by request-slot correction and
+ # production work that legitimately outlives its budget. Reconciliation is
+ # a distinct classifier stage and must not inherit that expired deadline.
+ monotonic_values = iter((100.0, 101.0, 112.0, 113.0))
+
+ async def bounded_wait_for(awaitable: Awaitable[object], *, timeout: float) -> object:
+ timeouts.append(timeout)
+ return await awaitable
+
+ timeout_error = request_policy_module.asyncio.TimeoutError
+ monkeypatch.setattr(request_policy_module.settings, "COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDS", 10.0)
+ monkeypatch.setattr(request_policy_module, "time", SimpleNamespace(monotonic=lambda: next(monotonic_values)))
+ monkeypatch.setattr(
+ request_policy_module,
+ "asyncio",
+ SimpleNamespace(wait_for=bounded_wait_for, TimeoutError=timeout_error),
+ )
+
+ policy, prompts = await _classify_with_terminal_action_reconciliation(
+ "Sign in with the saved credential and create the requested service setup.",
+ _terminal_action_packet("The requested service setup is created."),
+ {"version": "1", "criterion_id": "c0", "terminal_action_family": "request"},
+ )
+
+ assert len(prompts) == 2
+ assert [(criterion.id, criterion.kind) for criterion in policy.completion_criteria] == [("c0", "terminal_action")]
+ assert timeouts == [9.0, 9.0]
+
+
def test_p9_exact_value_unpinnable_output_mint_degrades() -> None:
policy = RequestPolicy(
completion_criteria=[
diff --git a/tests/unit/test_copilot_request_policy_prompt.py b/tests/unit/test_copilot_request_policy_prompt.py
index 8defa4514..4ffacfde5 100644
--- a/tests/unit/test_copilot_request_policy_prompt.py
+++ b/tests/unit/test_copilot_request_policy_prompt.py
@@ -44,6 +44,16 @@ def _render(**overrides: str) -> str:
)
+def _render_terminal_action_reconciliation() -> str:
+ return prompt_engine.load_prompt(
+ template=PROMPT_NAME,
+ terminal_action_reconciliation=True,
+ user_message="Sign in with a saved credential and create a service request.",
+ reconciliation_credential_input_kind="credential_name",
+ reconciliation_criteria='[{"id":"c0","outcome":"A service request is created.","kind":"outcome"}]',
+ )
+
+
class TestRequestPolicyPromptStructure:
def test_existing_login_workflow_without_credentials_clause_still_present(self) -> None:
rendered = _render()
@@ -115,6 +125,24 @@ class TestRequestPolicyPromptStructure:
assert "classification_output_key=login_only and expected_classification=true" in rendered
assert 'The only supported non-null value is "registered_download"' in rendered
+ def test_terminal_action_guidance_distinguishes_order_from_login_prefix(self) -> None:
+ rendered = _render()
+
+ assert "place an order" in rendered
+ assert "replace that post-login end state" in rendered
+ assert "successful authentication" in rendered
+ assert "whose only requested end state is authentication" in rendered
+
+ def test_terminal_action_reconciliation_mode_is_bounded_to_existing_criteria(self) -> None:
+ rendered = _render_terminal_action_reconciliation()
+
+ assert "TERMINAL ACTION RECONCILIATION MODE" in rendered
+ assert '"criterion_id":"c0"' in rendered
+ assert "select exactly one existing criterion ID" in rendered
+ assert "do not invent, remove, duplicate, paraphrase, or re-order criteria" in rendered
+ assert "successful authentication itself, including MFA" in rendered
+ assert "placing an order" in rendered
+
def test_completion_criteria_schema_exposes_judgment_truth_condition_fields(self) -> None:
rendered = _render()
assert "judgment_predicate: null unless expected_output_shape=goal_judgment_boolean" in rendered