Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
fix(SKY-12545): keep credentialed terminal actions past login (#7507)
Andrew Neilsone06e65ed5835
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2index d86e489f6..3dca42cc1 100644--- a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2@@ -1,6 +1,37 @@+{% if terminal_action_reconciliation %}+TERMINAL ACTION RECONCILIATION MODE++The primary RequestPolicy is credential-bearing but omitted a non-method-mandated terminal_action. Return only JSON in+this exact shape:+{"version":"1","criterion_id":"c0","terminal_action_family":"request"}++- criterion_id must select exactly one existing criterion ID from PRIMARY COMPLETION CRITERIA, or be null to abstain.+- terminal_action_family must be request|application|form|order when criterion_id is non-null, or null when abstaining.+- Select a criterion only when the latest request requires authentication followed by creating or submitting that+ criterion's request, application, or form outcome, or placing an order.+- Abstain when the requested end state is successful authentication itself, including MFA, or when no single existing+ criterion unambiguously represents the post-login terminal action.+- Preserve criterion identity: do not invent, remove, duplicate, paraphrase, or re-order criteria. Do not select a+ method-mandated criterion. The server changes only the selected criterion's kind and terminal_action_family.++Credential input kind: {{ reconciliation_credential_input_kind }}+Latest user message:+```{{ user_message }}```+PRIMARY COMPLETION CRITERIA:+```{{ reconciliation_criteria }}```+{% else %}Classify the latest Workflow Copilot request. Return only JSON with keys:testing_intent=require_test|skip_test|unspecified; authoring_intent=author_now|defer_authoring; credential_input_kind=none|raw_secret|credential_id|credential_name|website_stored_credential|placeholder; credential_refs; login_page_urls; requires_user_clarification; clarification_reason; completion_contract; completion_criteria; raw_secret_evidence; raw_secret_handling=none|block|redacted_draft.+terminal_action priority:+- Separate authentication from the requested end state before writing completion_criteria. Authentication is a prefix+ when the credentialed request continues to create or submit a request, application, or form, or to place an order.+- In that case, include the post-login end state exactly once as kind=terminal_action with the matching family. Do not+ replace that post-login end state with successful authentication. Example: "sign in and place an order" produces an+ order terminal_action criterion, not a sign-in criterion.+- Emit no terminal_action for a pure-login or MFA request whose only requested end state is authentication. Do not+ invent a post-login action that the user did not request.+testing_intent:- skip_test: the user asks for an untested draft, says they will test it themselves, says draft only, just sketch it, no validation run, or skip testing.- require_test: the user asks to build/create and test/run/verify/end-to-end it, or asks for a complete workflow that should be validated.@@ -48,6 +79,7 @@ completion_criteria:- Example: if the user asks to return the selected or highest-priority document name, emit the criterion with output_path=output.document_name and requested_output_evidence_source=independent_run_evidence.- kind=outcome|terminal_action|validation_classification. Use validation_classification only when the user's no-side-effect goal is itself a judgment/classification returned by the workflow, such as whether a path is login-gated. Use terminal_action only when the criterion's end state is a terminal action in the external site for one of the supported families below. Use outcome for all other outcomes.- terminal_action_family=request|application|form|order|null. Set the matching family when kind=terminal_action; set null otherwise.+- When a credentialed goal requires the workflow to authenticate and then create or submit a request, application, or form, or place an order, emit that post-login end state as kind=terminal_action with the matching terminal_action_family. Authentication is only the prefix: credential fill and login submit do not satisfy the terminal action. Do not emit terminal_action for a pure-login goal whose requested end state is successful authentication itself.- classification_output_key: null unless kind=validation_classification. Set it to the exact top-level block output field/key that will carry the classification scalar. Do not use JSONPath, regex, transcript paths, screenshots, prose conditions, or nested paths.- expected_classification: null unless kind=validation_classification. Set it to the exact expected classification target as a boolean or canonical enum/string value. Do not hide this target in outcome prose.- For validation-only utility/path checks whose goal is to classify whether the reachable path is login-only/login-gated, prefer kind=validation_classification with classification_output_key=login_only and expected_classification=true. Do not represent that classification target as a requested-output criterion such as output.login_only.@@ -114,3 +146,4 @@ Current workflow YAML:{% if active_completion_criteria %}ACTIVE COMPLETION CRITERIA (canonical phrasing for the current goal):```{{ active_completion_criteria }}```{% endif %}Global context: ```{{ global_llm_context }}```+{% endif %}diff --git a/skyvern/forge/sdk/copilot/completion_criteria_store.py b/skyvern/forge/sdk/copilot/completion_criteria_store.pyindex 79b1ab296..dff3f788e 100644--- a/skyvern/forge/sdk/copilot/completion_criteria_store.py+++ b/skyvern/forge/sdk/copilot/completion_criteria_store.py@@ -27,6 +27,7 @@ from skyvern.forge.sdk.copilot.request_policy import (RequestedOutputEvidenceSource,RequestedOutputPathMintSource,TerminalActionFamily,+ TerminalActionVerificationMode,_canonical_bool_string,_coerce_classification_output_key,_coerce_expected_classification,@@ -57,6 +58,7 @@ TRIPWIRE_CONSECUTIVE_ALL_NO_EVIDENCE = 2_CRITERION_LEVELS = ("definition", "run")_CRITERION_KINDS = ("outcome", "terminal_action", "validation_classification")_TERMINAL_ACTION_FAMILIES = ("request", "application", "form", "order")+_TERMINAL_ACTION_VERIFICATION_MODES = frozenset(get_args(TerminalActionVerificationMode))_MINT_DISPOSITIONS = frozenset(get_args(MintDisposition))_PINABILITIES = frozenset(get_args(Pinability))@@ -158,6 +160,8 @@ def criteria_to_json(criteria: tuple[CompletionCriterion, ...] | list[Completion}if criterion.deliverable_confirmation_criterion_id is not None:item["deliverable_confirmation_criterion_id"] = criterion.deliverable_confirmation_criterion_id+ if criterion.kind == "terminal_action":+ item["terminal_action_verification_mode"] = criterion.terminal_action_verification_modeif criterion.requested_output_corroborator:item["requested_output_corroborator"] = Trueif criterion.requested_output_path_mint_source is not None:@@ -210,6 +214,12 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:terminal_action_family = (family_raw if kind == "terminal_action" and family_raw in _TERMINAL_ACTION_FAMILIES else None)+ verification_mode_raw = item.get("terminal_action_verification_mode")+ terminal_action_verification_mode = (+ verification_mode_raw+ if kind == "terminal_action" and verification_mode_raw in _TERMINAL_ACTION_VERIFICATION_MODES+ else "family_record_v1"+ )mint_source_raw = item.get("requested_output_path_mint_source")requested_output_path_mint_source = cast(RequestedOutputPathMintSource | None,@@ -287,6 +297,9 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:requested_output_path_mint_source=requested_output_path_mint_source,kind=cast(CriterionKind, kind),terminal_action_family=cast(TerminalActionFamily | None, terminal_action_family),+ terminal_action_verification_mode=cast(+ TerminalActionVerificationMode, terminal_action_verification_mode+ ),classification_output_key=classification_output_key,expected_classification=expected_classification,requested_output_corroborator=bool(item.get("requested_output_corroborator")),@@ -315,6 +328,9 @@ def _criterion_reconcile_key(criterion: CompletionCriterion) -> str:expected_output_value_key = typed_expected_output_value_key(criterion.expected_output_value)expected_output_shape_key = criterion.expected_output_shape or ""requested_output_evidence_source_key = criterion.requested_output_evidence_source+ terminal_action_verification_mode_key = (+ criterion.terminal_action_verification_mode if criterion.kind == "terminal_action" else ""+ )classification_output_key = criterion.classification_output_key or ""expected_classification_key = (str(criterion.expected_classification) if criterion.expected_classification is not None else ""@@ -328,6 +344,7 @@ def _criterion_reconcile_key(criterion: CompletionCriterion) -> str:f"\x1fexpected_output_shape:{expected_output_shape_key}"f"\x1frequested_output_evidence_source:{requested_output_evidence_source_key}"f"\x1fkind:{criterion.kind}"+ f"\x1fterminal_action_verification_mode:{terminal_action_verification_mode_key}"f"\x1fclassification_output_key:{classification_output_key}"f"\x1fexpected_classification:{expected_classification_key}")@@ -343,6 +360,7 @@ def _criterion_reconcile_key(criterion: CompletionCriterion) -> str:f"contingent:{contingent_key}\x1fantecedent_path:{contingent_path_key}"f"\x1fdeliverable_kind:{deliverable_kind_key}"f"\x1fkind:{criterion.kind}"+ f"\x1fterminal_action_verification_mode:{terminal_action_verification_mode_key}"f"\x1foutcome:{normalized_criterion_outcome_key(criterion.outcome)}")@@ -351,6 +369,35 @@ def _outcome_key_set(criteria: tuple[CompletionCriterion, ...] | list[Completionreturn {_criterion_reconcile_key(criterion) for criterion in criteria}+def _preserve_stored_terminal_action_authority(+ fresh: list[CompletionCriterion], stored: tuple[CompletionCriterion, ...]+) -> list[CompletionCriterion]:+ """Carry semantic terminal-action authority into a fresh set before deciding its epoch."""+ stored_semantic_terminal_actions = [+ criterion+ for criterion in stored+ if criterion.kind == "terminal_action" and criterion.terminal_action_verification_mode == "semantic_outcome_v1"+ ]+ preserved: list[CompletionCriterion] = []+ for criterion in fresh:+ matches = []+ for stored_criterion in stored_semantic_terminal_actions:+ if criterion.kind not in {"outcome", "terminal_action"}:+ continue+ promoted = replace(+ criterion,+ kind="terminal_action",+ terminal_action_family=stored_criterion.terminal_action_family,+ terminal_action_verification_mode="semantic_outcome_v1",+ )+ if _criterion_reconcile_key(promoted) == _criterion_reconcile_key(stored_criterion):+ matches.append(promoted)+ if len(matches) == 1:+ criterion = matches[0]+ preserved.append(criterion)+ return preserved++def _word_tokens(text: str) -> list[str]:return "".join(char if char.isalnum() else " " for char in text.casefold()).split()@@ -521,6 +568,7 @@ def reconcile_completion_criteria(epoch=stored.goal_epoch,criteria=stored.criteria,)+ fresh = _preserve_stored_terminal_action_authority(fresh, stored.criteria)if _outcome_key_set(fresh) <= _outcome_key_set(stored.criteria):return ReconcileDecision(action="adopt_stored",diff --git a/skyvern/forge/sdk/copilot/completion_verification.py b/skyvern/forge/sdk/copilot/completion_verification.pyindex 6757ee855..cb29a91ca 100644--- a/skyvern/forge/sdk/copilot/completion_verification.py+++ b/skyvern/forge/sdk/copilot/completion_verification.py@@ -1321,7 +1321,9 @@ def grade_terminal_goal_record_criteria(eligible_criteria = [criterionfor criterion in criteria- if criterion.kind == "terminal_action" and criterion.terminal_action_family in _TERMINAL_RECORD_FAMILY_ARTIFACTS+ if criterion.kind == "terminal_action"+ and criterion.terminal_action_verification_mode == "family_record_v1"+ and criterion.terminal_action_family in _TERMINAL_RECORD_FAMILY_ARTIFACTS]if not eligible_criteria:return []diff --git a/skyvern/forge/sdk/copilot/enforcement.py b/skyvern/forge/sdk/copilot/enforcement.pyindex 95ace04c8..34574ea6d 100644--- a/skyvern/forge/sdk/copilot/enforcement.py+++ b/skyvern/forge/sdk/copilot/enforcement.py@@ -7,7 +7,7 @@ import copyimport jsonimport reimport time-from collections.abc import Mapping+from collections.abc import Mapping, Sequencefrom dataclasses import replacefrom typing import TYPE_CHECKING, Anyfrom urllib.parse import urlparse@@ -249,6 +249,15 @@ _SYNTHESIZED_BLOCK_PERSISTENCE_MUTATING_TOOLS = frozenset(# for either or an update_workflow re-author silently spends the one-shot rescout._SYNTHESIZED_BLOCK_REAUTHORING_TOOLS = frozenset({SYNTHESIZED_BLOCK_PERSISTENCE_TOOL, "update_workflow"})_SYNTHESIZED_BLOCK_COMMIT_TOOLS = frozenset({"click", "press_key"})+_LOGIN_SUBMIT_NAME_PATTERN = re.compile(+ r"^(?:log in|login|sign in|authenticate)(?: now| securely| to continue)?$",+ re.I,+)+_LOGIN_SUBMIT_SELECTOR_PATTERN = re.compile(+ r"^(?:(?:log in|login|sign in|authenticate)(?: submit| button| btn)?|"+ r"(?:submit|button|btn) (?:log in|login|sign in|authenticate))$",+ re.I,+)# Evidence sources confirmable only after a run — excluded from the pre-run scout-coverage gate._PRE_RUN_UNGATED_EVIDENCE_SOURCES = frozenset({"independent_run_evidence", "registered_output_parameter", "registered_artifact_content"}@@ -2312,7 +2321,7 @@ def _canonical_output_path(path: str) -> str:def _active_completion_criteria(ctx: AgentContext) -> tuple[CompletionCriterion, ...]:- turn_state = ctx.completion_criteria_turn_state+ turn_state = getattr(ctx, "completion_criteria_turn_state", None)if turn_state is None or turn_state.decision is None:return ()return turn_state.decision.criteria@@ -2583,6 +2592,20 @@ def _credential_password_demand_holds(ctx: Any, interactions: list[dict[str, Anyreturn bool(getattr(ctx, "last_scout_observation_has_password_control", False))+def _first_stable_login_submit_index(interactions: Sequence[Mapping[str, Any]], credential_index: int) -> int | None:+ for index, interaction in enumerate(interactions[credential_index + 1 :], start=credential_index + 1):+ tool_name = str(interaction.get("tool_name") or "").strip()+ if tool_name == "press_key" and str(interaction.get("key") or "").strip() == "Enter":+ return index+ if tool_name != "click":+ continue+ accessible_name = re.sub(r"[^a-z0-9]+", " ", str(interaction.get("accessible_name") or "").lower()).strip()+ selector = re.sub(r"[^a-z0-9]+", " ", str(interaction.get("selector") or "").lower()).strip()+ if _LOGIN_SUBMIT_NAME_PATTERN.fullmatch(accessible_name) or _LOGIN_SUBMIT_SELECTOR_PATTERN.fullmatch(selector):+ return index+ return None++def _credential_flow_scout_gap_incomplete(ctx: Any, trajectory: list[Any]) -> bool:"""Trajectory- and inventory-scoped mirror of the persist seam's credential scout gate: engagedcredentials (username/password fills) must have every required field filled plus a post-fill@@ -2604,31 +2627,63 @@ def _credential_flow_scout_gap_incomplete(ctx: Any, trajectory: list[Any]) -> bo# requires_submit is always True here: the predicate is deliberately stricter than the persist# gate, which demands a submit only when the block's code itself performs one.gap = credential_scout_gap(interactions, requirements, requires_submit=True)+ if gap.missing_submit and _active_non_method_mandated_terminal_actions(ctx):+ credential_index = _last_scout_credential_fill_index(interactions)+ if (+ credential_index is not None+ and _first_stable_login_submit_index(interactions, credential_index) is not None+ ):+ return bool(gap.missing_fields)return bool(gap.missing_fields) or gap.missing_submit+def _active_non_method_mandated_terminal_actions(ctx: AgentContext) -> tuple[CompletionCriterion, ...]:+ return tuple(+ criterion+ for criterion in _active_completion_criteria(ctx)+ if criterion.kind == "terminal_action" and not criterion.method_mandated+ )++def synthesized_trajectory_reaches_goal(ctx: AgentContext) -> bool:"""The scout trajectory covers an opening click followed by a commit, a durable entry followed by a commit,or a reached download target with a selector. Monotone in what the scout captured."""trajectory = ctx.scout_trajectoryif not trajectory:return False- download = ctx.reached_download_target- if download is not None and download.selector:+ if _active_non_method_mandated_terminal_actions(ctx):+ return _trajectory_reaches_post_credential_commit(ctx)+ return _trajectory_reaches_generic_goal(ctx, trajectory, include_download=True)+++def _trajectory_reaches_generic_goal(+ ctx: AgentContext,+ trajectory: list[Any],+ *,+ include_download: bool,+ allow_intermediate_interactions: bool = False,+) -> bool:+ """Apply the established download, open-to-commit, and durable-entry reach shapes to one trajectory slice."""+ download = getattr(ctx, "reached_download_target", None)+ if include_download and download is not None and download.selector:return True- if len(trajectory) == 2:- opening, commit = trajectory+ opening_trajectory_index: int | None = None+ ordered_pair_candidates = trajectory if allow_intermediate_interactions or len(trajectory) == 2 else []+ for interaction in ordered_pair_candidates:+ if not isinstance(interaction, dict):+ continue+ trajectory_index = interaction.get("trajectory_index")+ if not isinstance(trajectory_index, int):+ continueif (- isinstance(opening, dict)- and str(opening.get("tool_name") or "") == "click"- and isinstance(opening.get("trajectory_index"), int)- and isinstance(commit, dict)- and str(commit.get("tool_name") or "") in _SYNTHESIZED_BLOCK_COMMIT_TOOLS- and isinstance(commit.get("trajectory_index"), int)- and opening["trajectory_index"] < commit["trajectory_index"]- and not is_generic_entry_opener_click(commit)+ opening_trajectory_index is not None+ and trajectory_index > opening_trajectory_index+ and str(interaction.get("tool_name") or "") in _SYNTHESIZED_BLOCK_COMMIT_TOOLS+ and not is_generic_entry_opener_click(interaction)):return True+ if opening_trajectory_index is None and str(interaction.get("tool_name") or "") == "click":+ opening_trajectory_index = trajectory_indexlast_entry_index: int | None = Nonefor index, item in enumerate(trajectory):if isinstance(item, dict) and is_durable_fallback_entry_target(item):@@ -2665,40 +2720,44 @@ def _last_scout_credential_fill_index(trajectory: list[Any]) -> int | None:def _trajectory_reaches_post_credential_commit(ctx: AgentContext) -> bool:- """Deterministic floor for the terminal-action release: the scout captured a durable entry followed by a- commit that both sit past the last live credential fill, so a bare login prefix (whose only commit is the- login submit) never satisfies it while a business spine continued past login does."""+ """Apply the ordinary reach shapes only to the business spine after the credential submit."""trajectory = ctx.scout_trajectoryif not trajectory:return False- credential_index = _last_scout_credential_fill_index(trajectory)- start = 0 if credential_index is None else credential_index + 1- last_entry_index: int | None = None- for index in range(start, len(trajectory)):- item = trajectory[index]- if isinstance(item, dict) and is_durable_fallback_entry_target(item):- last_entry_index = index- if last_entry_index is None:+ interactions = [item for item in trajectory if isinstance(item, dict)]+ credential_index = _last_scout_credential_fill_index(interactions)+ if credential_index is None:+ return _trajectory_reaches_generic_goal(+ ctx,+ interactions,+ include_download=False,+ allow_intermediate_interactions=True,+ )+ credential_submit_index = _first_stable_login_submit_index(interactions, credential_index)+ latest_fill_source_url = str(interactions[credential_index].get("source_url") or "").strip()+ if credential_submit_index is None and latest_fill_source_url:+ credential_submit_index = first_matched_post_fill_submit_index(+ interactions,+ credential_index,+ {latest_fill_source_url},+ )+ if credential_submit_index is None:return False- return any(- isinstance(item, dict)- and str(item.get("tool_name") or "") in _SYNTHESIZED_BLOCK_COMMIT_TOOLS- and not is_generic_entry_opener_click(item)- for item in trajectory[last_entry_index + 1 :]+ return _trajectory_reaches_generic_goal(+ ctx,+ interactions[credential_submit_index + 1 :],+ include_download=False,+ allow_intermediate_interactions=True,)def reached_terminal_action_criterion_ids(ctx: AgentContext) -> set[str]:"""Active, non-method-mandated terminal_action criterion ids the scout has structurally reached: empty- until the trajectory shows a durable entry->commit past the credential flow. The method_mandated synthetic- durable-fill criterion is excluded so a login-only turn never self-releases through it."""+ until the post-credential trajectory shows an ordered open->commit pair or durable entry->commit. The+ method_mandated synthetic durable-fill criterion is excluded so a login-only turn never self-releases."""if not _trajectory_reaches_post_credential_commit(ctx):return set()- return {- criterion.id- for criterion in _active_completion_criteria(ctx)- if criterion.kind == "terminal_action" and not criterion.method_mandated- }+ return {criterion.id for criterion in _active_non_method_mandated_terminal_actions(ctx)}def record_reached_terminal_action_observation(ctx: AgentContext) -> None:@@ -2716,9 +2775,7 @@ def _request_expects_unreached_terminal_action(ctx: AgentContext) -> bool:# A terminal_action criterion is reached only once the scout observes its downstream page, which no# pre-run page scalar evidences; a goal-reaching login prefix would otherwise read goal-complete before# the scout crosses into the business spine and land the latch on a login-only trajectory.- for criterion in _active_completion_criteria(ctx):- if criterion.kind != "terminal_action" or criterion.method_mandated:- continue+ for criterion in _active_non_method_mandated_terminal_actions(ctx):if criterion.id not in ctx.scout_observed_terminal_criterion_ids:return Truereturn False@@ -2801,10 +2858,14 @@ def _should_force_synthesized_block_persistence(ctx: Any) -> bool:def _should_block_mutating_tool_after_synthesized_offer(ctx: Any, tool_name: str) -> bool:if tool_name not in _SYNTHESIZED_BLOCK_PERSISTENCE_MUTATING_TOOLS:return False- if uncovered_requested_output_paths(ctx):- return False- if _credential_flow_scout_gap_incomplete(ctx, getattr(ctx, "scout_trajectory", None) or []):- return False+ if _active_non_method_mandated_terminal_actions(ctx):+ if not synthesized_trajectory_is_goal_complete(ctx):+ return False+ else:+ if uncovered_requested_output_paths(ctx):+ return False+ if _credential_flow_scout_gap_incomplete(ctx, getattr(ctx, "scout_trajectory", None) or []):+ return Falseif getattr(ctx, "update_workflow_called", False) and not synthesized_persistence_reopened(ctx):return Falseif not _turn_intent_can_update_and_run_without_user_input(getattr(ctx, "turn_intent", None)):diff --git a/skyvern/forge/sdk/copilot/request_policy.py b/skyvern/forge/sdk/copilot/request_policy.pyindex da289ca36..929848d92 100644--- a/skyvern/forge/sdk/copilot/request_policy.py+++ b/skyvern/forge/sdk/copilot/request_policy.py@@ -13,6 +13,7 @@ import structlogfrom skyvern.config import settingsfrom skyvern.forge import appfrom skyvern.forge.prompts import prompt_engine+from skyvern.forge.sdk.api.llm.api_handler import LLMAPIHandlerfrom skyvern.forge.sdk.copilot.config import CopilotConfigfrom skyvern.forge.sdk.copilot.context import StructuredContext, sanitize_global_llm_context_for_promptfrom skyvern.forge.sdk.copilot.llm_errors import is_retriable_llm_error@@ -70,6 +71,10 @@ _CLASSIFICATION_RESPONSE_FIELDS = {"raw_secret_handling","clarification_reason",}+_TERMINAL_ACTION_RECONCILIATION_RESPONSE_FIELDS = frozenset({"version", "criterion_id", "terminal_action_family"})+_TERMINAL_ACTION_RECONCILABLE_CREDENTIAL_KINDS = frozenset(+ {"credential_id", "credential_name", "website_stored_credential"}+)_LONE_REGISTERED_DOWNLOAD_OUTPUT_PATH = "output.downloaded_files"REGISTERED_DOWNLOAD_COMPLETION_CRITERION_ID = "__copilot_registered_download__downloaded_files_non_empty"ClarificationReason = Literal[@@ -193,6 +198,7 @@ CriterionLevel = Literal["definition", "run"]_CRITERION_LEVELS: frozenset[str] = frozenset({"definition", "run"})CriterionKind = Literal["outcome", "terminal_action", "validation_classification"]TerminalActionFamily = Literal["request", "application", "form", "order"]+TerminalActionVerificationMode = Literal["family_record_v1", "semantic_outcome_v1"]ClassificationTarget = str | boolExpectedOutputValue = str | boolExpectedOutputShape = Literal[@@ -314,6 +320,9 @@ class CompletionCriterion:requested_output_path_mint_source: RequestedOutputPathMintSource | None = Nonekind: CriterionKind = "outcome"terminal_action_family: TerminalActionFamily | None = None+ # Native terminal actions retain the historical structured-family grader. A criterion+ # promoted from a generic outcome must opt into semantic verification instead.+ terminal_action_verification_mode: TerminalActionVerificationMode = "family_record_v1"classification_output_key: str | None = Noneexpected_classification: ClassificationTarget | None = Nonerequested_output_corroborator: bool = False@@ -328,6 +337,13 @@ class CompletionCriterion:mint_disposition: MintDisposition = "decidable"+@dataclass(frozen=True)+class TerminalActionReconciliationV1:+ version: Literal["1"]+ criterion_id: str | None+ terminal_action_family: TerminalActionFamily | None++@dataclassclass RequestPolicy:testing_intent: str = "unspecified"@@ -388,6 +404,11 @@ class RequestPolicy:"completion_criteria_method_mandated_count": sum(1 for criterion in self.completion_criteria if criterion.method_mandated),+ "completion_criteria_terminal_action_count": sum(+ 1+ for criterion in self.completion_criteria+ if criterion.kind == "terminal_action" and not criterion.method_mandated+ ),"raw_secret_detected": self.raw_secret_detected,"has_raw_secret_evidence": self.raw_secret_evidence is not None,"raw_secret_handling": self.raw_secret_handling,@@ -829,6 +850,28 @@ def _coerce_classifier_payload(raw: Any) -> dict[str, Any] | None:return raw+def _parse_terminal_action_reconciliation(raw: Any) -> TerminalActionReconciliationV1 | None:+ if isinstance(raw, str):+ raw = parse_final_response(raw)+ if not isinstance(raw, dict) or set(raw) != _TERMINAL_ACTION_RECONCILIATION_RESPONSE_FIELDS:+ return None+ if raw.get("version") != "1":+ return None+ criterion_id = raw.get("criterion_id")+ family = raw.get("terminal_action_family")+ if criterion_id is None and family is None:+ return TerminalActionReconciliationV1(version="1", criterion_id=None, terminal_action_family=None)+ if not isinstance(criterion_id, str) or not criterion_id.strip():+ return None+ if not isinstance(family, str) or family not in _TERMINAL_ACTION_FAMILIES:+ return None+ return TerminalActionReconciliationV1(+ version="1",+ criterion_id=criterion_id,+ terminal_action_family=cast(TerminalActionFamily, family),+ )++def _request_slot_anchor(item: dict[str, Any]) -> tuple[str, str] | None:source_id = item.get("request_slot_source_id")source_quote = item.get("request_slot_source_quote")@@ -2582,10 +2625,16 @@ def _apply_explicit_code_block_credential_draft_policy(policy: RequestPolicy, uspolicy.clarification_question = None-async def _run_request_policy_classifier(handler: Any, prompt: str) -> tuple[Any | None, str, int]:+async def _run_request_policy_classifier(+ handler: LLMAPIHandler,+ prompt: str,+ *,+ deadline: float | None = None,+) -> tuple[Any | None, str, int]:# Diverges from turn-intent on purpose: a retriable provider error (429/5xx) retries once# within the budget, while a timeout is never retried (retrying cannot beat the budget).- deadline = time.monotonic() + settings.COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDS+ if deadline is None:+ deadline = time.monotonic() + settings.COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDSretry_count = 0while True:remaining = deadline - time.monotonic()@@ -2607,12 +2656,153 @@ async def _run_request_policy_classifier(handler: Any, prompt: str) -> tuple[Anyreturn None, "provider_error", retry_count+def _terminal_action_reconciliation_candidates(policy: RequestPolicy) -> list[CompletionCriterion]:+ return [+ criterion+ for criterion in policy.completion_criteria+ if criterion.kind == "outcome" and not criterion.method_mandated+ ]+++def _render_terminal_action_reconciliation_criteria(criteria: list[CompletionCriterion]) -> str:+ return json.dumps(+ [+ {+ "id": criterion.id,+ "outcome": criterion.outcome,+ "contingent_on": criterion.contingent_on,+ "implicit": criterion.implicit,+ "method_mandated": criterion.method_mandated,+ "level": criterion.level,+ "output_path": criterion.output_path,+ "kind": criterion.kind,+ }+ for criterion in criteria+ ],+ ensure_ascii=True,+ separators=(",", ":"),+ )+++def _policy_needs_terminal_action_reconciliation(policy: RequestPolicy) -> bool:+ if policy.requires_user_clarification:+ return False+ if policy.credential_input_kind not in _TERMINAL_ACTION_RECONCILABLE_CREDENTIAL_KINDS:+ return False+ return not any(+ criterion.kind == "terminal_action" and not criterion.method_mandated+ for criterion in policy.completion_criteria+ )+++def _log_terminal_action_reconciliation_omitted(+ policy: RequestPolicy,+ *,+ reason: str,+ failure_kind: str = "none",+ retry_count: int = 0,+) -> None:+ LOG.info(+ "copilot_terminal_action_reconciliation_omitted",+ reason=reason,+ failure_kind=failure_kind,+ retry_count=retry_count,+ credential_input_kind=policy.credential_input_kind,+ completion_criteria_count=len(policy.completion_criteria),+ )+++async def _reconcile_missing_terminal_action(+ policy: RequestPolicy,+ *,+ user_message: str,+ handler: LLMAPIHandler,+) -> None:+ if not _policy_needs_terminal_action_reconciliation(policy):+ return+ candidates = _terminal_action_reconciliation_candidates(policy)+ if not candidates:+ _log_terminal_action_reconciliation_omitted(policy, reason="no_eligible_criteria")+ return++ prompt = prompt_engine.load_prompt(+ template=PROMPT_NAME,+ terminal_action_reconciliation=True,+ user_message=escape_code_fences(redact_raw_secrets_for_prompt(user_message)),+ reconciliation_credential_input_kind=policy.credential_input_kind,+ reconciliation_criteria=escape_code_fences(_render_terminal_action_reconciliation_criteria(candidates)),+ )+ raw, failure_kind, retry_count = await _run_request_policy_classifier(+ handler,+ prompt,+ )+ if raw is None:+ _log_terminal_action_reconciliation_omitted(+ policy,+ reason="classifier_failed",+ failure_kind=failure_kind,+ retry_count=retry_count,+ )+ return+ reconciliation = _parse_terminal_action_reconciliation(raw)+ if reconciliation is None:+ _log_terminal_action_reconciliation_omitted(+ policy,+ reason="malformed_response",+ failure_kind=failure_kind,+ retry_count=retry_count,+ )+ return+ if reconciliation.criterion_id is None:+ _log_terminal_action_reconciliation_omitted(+ policy,+ reason="model_abstained",+ failure_kind=failure_kind,+ retry_count=retry_count,+ )+ return++ matching_indexes = [+ index for index, criterion in enumerate(candidates) if criterion.id == reconciliation.criterion_id+ ]+ if len(matching_indexes) != 1:+ _log_terminal_action_reconciliation_omitted(+ policy,+ reason="unknown_or_ambiguous_criterion",+ failure_kind=failure_kind,+ retry_count=retry_count,+ )+ return+ selected = candidates[matching_indexes[0]]+ policy_index = next(+ (index for index, criterion in enumerate(policy.completion_criteria) if criterion is selected),+ None,+ )+ if policy_index is None:+ _log_terminal_action_reconciliation_omitted(policy, reason="criterion_not_in_policy")+ return+ policy.completion_criteria[policy_index] = replace(+ selected,+ kind="terminal_action",+ terminal_action_family=reconciliation.terminal_action_family,+ terminal_action_verification_mode="semantic_outcome_v1",+ )+ LOG.info(+ "copilot_terminal_action_reconciled",+ criterion_id=selected.id,+ terminal_action_family=reconciliation.terminal_action_family,+ credential_input_kind=policy.credential_input_kind,+ completion_criteria_count=len(policy.completion_criteria),+ retry_count=retry_count,+ )++async def _classify_request(user_message: str,workflow_yaml: str,chat_history: list[WorkflowCopilotChatHistoryMessage],global_llm_context: str,- handler: Any,+ handler: LLMAPIHandler | None,*,active_criteria: list[CompletionCriterion] | None = None,config: CopilotConfig | None = None,@@ -2671,6 +2861,7 @@ async def _classify_request()prompt = prompt_engine.load_prompt(template=PROMPT_NAME,+ terminal_action_reconciliation=False,user_message=escape_code_fences(safe_user_message),raw_secret_present=str(raw_secret_present).lower(),workflow_yaml=escape_code_fences(redact_raw_secrets_for_prompt(workflow_yaml)[:2048]),@@ -2691,7 +2882,12 @@ async def _classify_request(separators=(",", ":"),),)- raw, failure_kind, retry_count = await _run_request_policy_classifier(handler, prompt)+ classifier_deadline = time.monotonic() + settings.COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDS+ raw, failure_kind, retry_count = await _run_request_policy_classifier(+ handler,+ prompt,+ deadline=classifier_deadline,+ )if raw is None:LOG.warning("request-policy classifier failed", failure_kind=failure_kind, retry_count=retry_count)return _classifier_fallback_policy(@@ -2835,6 +3031,11 @@ async def _classify_request(policy.requires_user_clarification = Falsepolicy.raw_secret_evidence = None_degrade_pathless_contingent_criteria(policy)+ await _reconcile_missing_terminal_action(+ policy,+ user_message=safe_user_message,+ handler=handler,+ )policy.completion_contract_status = ("present" if policy.completion_contract or policy.graded_completion_criteria() else "absent")@@ -3325,7 +3526,7 @@ async def build_request_policy(chat_history: list[WorkflowCopilotChatHistoryMessage],global_llm_context: str,organization_id: str,- handler: Any,+ handler: LLMAPIHandler | None,active_criteria: list[CompletionCriterion] | None = None,config: CopilotConfig | None = None,) -> RequestPolicy:diff --git a/tests/unit/test_copilot_code_block_persist_seam.py b/tests/unit/test_copilot_code_block_persist_seam.pyindex ed8247c2c..5eddfe89e 100644--- a/tests/unit/test_copilot_code_block_persist_seam.py+++ b/tests/unit/test_copilot_code_block_persist_seam.py@@ -16458,7 +16458,7 @@ class TestTerminalActionScoutGate:def test_login_prefix_with_unreached_terminal_action_is_not_goal_complete(self) -> None:ctx = self._login_prefix_ctx(self._terminal_action_criterion())- assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is Falseassert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is Falsedef test_login_is_the_whole_goal_stays_goal_complete(self) -> None:@@ -16469,10 +16469,10 @@ class TestTerminalActionScoutGate:ctx = self._login_prefix_ctx(self._terminal_action_criterion(method_mandated=True))assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True- def test_scout_observed_terminal_action_releases_goal_complete(self) -> None:+ def test_observed_terminal_action_does_not_bypass_post_credential_floor(self) -> None:ctx = self._login_prefix_ctx(self._terminal_action_criterion())ctx.scout_observed_terminal_criterion_ids = {"start_service_request"}- assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True+ assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is Falsedef test_post_credential_business_spine_records_terminal_action_observation(self) -> None:ctx = self._login_prefix_ctx(self._terminal_action_criterion())@@ -16482,6 +16482,176 @@ class TestTerminalActionScoutGate:assert ctx.scout_observed_terminal_criterion_ids == {"start_service_request"}assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True+ def test_post_login_open_then_submit_reaches_terminal_action(self) -> None:+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())+ ctx.scout_trajectory = list(ctx.scout_trajectory) + [+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 4},+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 5},+ ]++ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}+ enforcement_module.record_reached_terminal_action_observation(ctx)+ assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True++ def test_post_login_three_click_business_spine_reaches_terminal_action(self) -> None:+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())+ ctx.scout_trajectory = list(ctx.scout_trajectory) + [+ {"tool_name": "click", "selector": "#open-item", "trajectory_index": 4},+ {"tool_name": "click", "selector": "#add-to-cart", "trajectory_index": 5},+ {"tool_name": "click", "selector": "#place-order", "trajectory_index": 6},+ ]++ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}++ def test_post_login_non_committing_click_sequence_stays_gated(self) -> None:+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())+ ctx.scout_trajectory = list(ctx.scout_trajectory) + [+ {"tool_name": "click", "selector": "button", "role": "button", "trajectory_index": 4},+ {"tool_name": "click", "selector": "button", "role": "button", "trajectory_index": 5},+ {"tool_name": "click", "selector": "button", "role": "button", "trajectory_index": 6},+ ]++ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is False+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == set()++ def test_sourceless_enter_login_submit_establishes_business_boundary(self) -> None:+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())+ ctx.scout_trajectory = [+ {+ "tool_name": "fill_credential_field",+ "credential_id": "cred_1",+ "credential_field": "username",+ "trajectory_index": 0,+ },+ {"tool_name": "press_key", "key": "Enter", "trajectory_index": 1},+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 2},+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 3},+ ]+ ctx.scouted_credential_field_inventory_by_credential_id = {"cred_1": frozenset({"username"})}++ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}++ def test_sourceless_non_enter_key_does_not_establish_login_boundary(self) -> None:+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())+ ctx.scout_trajectory = [+ {+ "tool_name": "fill_credential_field",+ "credential_id": "cred_1",+ "credential_field": "username",+ "trajectory_index": 0,+ },+ {"tool_name": "press_key", "key": "Tab", "trajectory_index": 1},+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 2},+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 3},+ ]+ ctx.scouted_credential_field_inventory_by_credential_id = {"cred_1": frozenset({"username"})}++ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is False+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == set()++ def test_post_login_boundary_ignores_divergent_sourced_click(self) -> None:+ login_url = "https://portal.example.test/login"+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())+ ctx.scout_trajectory = [+ {+ "tool_name": "fill_credential_field",+ "credential_id": "cred_1",+ "credential_field": "username",+ "source_url": login_url,+ "trajectory_index": 0,+ },+ {+ "tool_name": "click",+ "accessible_name": "Learn more",+ "source_url": "https://portal.example.test/help",+ "trajectory_index": 1,+ },+ {+ "tool_name": "click",+ "selector": "button[type='submit']",+ "source_url": login_url,+ "trajectory_index": 2,+ },+ {+ "tool_name": "click",+ "selector": "#open-request",+ "source_url": self._BUSINESS_URL,+ "trajectory_index": 3,+ },+ {+ "tool_name": "click",+ "selector": "#submit-request",+ "source_url": self._BUSINESS_URL,+ "trajectory_index": 4,+ },+ ]+ ctx.scouted_credential_field_inventory_by_credential_id = {"cred_1": frozenset({"username"})}++ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}++ def test_sourceless_divergent_click_uses_stable_login_submit_identity(self) -> None:+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())+ ctx.scout_trajectory = [+ {+ "tool_name": "fill_credential_field",+ "credential_id": "cred_1",+ "credential_field": "username",+ "trajectory_index": 0,+ },+ {+ "tool_name": "click",+ "selector": "#login-help",+ "accessible_name": "Trouble signing in?",+ "trajectory_index": 1,+ },+ {"tool_name": "click", "accessible_name": "Sign in", "trajectory_index": 2},+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 3},+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 4},+ ]+ ctx.scouted_credential_field_inventory_by_credential_id = {"cred_1": frozenset({"username"})}++ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}++ def test_mixed_fill_sources_use_latest_sourceless_login_identity(self) -> None:+ helper = TestCredentialScoutGapMatcher+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())+ ctx.scout_trajectory = [+ helper._fill("cred_1", "username", helper._PAGE_ONE),+ helper._fill("cred_1", "password", ""),+ {"tool_name": "click", "accessible_name": "Sign in", "trajectory_index": 2},+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 3},+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 4},+ ]++ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == {"start_service_request"}+ enforcement_module.record_reached_terminal_action_observation(ctx)+ assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True++ def test_sourceless_login_submit_without_stable_identity_fails_closed(self) -> None:+ ctx = self._login_prefix_ctx(self._terminal_action_criterion())+ ctx.scout_trajectory = [+ {+ "tool_name": "fill_credential_field",+ "credential_id": "cred_1",+ "credential_field": "username",+ "trajectory_index": 0,+ },+ {"tool_name": "click", "accessible_name": "Help", "trajectory_index": 1},+ {"tool_name": "click", "selector": "button", "trajectory_index": 2},+ {"tool_name": "click", "selector": "#open-request", "trajectory_index": 3},+ {"tool_name": "click", "selector": "#submit-request", "trajectory_index": 4},+ ]+ ctx.scouted_credential_field_inventory_by_credential_id = {"cred_1": frozenset({"username"})}++ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is False+ assert enforcement_module.reached_terminal_action_criterion_ids(ctx) == set()+def test_login_only_trajectory_records_no_terminal_action_observation(self) -> None:ctx = self._login_prefix_ctx(self._terminal_action_criterion())enforcement_module.record_reached_terminal_action_observation(ctx)diff --git a/tests/unit/test_copilot_completion_criteria_lifecycle.py b/tests/unit/test_copilot_completion_criteria_lifecycle.pyindex 628db0951..cc2882859 100644--- a/tests/unit/test_copilot_completion_criteria_lifecycle.py+++ b/tests/unit/test_copilot_completion_criteria_lifecycle.py@@ -5,6 +5,7 @@ of the judge-unavailable success bypass."""from __future__ import annotationsimport asyncio+from dataclasses import replacefrom unittest.mock import MagicMockimport pytest@@ -316,6 +317,143 @@ def test_criteria_json_round_trip_preserves_fields(criteria: tuple[CompletionCriassert criteria_from_json(criteria_to_json(criteria)) == criteria+def test_terminal_action_verification_mode_round_trips() -> None:+ criterion = CompletionCriterion(+ id="c0",+ outcome="The service request is created.",+ kind="terminal_action",+ terminal_action_family="request",+ terminal_action_verification_mode="semantic_outcome_v1",+ )++ raw = criteria_to_json([criterion])++ assert raw[0]["terminal_action_verification_mode"] == "semantic_outcome_v1"+ assert criteria_from_json(raw) == (criterion,)+ legacy = {key: value for key, value in raw[0].items() if key != "terminal_action_verification_mode"}+ assert criteria_from_json([legacy])[0].terminal_action_verification_mode == "family_record_v1"+ stored = StoredCriteriaSnapshot(+ active=StoredCriteriaSet(+ set_id="wccs_1",+ goal_epoch=1,+ criteria=(replace(criterion, terminal_action_verification_mode="family_record_v1"),),+ ),+ next_epoch=2,+ )+ decision = reconcile_completion_criteria(stored, [criterion], actionable=True)+ assert decision.action == "create"+ assert decision.criteria[0].terminal_action_verification_mode == "semantic_outcome_v1"+++def test_terminal_action_reconciliation_cannot_regress_semantic_authority() -> None:+ semantic = CompletionCriterion(+ id="c0",+ outcome="The service request is created.",+ kind="terminal_action",+ terminal_action_family="request",+ terminal_action_verification_mode="semantic_outcome_v1",+ )+ stored = StoredCriteriaSnapshot(+ active=StoredCriteriaSet(set_id="wccs_1", goal_epoch=1, criteria=(semantic,)),+ next_epoch=2,+ )++ decision = reconcile_completion_criteria(+ stored,+ [replace(semantic, terminal_action_verification_mode="family_record_v1")],+ actionable=True,+ )++ assert decision.action == "adopt_stored"+ assert decision.criteria == (semantic,)+++def test_terminal_action_reconciliation_abstention_cannot_drop_stored_semantic_authority() -> None:+ semantic = CompletionCriterion(+ id="c0",+ outcome="The service request is created.",+ kind="terminal_action",+ terminal_action_family="request",+ terminal_action_verification_mode="semantic_outcome_v1",+ )+ stored = StoredCriteriaSnapshot(+ active=StoredCriteriaSet(set_id="wccs_1", goal_epoch=1, criteria=(semantic,)),+ next_epoch=2,+ )++ decision = reconcile_completion_criteria(+ stored,+ [+ replace(+ semantic,+ kind="outcome",+ terminal_action_family=None,+ terminal_action_verification_mode="family_record_v1",+ )+ ],+ actionable=True,+ )++ assert decision.action == "adopt_stored"+ assert decision.criteria == (semantic,)+++def test_terminal_action_semantic_authority_survives_fresh_criterion_reindexing() -> None:+ semantic = CompletionCriterion(+ id="c0",+ outcome="The service request is created.",+ kind="terminal_action",+ terminal_action_family="request",+ terminal_action_verification_mode="semantic_outcome_v1",+ )+ stored = StoredCriteriaSnapshot(+ active=StoredCriteriaSet(set_id="wccs_1", goal_epoch=1, criteria=(semantic,)),+ next_epoch=2,+ )+ authenticated = CompletionCriterion(id="c0", outcome="The user is authenticated.")+ reindexed = replace(+ semantic,+ id="c1",+ kind="outcome",+ terminal_action_family=None,+ terminal_action_verification_mode="family_record_v1",+ )++ decision = reconcile_completion_criteria(stored, [authenticated, reindexed], actionable=True)++ assert decision.action == "create"+ assert [+ (criterion.id, criterion.kind, criterion.terminal_action_verification_mode) for criterion in decision.criteria+ ] == [+ ("c0", "outcome", "family_record_v1"),+ ("c1", "terminal_action", "semantic_outcome_v1"),+ ]+++def test_expanded_criteria_preserve_stored_terminal_action_semantic_authority() -> None:+ semantic = CompletionCriterion(+ id="c0",+ outcome="The service request is created.",+ kind="terminal_action",+ terminal_action_family="request",+ terminal_action_verification_mode="semantic_outcome_v1",+ )+ added = CompletionCriterion(id="c1", outcome="The confirmation number is returned.")+ stored = StoredCriteriaSnapshot(+ active=StoredCriteriaSet(set_id="wccs_1", goal_epoch=1, criteria=(semantic,)),+ next_epoch=2,+ )++ decision = reconcile_completion_criteria(+ stored,+ [replace(semantic, terminal_action_verification_mode="family_record_v1"), added],+ actionable=True,+ )++ assert decision.action == "create"+ assert decision.criteria == (semantic, added)++def test_typed_criteria_list_round_trip_preserves_grading_metadata() -> None:criterion = CompletionCriterion(id="c0",diff --git a/tests/unit/test_copilot_completion_verification.py b/tests/unit/test_copilot_completion_verification.pyindex 0651ba91d..798625580 100644--- a/tests/unit/test_copilot_completion_verification.py+++ b/tests/unit/test_copilot_completion_verification.py@@ -1588,6 +1588,49 @@ def test_terminal_goal_record_satisfies_flat_submit_payload() -> None:]+async def test_reconciled_terminal_action_requires_semantic_verification_authority() -> None:+ criterion = CompletionCriterion(+ id="c0",+ outcome="The requested service setup is created.",+ kind="terminal_action",+ terminal_action_family="request",+ terminal_action_verification_mode="semantic_outcome_v1",+ )+ snapshot = RunEvidenceSnapshot(+ block_outputs={+ "terminal_result": {+ "request_id": "QC-2002-DEMO",+ "submitted": True,+ "submission_result": "created",+ }+ }+ )++ assert grade_terminal_goal_record_criteria([criterion], snapshot) == []+ captured_prompt = ""++ async def handler(*, prompt: str, prompt_name: str) -> dict[str, object]:+ nonlocal captured_prompt+ captured_prompt = prompt+ return {+ "verdicts": [+ {+ "criterion_id": "c0",+ "satisfied": True,+ "reason_code": "evidence_confirms",+ "evidence_ref": "block_outputs:terminal_result",+ }+ ]+ }++ result = await evaluate_completion_criteria([criterion], snapshot, handler)++ assert result.is_fully_satisfied() is True+ assert criterion.outcome in captured_prompt+ assert "QC-2002-DEMO" in captured_prompt++def test_terminal_goal_record_accepts_family_artifact_without_self_asserted_boolean() -> None:payload = _terminal_goal_payload(submitted=None)snapshot = RunEvidenceSnapshot(block_outputs={"submit_water_request": payload})diff --git a/tests/unit/test_copilot_request_policy_completion_criteria.py b/tests/unit/test_copilot_request_policy_completion_criteria.pyindex 9ce8820ed..94e1154aa 100644--- a/tests/unit/test_copilot_request_policy_completion_criteria.py+++ b/tests/unit/test_copilot_request_policy_completion_criteria.py@@ -2,11 +2,16 @@ from __future__ import annotationsimport hashlibimport json+from collections.abc import Awaitablefrom dataclasses import replace+from types import SimpleNamespacefrom typing import Anyimport pytest+from structlog.testing import capture_logs+from skyvern.forge.sdk.copilot import enforcement as enforcement_module+from skyvern.forge.sdk.copilot import request_policy as request_policy_modulefrom skyvern.forge.sdk.copilot.completion_criteria_store import (StoredCriteriaSet,StoredCriteriaSnapshot,@@ -15,7 +20,8 @@ from skyvern.forge.sdk.copilot.completion_criteria_store import (criteria_to_json,reconcile_completion_criteria,)-from skyvern.forge.sdk.copilot.config import CopilotConfig+from skyvern.forge.sdk.copilot.config import BlockAuthoringPolicy, CopilotConfig+from skyvern.forge.sdk.copilot.context import CopilotContextfrom skyvern.forge.sdk.copilot.request_policy import (REGISTERED_DOWNLOAD_COMPLETION_CRITERION_ID,CompletionCriterion,@@ -33,8 +39,79 @@ from skyvern.forge.sdk.copilot.request_policy import (is_fallback_floor_base_criterion,request_policy_has_present_completion_contract,)+from skyvern.forge.sdk.copilot.tools import workflow_update as workflow_update_module+from skyvern.forge.sdk.copilot.turn_intent import TurnIntent, TurnIntentAuthority, TurnIntentMode+from tests.unit.conftest import make_copilot_contextfrom tests.unit.copilot_test_helpers import make_completion_criterion as _criterion+_TERMINAL_ACTION_CREATE_OUTCOMES = (+ "A new service request is submitted.",+ "The requested service setup is created.",+ "The account's start-service request is created.",+)+_TERMINAL_ACTION_ABSTAIN = {"version": "1", "criterion_id": None, "terminal_action_family": None}+++def _terminal_action_packet(*outcomes: str) -> dict[str, object]:+ return {+ "testing_intent": "require_test",+ "credential_input_kind": "credential_name",+ "credential_refs": ["Portal Login"],+ "requires_user_clarification": False,+ "completion_criteria": [+ {+ "outcome": outcome,+ "implicit": False,+ "method_mandated": False,+ "level": "run",+ "kind": "outcome",+ "terminal_action_family": None,+ }+ for outcome in outcomes+ ],+ }+++async def _classify_with_terminal_action_reconciliation(+ user_message: str,+ packet: dict[str, object],+ reconciliation: dict[str, object],+) -> tuple[RequestPolicy, list[str]]:+ prompts: list[str] = []++ async def handler(prompt: str, prompt_name: str) -> dict[str, object]:+ assert prompt_name == request_policy_module.PROMPT_NAME+ prompts.append(prompt)+ if "TERMINAL ACTION RECONCILIATION MODE" in prompt:+ return reconciliation+ return packet++ policy = await request_policy_module._classify_request(user_message, "", [], "", handler)+ return policy, prompts+++def _terminal_action_enforcement_ctx(criteria: tuple[CompletionCriterion, ...]) -> CopilotContext:+ ctx = make_copilot_context()+ ctx.turn_intent = TurnIntent(+ mode=TurnIntentMode.BUILD,+ authority=TurnIntentAuthority(may_update_workflow=True, may_run_blocks=True),+ )+ ctx.block_authoring_policy = BlockAuthoringPolicy.CODE_ONLY_BROWSER+ ctx.completion_criteria_turn_state = SimpleNamespace(decision=SimpleNamespace(criteria=criteria))+ ctx.scout_trajectory = [+ {+ "tool_name": "fill_credential_field",+ "credential_id": "credential_ref",+ "credential_field": "username",+ "trajectory_index": 0,+ },+ {"tool_name": "click", "accessible_name": "Sign in", "trajectory_index": 1},+ ]+ ctx.synthesized_block_offered = True+ ctx.synthesized_block_offered_trajectory_len = len(ctx.scout_trajectory)+ ctx.synthesized_block_offered_goal_complete = enforcement_module.synthesized_trajectory_is_goal_complete(ctx)+ return ctx+async def _policy_for_message(user_message: str,@@ -86,6 +163,115 @@ def _criteria_fingerprint(criteria: list[CompletionCriterion]) -> str:return hashlib.sha256(payload.encode("utf-8")).hexdigest()[email protected]("outcome", _TERMINAL_ACTION_CREATE_OUTCOMES)+async def test_credentialed_create_omission_is_reconciled_before_custody_and_foreclosure(outcome: str) -> None:+ policy, prompts = await _classify_with_terminal_action_reconciliation(+ "Sign in with the saved credential and create the requested service setup.",+ _terminal_action_packet(outcome),+ {"version": "1", "criterion_id": "c0", "terminal_action_family": "request"},+ )++ assert len(prompts) == 2+ assert '"id":"c0"' in prompts[1] and '"kind":"outcome"' in prompts[1]+ stored = criteria_from_json(criteria_to_json(policy.completion_criteria))+ assert [+ (+ item.id,+ item.outcome,+ item.kind,+ item.terminal_action_family,+ item.terminal_action_verification_mode,+ )+ for item in stored+ ] == [("c0", outcome, "terminal_action", "request", "semantic_outcome_v1")]+ assert stored[0].method_mandated is False+ ctx = _terminal_action_enforcement_ctx(stored)+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is False+ assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is False+ assert enforcement_module._should_block_mutating_tool_after_synthesized_offer(ctx, "click") is False++ ctx.update_workflow_called = True+ with capture_logs() as logs:+ workflow_update_module._log_imposition_skipped_after_update(ctx)+ (event,) = (entry for entry in logs if entry["event"] == "copilot_imposition_skipped_after_update")+ assert (event["reaches_goal"], event["goal_complete"]) == (False, False)+++ ("user_message", "outcome"),+ [+ ("Sign in with the saved credential named Portal Login.", "The user is authenticated."),+ ("Sign in with the saved credential and complete MFA.", "Authentication including MFA completes."),+ ],+)+async def test_login_only_reconciliation_abstains_and_keeps_generic_goal_floor(user_message: str, outcome: str) -> None:+ with capture_logs() as logs:+ policy, prompts = await _classify_with_terminal_action_reconciliation(+ user_message, _terminal_action_packet(outcome), _TERMINAL_ACTION_ABSTAIN+ )++ assert len(prompts) == 2+ assert [(item.id, item.outcome, item.kind) for item in policy.completion_criteria] == [("c0", outcome, "outcome")]+ assert [+ entry["reason"] for entry in logs if entry["event"] == "copilot_terminal_action_reconciliation_omitted"+ ] == ["model_abstained"]+ ctx = _terminal_action_enforcement_ctx(tuple(policy.completion_criteria))+ assert enforcement_module.synthesized_trajectory_reaches_goal(ctx) is True+ assert enforcement_module.synthesized_trajectory_is_goal_complete(ctx) is True+++async def test_ambiguous_terminal_action_reconciliation_abstains_without_rekeying() -> None:+ malformed = {"version": "1", "criterion_id": ["c0", "c1"], "terminal_action_family": "request"}+ with capture_logs() as logs:+ policy, _ = await _classify_with_terminal_action_reconciliation(+ "Sign in and create a service request.",+ _terminal_action_packet("A service request is created.", "The created request is visible."),+ malformed,+ )++ assert [(item.id, item.kind) for item in policy.completion_criteria] == [("c0", "outcome"), ("c1", "outcome")]+ assert [+ entry["reason"] for entry in logs if entry["event"] == "copilot_terminal_action_reconciliation_omitted"+ ] == ["malformed_response"]+++async def test_terminal_action_reconciliation_gets_its_own_classifier_deadline(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ timeouts: list[float] = []+ # The initial classification may be followed by request-slot correction and+ # production work that legitimately outlives its budget. Reconciliation is+ # a distinct classifier stage and must not inherit that expired deadline.+ monotonic_values = iter((100.0, 101.0, 112.0, 113.0))++ async def bounded_wait_for(awaitable: Awaitable[object], *, timeout: float) -> object:+ timeouts.append(timeout)+ return await awaitable++ timeout_error = request_policy_module.asyncio.TimeoutError+ monkeypatch.setattr(request_policy_module.settings, "COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDS", 10.0)+ monkeypatch.setattr(request_policy_module, "time", SimpleNamespace(monotonic=lambda: next(monotonic_values)))+ monkeypatch.setattr(+ request_policy_module,+ "asyncio",+ SimpleNamespace(wait_for=bounded_wait_for, TimeoutError=timeout_error),+ )++ policy, prompts = await _classify_with_terminal_action_reconciliation(+ "Sign in with the saved credential and create the requested service setup.",+ _terminal_action_packet("The requested service setup is created."),+ {"version": "1", "criterion_id": "c0", "terminal_action_family": "request"},+ )++ assert len(prompts) == 2+ assert [(criterion.id, criterion.kind) for criterion in policy.completion_criteria] == [("c0", "terminal_action")]+ assert timeouts == [9.0, 9.0]++def test_p9_exact_value_unpinnable_output_mint_degrades() -> None:policy = RequestPolicy(completion_criteria=[diff --git a/tests/unit/test_copilot_request_policy_prompt.py b/tests/unit/test_copilot_request_policy_prompt.pyindex 8defa4514..4ffacfde5 100644--- a/tests/unit/test_copilot_request_policy_prompt.py+++ b/tests/unit/test_copilot_request_policy_prompt.py@@ -44,6 +44,16 @@ def _render(**overrides: str) -> str:)+def _render_terminal_action_reconciliation() -> str:+ return prompt_engine.load_prompt(+ template=PROMPT_NAME,+ terminal_action_reconciliation=True,+ user_message="Sign in with a saved credential and create a service request.",+ reconciliation_credential_input_kind="credential_name",+ reconciliation_criteria='[{"id":"c0","outcome":"A service request is created.","kind":"outcome"}]',+ )++class TestRequestPolicyPromptStructure:def test_existing_login_workflow_without_credentials_clause_still_present(self) -> None:rendered = _render()@@ -115,6 +125,24 @@ class TestRequestPolicyPromptStructure:assert "classification_output_key=login_only and expected_classification=true" in renderedassert 'The only supported non-null value is "registered_download"' in rendered+ def test_terminal_action_guidance_distinguishes_order_from_login_prefix(self) -> None:+ rendered = _render()++ assert "place an order" in rendered+ assert "replace that post-login end state" in rendered+ assert "successful authentication" in rendered+ assert "whose only requested end state is authentication" in rendered++ def test_terminal_action_reconciliation_mode_is_bounded_to_existing_criteria(self) -> None:+ rendered = _render_terminal_action_reconciliation()++ assert "TERMINAL ACTION RECONCILIATION MODE" in rendered+ assert '"criterion_id":"c0"' in rendered+ assert "select exactly one existing criterion ID" in rendered+ assert "do not invent, remove, duplicate, paraphrase, or re-order criteria" in rendered+ assert "successful authentication itself, including MFA" in rendered+ assert "placing an order" in rendered+def test_completion_criteria_schema_exposes_judgment_truth_condition_fields(self) -> None:rendered = _render()assert "judgment_predicate: null unless expected_output_shape=goal_judgment_boolean" in rendered