Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
SKY-11559: Verify validation classifications deterministically (#6902)
Andrew Neilsone1f190a98597
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2index 0da6518db..b0391b3f2 100644--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2@@ -164,6 +164,7 @@ IMPORTANT WORKFLOW RULES:* When extracting table-like or list-like records (locations, affiliations, prices, inventory, line items, statuses), iterate the actual row/item elements (`tr`, `[role="row"]`, or equivalent repeated item containers) and read fields from that same row/item. Do NOT scan broad containers such as `section`, `.card`, `article`, or `li` and then infer fields from mixed text; that can combine several rows into one record. Derive summary fields only from the parsed row objects, not from broad page text.* When a successful search/open-details block reaches a details or profile page, include the visible details/profile page label in the block's returned `evidence_text`. Keep structured record fields focused on the requested data; use the page label only as supporting evidence.* When a validation-only workflow must stop on a pre-submit Review page, the terminal Review block must return page-grounded values, not only booleans. Include an explicit validation-only marker such as `validation_only: true` or `submit_mode: "validation_only"`, include `review_values` or `review_fields` as visible Review-page label/value strings, include `evidence_text` containing the visible Review-page text that verbatim contains those values, include an explicit false submit/finalize-click signal, and do not click Submit/Finalize.+* When REQUEST POLICY includes `validation_classification_output_contracts`, those contracts are mandatory code-block return fields. For each contract, the code block that classifies the run MUST explicitly `return` a dict containing the exact `return_key` as a top-level key, with the exact JSON type/value shown by `expected_value` when the classification is true for the reached page. Do not rely on schema defaults or exposed top-level locals for these values. The matching `code_artifact_metadata` row for that block must declare the same `return_key` under `goal_value_paths` and in `extraction_schema`; update_and_run_blocks will reject code-only output blocks without that metadata. For login-gated validation, also return page-grounded display fields such as `visible_page_path_label` with the full visible section/breadcrumb label and `observed_gate_phrase` with the exact visible sign-in/register gate phrase when present. Synonyms such as `path_login_only`, `classification`, or prose in `evidence_text` do not replace the required key unless that exact key is the contract's `return_key`.* Save the complete reusable workflow, but run only a small verified frontier when the workflow has several page-changing stages. `update_and_run_blocks` accepts the full YAML plus a subset of `block_labels`; for form/search workflows, first run the entry/form-state frontier, then continue with submit/results, then expansion/extraction. After a verified prefix or a budgeted run that leaves the browser on a later page, preserve the upstream workflow text and run only the missing downstream labels. Do not remove later blocks from the YAML just to shorten a test; do not submit a long chain of all blocks at once when each stage can be verified independently.* Set `complete_criterion` on every decomposed `navigation` block to the observable post-action page state — without it, smaller blocks still return vague "continue" verdicts.* Use descriptive, unique labels for blocks (snake_case format).diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2index dd75a36d3..ef0590b24 100644--- a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2@@ -25,15 +25,18 @@ completion_contract:- Set to null when the user does not provide an explicit completion condition.completion_criteria:-- An array of the END-STATE OUTCOMES that must be true for the user's goal to be done. Each item is {outcome, contingent_on, contingent_antecedent_output_path, deliverable_kind, implicit, method_mandated, level, output_path, expected_output_value, expected_output_shape, kind, terminal_action_family}.+- An array of the END-STATE OUTCOMES that must be true for the user's goal to be done. Each item is {outcome, contingent_on, contingent_antecedent_output_path, deliverable_kind, implicit, method_mandated, level, output_path, expected_output_value, expected_output_shape, kind, terminal_action_family, classification_output_key, expected_classification}.- outcome: a short statement of an end state the workflow must achieve (an END, not a step). Extract the goal as outcomes the user wants to be true, not the recipe they describe.- contingent_on: null for ordinary criteria. Use a short antecedent string only when this criterion should be graded only if that antecedent happens, for example {"contingent_on": "the provider site blocks online submission", "contingent_antecedent_output_path": "output.blocker", "outcome": "the blocker is reported to the user"}. The antecedent goes in contingent_on; the consequent to grade goes in outcome. Do not encode if/then criteria only in prose.- contingent_antecedent_output_path: null unless the antecedent is represented by a structured run output field. The only valid form is output.<field> (for example output.blocker). Do not emit transcript paths, selectors, regexes, JSONPath, screenshots, nested paths, or prose conditions here.- deliverable_kind: null unless this criterion's requested output is satisfied by execution-owned browser-download registration rather than a user-authored extraction field. The only supported non-null value is "registered_download".- expected_output_value: null unless this criterion requires an exact structured output value the user explicitly provided. Put that value here as data; never hide it in outcome prose.- expected_output_shape: null unless this criterion requires a generated/provider structured output value whose exact value is not known before the run. Use only one of: reference_code, numeric_identifier, date, address, status_label, money_amount, owner_label. Do not use this as a non-empty-value proxy; leave null when the field's shape is not one of those declared types.-- kind=outcome|terminal_action. Use terminal_action only when the criterion's end state is a terminal action in the external site for one of the supported families below. Use outcome for all other outcomes.-- terminal_action_family=request|application|form|order|null. Set the matching family when kind=terminal_action; set null when kind=outcome.+- kind=outcome|terminal_action|validation_classification. Use validation_classification only when the user's no-side-effect goal is itself a judgment/classification returned by the workflow, such as whether a path is login-gated. Use terminal_action only when the criterion's end state is a terminal action in the external site for one of the supported families below. Use outcome for all other outcomes.+- terminal_action_family=request|application|form|order|null. Set the matching family when kind=terminal_action; set null otherwise.+- classification_output_key: null unless kind=validation_classification. Set it to the exact top-level block output field/key that will carry the classification scalar. Do not use JSONPath, regex, transcript paths, screenshots, prose conditions, or nested paths.+- expected_classification: null unless kind=validation_classification. Set it to the exact expected classification target as a boolean or canonical enum/string value. Do not hide this target in outcome prose.+- For validation-only utility/path checks whose goal is to classify whether the reachable path is login-only/login-gated, prefer kind=validation_classification with classification_output_key=login_only and expected_classification=true. Do not represent that classification target as a requested-output criterion such as output.login_only.- Capture ALL distinct outcomes the goal requires, including implicit constraints the user did not state as a step (for example "added exactly once / not duplicated", "the item is present in the cart"). Mark such items implicit=true.- A prescribed HOW step is NOT a criterion. "search for X", "open the result lightbox", "click add to cart", "go to the cart page" are methods/navigation, not outcomes — do not list them. A different or shorter path that reaches the same end state must still satisfy the goal.- Telling/showing/reporting results to the user is NOT a criterion. "tell me what it extracted", "show me the data", "confirm to me it worked" describe the chat reply, which no workflow definition or run can evidence — capture only the underlying workflow outcome (the data is extracted into the run output, the form is submitted) and list nothing for the reporting itself.diff --git a/skyvern/forge/sdk/copilot/agent.py b/skyvern/forge/sdk/copilot/agent.pyindex 91c50777a..355d954bf 100644--- a/skyvern/forge/sdk/copilot/agent.py+++ b/skyvern/forge/sdk/copilot/agent.py@@ -1491,7 +1491,7 @@ async def _build_goal_satisfied_exit_result(ctx: CopilotContext, global_llm_contverified_workflow, verified_yaml = _verified_workflow_or_none(ctx)clean_test = ctx.last_test_ok is True and ctx.last_full_workflow_test_ok is Trueif clean_test and verified_goal_claim_authorized(ctx):- user_response = _VERIFIED_WORKFLOW_SUCCESS_REPLY+ user_response = _verified_workflow_success_reply(ctx)elif clean_test:user_response = ("I built the workflow and the test run completed, but the goal outcome was not "@@ -1687,6 +1687,120 @@ _UNVALIDATED_PROPOSAL_AFFORDANCE = ("I have a draft workflow proposal. Use Review to inspect it, Accept to save it, or Reject to discard it. ""It has not been tested or verified end-to-end.")+_VERIFIED_CLASSIFICATION_CONTEXT_KEYS = (+ "visible_page_path_label",+ "safest_reachable_next_step",+ "recommended_next_action",+)+_VERIFIED_CLASSIFICATION_GATE_KEYS = (+ "observed_gate_phrase",+ "gate_phrase",+ "gate_summary",+ "blocked_by",+)+_VERIFIED_CLASSIFICATION_GATE_PHRASES = ("Sign in or register to continue",)+_VERIFIED_TERMINAL_VALUE_MAX_CHARS = 180+++def _terminal_summary_scalar(value: Any) -> str | None:+ if isinstance(value, bool):+ return "true" if value else "false"+ if isinstance(value, int | float) and not isinstance(value, bool):+ return str(value)+ if not isinstance(value, str):+ return None+ cleaned = " ".join(value.strip().split())+ if not cleaned or contains_internal_machinery_leak(cleaned):+ return None+ if len(cleaned) > _VERIFIED_TERMINAL_VALUE_MAX_CHARS:+ cleaned = cleaned[: _VERIFIED_TERMINAL_VALUE_MAX_CHARS - 1].rstrip() + "..."+ return cleaned+++def _verified_output_value(ctx: CopilotContext, output_key: str | None) -> Any:+ if not output_key:+ return None+ block_outputs = ctx.verified_terminal_block_outputs or ctx.verified_block_outputs or {}+ for output in block_outputs.values():+ if isinstance(output, dict) and output_key in output:+ return output.get(output_key)+ return None+++def _verified_adjacent_output_parts(ctx: CopilotContext) -> list[str]:+ parts: list[str] = []+ block_outputs = ctx.verified_terminal_block_outputs or ctx.verified_block_outputs or {}+ login_gate_phrase_found = False++ def append_gate_phrase(value: Any) -> None:+ nonlocal login_gate_phrase_found+ if isinstance(value, str):+ for phrase in _VERIFIED_CLASSIFICATION_GATE_PHRASES:+ if phrase.lower() in value.lower():+ login_gate_phrase_found = True+ parts.append(f"observed_gate_phrase={phrase}")+ return+ if isinstance(value, list | tuple | set):+ for item in value:+ append_gate_phrase(item)++ for output in block_outputs.values():+ if not isinstance(output, dict):+ continue+ for key in (*_VERIFIED_CLASSIFICATION_CONTEXT_KEYS, *_VERIFIED_CLASSIFICATION_GATE_KEYS):+ value = _terminal_summary_scalar(output.get(key))+ if value is not None:+ parts.append(f"{key}={value}")+ for value in output.values():+ append_gate_phrase(value)+ if login_gate_phrase_found:+ parts.append("login-gated")+ return list(dict.fromkeys(parts))+++def _verified_classification_summary(ctx: CopilotContext) -> str | None:+ if not verified_goal_claim_authorized(ctx) or not outcome_fully_verified(ctx):+ return None+ result = ctx.completion_verification_result+ if result is None or not result.is_fully_satisfied():+ return None+ policy = ctx.request_policy if isinstance(ctx.request_policy, RequestPolicy) else None+ if policy is None:+ return None+ verdict_by_id = {verdict.criterion_id: verdict for verdict in result.verdicts}+ parts: list[str] = []+ login_gated_confirmed = False+ for criterion in policy.completion_criteria:+ if criterion.kind != "validation_classification":+ continue+ verdict = verdict_by_id.get(criterion.id)+ if verdict is None or not verdict.satisfied:+ continue+ output_key = criterion.classification_output_key or verdict.output_path+ actual = _verified_output_value(ctx, output_key)+ display_value = _terminal_summary_scalar(actual)+ if output_key and display_value is not None:+ parts.append(f"{output_key}={display_value}")+ actual_text = actual.strip().lower() if isinstance(actual, str) else None+ if (+ isinstance(output_key, str)+ and output_key in {"login_only", "login_gated", "path_login_only"}+ and actual is True+ ) or actual_text in {"login-gated", "login_gated"}:+ login_gated_confirmed = True+ if login_gated_confirmed:+ parts.append("login-gated")+ parts.extend(_verified_adjacent_output_parts(ctx))+ if not parts:+ return None+ return "; ".join(dict.fromkeys(parts))+++def _verified_workflow_success_reply(ctx: CopilotContext) -> str:+ summary = _verified_classification_summary(ctx)+ if summary is None:+ return _VERIFIED_WORKFLOW_SUCCESS_REPLY+ return f"{_VERIFIED_WORKFLOW_SUCCESS_REPLY} Verified result: {summary}."@dataclass(frozen=True)@@ -1722,7 +1836,7 @@ def _render_typed_run_outcome_reply(if not has_verified_workflow or not verified_goal_claim_authorized(ctx):return Nonereturn _TypedRunOutcomeReply(- user_response=f"{_VERIFIED_WORKFLOW_SUCCESS_REPLY} The latest run demonstrated the requested outcome.",+ user_response=f"{_verified_workflow_success_reply(ctx)} The latest run demonstrated the requested outcome.",demonstrated=True,)@@ -1785,7 +1899,7 @@ def _verified_terminal_preserve_result(if verified_workflow is None:return Nonefinal_text, outcome = apply_repeated_reply_guard(- final_text=_VERIFIED_WORKFLOW_SUCCESS_REPLY,+ final_text=_verified_workflow_success_reply(ctx),attempted_kind=derive_response_kind(ctx.turn_intent),blocked_signatures=ctx.blocked_reply_signatures,terminal_reason="verified_goal_satisfied",@@ -2715,7 +2829,7 @@ async def _translate_to_agent_result()if verified_terminal_ready:resp_type = "REPLY"- user_response = _VERIFIED_WORKFLOW_SUCCESS_REPLY+ user_response = _verified_workflow_success_reply(ctx)agent_admits_incomplete = Falseelse:# Default-true preserves backwards-compat with stale prompts and missing fields.diff --git a/skyvern/forge/sdk/copilot/completion_criteria_store.py b/skyvern/forge/sdk/copilot/completion_criteria_store.pyindex c4ac27145..aabae500d 100644--- a/skyvern/forge/sdk/copilot/completion_criteria_store.py+++ b/skyvern/forge/sdk/copilot/completion_criteria_store.py@@ -20,6 +20,8 @@ from skyvern.forge.sdk.copilot.request_policy import (CriterionKind,ExpectedOutputShape,TerminalActionFamily,+ _coerce_classification_output_key,+ _coerce_expected_classification,_coerce_expected_output_shape,_normalize_contingent_antecedent_output_path,_normalize_deliverable_kind,@@ -36,7 +38,7 @@ CRITERIA_SET_STATUS_ACTIVE = "active"CRITERIA_SET_STATUS_SUPERSEDED = "superseded"TRIPWIRE_CONSECUTIVE_ALL_NO_EVIDENCE = 2_CRITERION_LEVELS = ("definition", "run")-_CRITERION_KINDS = ("outcome", "terminal_action")+_CRITERION_KINDS = ("outcome", "terminal_action", "validation_classification")_TERMINAL_ACTION_FAMILIES = ("request", "application", "form", "order")@@ -129,6 +131,8 @@ def criteria_to_json(criteria: tuple[CompletionCriterion, ...] | list[Completion"expected_output_shape": criterion.expected_output_shape,"kind": criterion.kind,"terminal_action_family": criterion.terminal_action_family,+ "classification_output_key": criterion.classification_output_key,+ "expected_classification": criterion.expected_classification,}for criterion in criteria]@@ -149,6 +153,8 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:output_path = item.get("output_path")expected_output_value = item.get("expected_output_value")expected_output_shape = _coerce_expected_output_shape(item.get("expected_output_shape"))+ classification_output_key = _coerce_classification_output_key(item.get("classification_output_key"))+ expected_classification = _coerce_expected_classification(item.get("expected_classification"))contingent_on = item.get("contingent_on")contingent_antecedent_output_path = _normalize_contingent_antecedent_output_path(item.get("contingent_antecedent_output_path")@@ -159,6 +165,17 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:terminal_action_family = (family_raw if kind == "terminal_action" and family_raw in _TERMINAL_ACTION_FAMILIES else None)+ stored_output_path = output_path.strip() if isinstance(output_path, str) and output_path.strip() else None+ stored_expected_output_value = (+ expected_output_value.strip()+ if isinstance(expected_output_value, str) and expected_output_value.strip()+ else None+ )+ stored_expected_output_shape = cast(ExpectedOutputShape | None, expected_output_shape)+ if kind == "validation_classification":+ stored_output_path = None+ stored_expected_output_value = None+ stored_expected_output_shape = Nonecriteria.append(CompletionCriterion(id=criterion_id,@@ -171,13 +188,13 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:implicit=bool(item.get("implicit")),method_mandated=bool(item.get("method_mandated")),level=level if isinstance(level, str) and level in _CRITERION_LEVELS else "run", # type: ignore[arg-type]- output_path=output_path.strip() if isinstance(output_path, str) and output_path.strip() else None,- expected_output_value=expected_output_value.strip()- if isinstance(expected_output_value, str) and expected_output_value.strip()- else None,- expected_output_shape=cast(ExpectedOutputShape | None, expected_output_shape),+ output_path=stored_output_path,+ expected_output_value=stored_expected_output_value,+ expected_output_shape=stored_expected_output_shape,kind=cast(CriterionKind, kind),terminal_action_family=cast(TerminalActionFamily | None, terminal_action_family),+ classification_output_key=classification_output_key,+ expected_classification=expected_classification,))return tuple(criteria)@@ -189,6 +206,10 @@ def _criterion_reconcile_key(criterion: CompletionCriterion) -> str:deliverable_kind_key = criterion.deliverable_kind or ""expected_output_value_key = criterion.expected_output_value or ""expected_output_shape_key = criterion.expected_output_shape or ""+ classification_output_key = criterion.classification_output_key or ""+ expected_classification_key = (+ str(criterion.expected_classification) if criterion.expected_classification is not None else ""+ )if criterion.output_path:return (f"contingent:{contingent_key}\x1fantecedent_path:{contingent_path_key}"@@ -196,10 +217,22 @@ def _criterion_reconcile_key(criterion: CompletionCriterion) -> str:f"\x1foutput_path:{criterion.output_path}"f"\x1fexpected_output_value:{expected_output_value_key}"f"\x1fexpected_output_shape:{expected_output_shape_key}"+ f"\x1fkind:{criterion.kind}"+ f"\x1fclassification_output_key:{classification_output_key}"+ f"\x1fexpected_classification:{expected_classification_key}"+ )+ if criterion.kind == "validation_classification":+ return (+ f"contingent:{contingent_key}\x1fantecedent_path:{contingent_path_key}"+ f"\x1fdeliverable_kind:{deliverable_kind_key}"+ f"\x1fkind:{criterion.kind}"+ f"\x1fclassification_output_key:{classification_output_key}"+ f"\x1fexpected_classification:{expected_classification_key}")return (f"contingent:{contingent_key}\x1fantecedent_path:{contingent_path_key}"f"\x1fdeliverable_kind:{deliverable_kind_key}"+ f"\x1fkind:{criterion.kind}"f"\x1foutcome:{normalized_criterion_outcome_key(criterion.outcome)}")diff --git a/skyvern/forge/sdk/copilot/completion_output_grounding.py b/skyvern/forge/sdk/copilot/completion_output_grounding.pyindex 8f6461e25..030a60c3a 100644--- a/skyvern/forge/sdk/copilot/completion_output_grounding.py+++ b/skyvern/forge/sdk/copilot/completion_output_grounding.py@@ -40,6 +40,7 @@ def split_requested_output_criteria(criterion.output_pathand criterion.level != "definition"and not criterion.method_mandated+ and criterion.kind != "validation_classification"and _normalize_output_path(criterion.output_path)):requested.append(criterion)diff --git a/skyvern/forge/sdk/copilot/completion_verification.py b/skyvern/forge/sdk/copilot/completion_verification.pyindex b35862840..8569879f9 100644--- a/skyvern/forge/sdk/copilot/completion_verification.py+++ b/skyvern/forge/sdk/copilot/completion_verification.py@@ -282,6 +282,7 @@ class RunEvidenceSnapshot:_UNAVAILABLE = CompletionVerificationResult(status="unavailable")_MISSING_VERDICT_EVIDENCE = "judge did not return a verdict for this criterion"+_INCOMPLETE_VALIDATION_CLASSIFICATION_CONTRACT = "incomplete typed classification contract"_MISSING_REGISTERED_DOWNLOAD_EVIDENCE = "run output did not include a non-empty registered browser download"@@ -395,8 +396,21 @@ def _render_criteria(criteria: list[CompletionCriterion]) -> str:)deliverable_kind = f" [deliverable_kind={criterion.deliverable_kind}]" if criterion.deliverable_kind else ""output_path = f" [required_output_path={criterion.output_path}]" if criterion.output_path else ""+ classification_output_key = (+ f" [classification_output_key={criterion.classification_output_key}]"+ if criterion.classification_output_key+ else ""+ )+ expected_classification = (+ f" [expected_classification={criterion.expected_classification}]"+ if criterion.expected_classification is not None+ else ""+ )parts.append(- f"- {criterion.id}: {criterion.outcome}{contingent_on}{antecedent_output_path}{deliverable_kind}{output_path}{flags}"+ f"- {criterion.id}: {criterion.outcome}"+ f" [kind={criterion.kind}]"+ f"{contingent_on}{antecedent_output_path}{deliverable_kind}{output_path}"+ f"{classification_output_key}{expected_classification}{flags}")return "\n".join(parts)@@ -837,6 +851,166 @@ def grade_present_value_criteria(return verdicts+def _classification_string(value: str) -> str:+ return " ".join(value.casefold().split())+++def _classification_value_matches(expected: str | bool, actual: Any) -> bool:+ if isinstance(expected, bool):+ return isinstance(actual, bool) and actual is expected+ if not isinstance(actual, str):+ return False+ normalized_expected = _classification_string(expected)+ normalized_actual = _classification_string(actual)+ return bool(normalized_expected) and normalized_actual == normalized_expected+++def _is_empty_classification_value(value: Any) -> bool:+ return value is None or value == "" or value == [] or value == {}+++def _classification_canonical_value(expected: str | bool, actual: Any) -> str | bool | None:+ if not _classification_value_matches(expected, actual):+ return None+ if isinstance(expected, bool):+ return actual+ return _classification_string(actual)+++def _classification_output_candidates(+ snapshot: RunEvidenceSnapshot,+ output_key: str,+) -> list[tuple[str, Any]]:+ candidates: list[tuple[str, Any]] = []+ for label, payload in snapshot.block_outputs.items():+ if isinstance(payload, dict) and output_key in payload:+ candidates.append((f"block_outputs:{label}.{output_key}", payload[output_key]))+ return sorted(candidates, key=lambda candidate: candidate[0])+++def _validation_classification_unsatisfied(+ criterion: CompletionCriterion,+ *,+ reason_code: Literal["no_evidence", "evidence_contradicts"],+ output_key: str,+ evidence_ref: str | None = None,+ missing_evidence: str | None = None,+) -> CriterionVerdict:+ return CriterionVerdict(+ criterion_id=criterion.id,+ state="unsatisfied",+ reason_code=reason_code,+ evidence_ref=evidence_ref,+ missing_evidence=missing_evidence,+ output_path=output_key,+ grounding_mode="exact_value",+ has_exact_value=True,+ )+++def grade_validation_classification_criteria(+ criteria: list[CompletionCriterion], snapshot: RunEvidenceSnapshot+) -> list[CriterionVerdict]:+ verdicts: list[CriterionVerdict] = []+ for criterion in criteria:+ if criterion.kind != "validation_classification":+ continue+ output_key = criterion.classification_output_key+ expected = criterion.expected_classification+ if output_key is None or expected is None:+ verdicts.append(+ _validation_classification_unsatisfied(+ criterion,+ reason_code="no_evidence",+ output_key=output_key or "",+ missing_evidence=_INCOMPLETE_VALIDATION_CLASSIFICATION_CONTRACT,+ )+ )+ continue+ candidates = _classification_output_candidates(snapshot, output_key)+ if not candidates:+ verdicts.append(+ _validation_classification_unsatisfied(+ criterion,+ reason_code="no_evidence",+ output_key=output_key,+ missing_evidence=f"missing classification output key {output_key}",+ )+ )+ continue+ if len(candidates) == 1:+ evidence_ref, actual = candidates[0]+ if _is_empty_classification_value(actual):+ verdicts.append(+ _validation_classification_unsatisfied(+ criterion,+ reason_code="no_evidence",+ output_key=output_key,+ evidence_ref=evidence_ref,+ missing_evidence=f"empty classification output key {output_key}",+ )+ )+ continue+ if not _classification_value_matches(expected, actual):+ verdicts.append(+ _validation_classification_unsatisfied(+ criterion,+ reason_code="evidence_contradicts",+ output_key=output_key,+ evidence_ref=evidence_ref,+ missing_evidence=f"classification output key {output_key} did not match expected value",+ )+ )+ continue+ verdicts.append(+ CriterionVerdict(+ criterion_id=criterion.id,+ state="satisfied",+ reason_code="evidence_confirms",+ evidence_ref=evidence_ref,+ output_path=output_key,+ grounding_mode="exact_value",+ has_exact_value=True,+ )+ )+ continue+ matching_canonical_values: set[str | bool] = set()+ contradiction_evidence_ref: str | None = None+ for evidence_ref, actual in candidates:+ if _is_empty_classification_value(actual):+ contradiction_evidence_ref = evidence_ref+ break+ canonical_value = _classification_canonical_value(expected, actual)+ if canonical_value is None:+ contradiction_evidence_ref = evidence_ref+ break+ matching_canonical_values.add(canonical_value)+ if contradiction_evidence_ref is not None or len(matching_canonical_values) != 1:+ verdicts.append(+ _validation_classification_unsatisfied(+ criterion,+ reason_code="evidence_contradicts",+ output_key=output_key,+ evidence_ref=contradiction_evidence_ref,+ missing_evidence=f"classification output key {output_key} had contradictory evidence",+ )+ )+ continue+ evidence_ref = candidates[0][0]+ verdicts.append(+ CriterionVerdict(+ criterion_id=criterion.id,+ state="satisfied",+ reason_code="evidence_confirms",+ evidence_ref=evidence_ref,+ output_path=output_key,+ grounding_mode="exact_value",+ has_exact_value=True,+ )+ )+ return verdicts++def grade_record_semantic_consistency(criteria: list[CompletionCriterion], snapshot: RunEvidenceSnapshot) -> list[CriterionVerdict]:diff --git a/skyvern/forge/sdk/copilot/context.py b/skyvern/forge/sdk/copilot/context.pyindex e991fad6a..9f164121a 100644--- a/skyvern/forge/sdk/copilot/context.py+++ b/skyvern/forge/sdk/copilot/context.py@@ -597,6 +597,7 @@ class CopilotContext(AgentContext):# state untouched, because the browser session is now in post-failure# state and the prefix labels can no longer be trusted as an anchor.verified_block_outputs: dict[str, Any] = field(default_factory=dict)+ verified_terminal_block_outputs: dict[str, Any] = field(default_factory=dict)verified_prefix_labels: list[str] = field(default_factory=list)verified_prefix_current_url: str | None = Nonelast_requested_block_labels: list[str] = field(default_factory=list)diff --git a/skyvern/forge/sdk/copilot/request_policy.py b/skyvern/forge/sdk/copilot/request_policy.pyindex 03a4b30fc..01956dfc1 100644--- a/skyvern/forge/sdk/copilot/request_policy.py+++ b/skyvern/forge/sdk/copilot/request_policy.py@@ -4,7 +4,7 @@ import asyncioimport jsonimport reimport time-from dataclasses import dataclass, field+from dataclasses import dataclass, field, replacefrom typing import Any, Literal, cast, get_argsfrom urllib.parse import urlparse@@ -155,8 +155,17 @@ _MAX_TRACE_COMPLETION_CRITERIA = 8_COMPLETION_CRITERION_OUTCOME_MAX_CHARS = 200_COMPLETION_CRITERION_CONTINGENT_ON_MAX_CHARS = 200_COMPLETION_CRITERION_EXPECTED_VALUE_MAX_CHARS = 500+_COMPLETION_CRITERION_CLASSIFICATION_TARGET_MAX_CHARS = 120+_CLASSIFICATION_OUTPUT_KEY_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")_CONTINGENT_ANTECEDENT_OUTPUT_PATH_RE = re.compile(r"^output\.[A-Za-z_][A-Za-z0-9_]*$")_REQUESTED_OUTPUT_CRITERION_ID_PREFIX = "__copilot_requested_output__"+_VALIDATION_CLASSIFICATION_BOOLEAN_OUTPUT_TARGETS: dict[str, tuple[str, bool]] = {+ "output.login_only": ("login_only", True),+ "output.login_gated": ("login_gated", True),+}+_VALIDATION_CLASSIFICATION_LABEL_OUTPUT_TARGETS: dict[str, str] = {+ "output.path_classification": "path_classification",+}FALLBACK_FLOOR_CRITERION_ID_PREFIX = "__copilot_fallback_floor__"_FALLBACK_FLOOR_BASE_ID = f"{FALLBACK_FLOOR_CRITERION_ID_PREFIX}run"@@ -166,8 +175,9 @@ _FALLBACK_FLOOR_CREDENTIAL_OUTCOME = "The credentialed step authenticates and reCriterionLevel = Literal["definition", "run"]_CRITERION_LEVELS: frozenset[str] = frozenset({"definition", "run"})-CriterionKind = Literal["outcome", "terminal_action"]+CriterionKind = Literal["outcome", "terminal_action", "validation_classification"]TerminalActionFamily = Literal["request", "application", "form", "order"]+ClassificationTarget = str | boolExpectedOutputShape = Literal["reference_code","numeric_identifier",@@ -177,7 +187,7 @@ ExpectedOutputShape = Literal["money_amount","owner_label",]-_CRITERION_KINDS: frozenset[str] = frozenset({"outcome", "terminal_action"})+_CRITERION_KINDS: frozenset[str] = frozenset({"outcome", "terminal_action", "validation_classification"})_TERMINAL_ACTION_FAMILIES: frozenset[str] = frozenset({"request", "application", "form", "order"})_EXPECTED_OUTPUT_SHAPES: frozenset[str] = frozenset(get_args(ExpectedOutputShape))@@ -234,6 +244,8 @@ class CompletionCriterion:expected_output_shape: ExpectedOutputShape | None = Nonekind: CriterionKind = "outcome"terminal_action_family: TerminalActionFamily | None = None+ classification_output_key: str | None = None+ expected_classification: ClassificationTarget | None = None@dataclass@@ -329,6 +341,22 @@ class RequestPolicy:lines.append(f"completion_contract: {self.completion_contract}")if self.raw_secret_detected:lines.append(f"raw_secret_detected: {self.raw_secret_detected}")+ validation_classification_criteria = [+ criterion+ for criterion in self.graded_completion_criteria()+ if criterion.kind == "validation_classification"+ and criterion.classification_output_key+ and criterion.expected_classification is not None+ ]+ if validation_classification_criteria:+ lines.append("validation_classification_output_contracts:")+ for criterion in validation_classification_criteria:+ lines.append(+ f"- criterion_id: {criterion.id}; "+ f"return_key: {criterion.classification_output_key}; "+ f"expected_value: {json.dumps(criterion.expected_classification)}; "+ "return_location: top_level_block_output"+ )if self.resolved_credentials:lines += ["resolved_credentials:",@@ -560,6 +588,22 @@ def _coerce_expected_output_shape(value: Any) -> ExpectedOutputShape | None:return None+def _coerce_classification_output_key(value: Any) -> str | None:+ if not isinstance(value, str):+ return None+ key = value.strip()+ return key if _CLASSIFICATION_OUTPUT_KEY_RE.fullmatch(key) else None+++def _coerce_expected_classification(value: Any) -> ClassificationTarget | None:+ if isinstance(value, bool):+ return value+ if isinstance(value, str):+ collapsed = " ".join(value.split())[:_COMPLETION_CRITERION_CLASSIFICATION_TARGET_MAX_CHARS].strip()+ return collapsed or None+ return None++def _coerce_classifier_payload(raw: Any) -> dict[str, Any] | None:if isinstance(raw, str):raw = parse_final_response(raw)@@ -633,7 +677,7 @@ def _parse_completion_criteria(raw: Any) -> list[CompletionCriterion]:if not isinstance(raw, list):return []criteria: list[CompletionCriterion] = []- seen: set[tuple[str, str, str, str]] = set()+ seen: set[tuple[str, str, str, str, str, str]] = set()for item in raw:if not isinstance(item, dict):continue@@ -664,17 +708,33 @@ def _parse_completion_criteria(raw: Any) -> list[CompletionCriterion]:item.get("contingent_antecedent_output_path"))deliverable_kind = _normalize_deliverable_kind(item.get("deliverable_kind"))+ kind = _coerce_criterion_kind(item.get("kind"))+ classification_output_key = (+ _coerce_classification_output_key(item.get("classification_output_key"))+ if kind == "validation_classification"+ else None+ )+ expected_classification = (+ _coerce_expected_classification(item.get("expected_classification"))+ if kind == "validation_classification"+ else None+ )+ if kind == "validation_classification":+ output_path = None+ expected_output_value = None+ expected_output_shape = Nonekey = (contingent_on or "",contingent_antecedent_output_path or "",- output_path or normalized_criterion_outcome_key(outcome),+ output_path or classification_output_key or normalized_criterion_outcome_key(outcome),deliverable_kind or "",+ kind,+ str(expected_classification) if expected_classification is not None else "",)if key in seen:continueseen.add(key)level_raw = item.get("level")- kind = _coerce_criterion_kind(item.get("kind"))criteria.append(CompletionCriterion(id=f"c{len(criteria)}",@@ -692,6 +752,8 @@ def _parse_completion_criteria(raw: Any) -> list[CompletionCriterion]:expected_output_shape=expected_output_shape,kind=kind,terminal_action_family=_coerce_terminal_action_family(item.get("terminal_action_family"), kind),+ classification_output_key=classification_output_key,+ expected_classification=expected_classification,))if len(criteria) >= _MAX_COMPLETION_CRITERIA:@@ -1125,6 +1187,54 @@ def _apply_requested_output_completion_criteria()+def _validation_classification_target_for_legacy_criterion(+ criterion: CompletionCriterion,+) -> tuple[str, ClassificationTarget] | None:+ if criterion.kind != "outcome" or criterion.level == "definition" or criterion.method_mandated:+ return None+ if not criterion.output_path:+ return None+ boolean_target = _VALIDATION_CLASSIFICATION_BOOLEAN_OUTPUT_TARGETS.get(criterion.output_path)+ if boolean_target is not None:+ return boolean_target+ label_key = _VALIDATION_CLASSIFICATION_LABEL_OUTPUT_TARGETS.get(criterion.output_path)+ if label_key is None or criterion.expected_output_value is None:+ return None+ return (label_key, criterion.expected_output_value)+++def _apply_validation_classification_completion_criteria(policy: RequestPolicy) -> None:+ """Promote legacy classifier output carriers into typed validation-classification contracts."""++ criteria: list[CompletionCriterion] = []+ seen_classification_targets: set[tuple[str, str]] = set()+ for criterion in policy.completion_criteria:+ target = _validation_classification_target_for_legacy_criterion(criterion)+ if target is not None:+ output_key, expected = target+ criterion = replace(+ criterion,+ kind="validation_classification",+ output_path=None,+ expected_output_value=None,+ expected_output_shape=None,+ deliverable_kind=None,+ terminal_action_family=None,+ classification_output_key=output_key,+ expected_classification=expected,+ )+ if criterion.kind == "validation_classification":+ target_key = (+ criterion.classification_output_key or "",+ str(criterion.expected_classification) if criterion.expected_classification is not None else "",+ )+ if target_key in seen_classification_targets:+ continue+ seen_classification_targets.add(target_key)+ criteria.append(criterion)+ policy.completion_criteria = criteria++def _render_active_criteria_for_prompt(criteria: list[CompletionCriterion] | None) -> str:if not criteria:return ""@@ -1150,6 +1260,10 @@ def _render_active_criteria_for_prompt(criteria: list[CompletionCriterion] | Nonitem["expected_output_value"] = criterion.expected_output_valueif criterion.expected_output_shape:item["expected_output_shape"] = criterion.expected_output_shape+ if criterion.classification_output_key:+ item["classification_output_key"] = criterion.classification_output_key+ if criterion.expected_classification is not None:+ item["expected_classification"] = criterion.expected_classificationrendered.append(item)return json.dumps(rendered)@@ -1466,6 +1580,7 @@ async def _classify_request(policy.requires_user_clarification = Falsepolicy.raw_secret_evidence = None_apply_requested_output_completion_criteria(policy, user_message, requested_output_path_aliases)+ _apply_validation_classification_completion_criteria(policy)policy.completion_contract_status = ("present" if policy.completion_contract or policy.graded_completion_criteria() else "absent")diff --git a/skyvern/forge/sdk/copilot/tools/completion.py b/skyvern/forge/sdk/copilot/tools/completion.pyindex 878d54527..9a5de96d7 100644--- a/skyvern/forge/sdk/copilot/tools/completion.py+++ b/skyvern/forge/sdk/copilot/tools/completion.py@@ -24,6 +24,7 @@ from skyvern.forge.sdk.copilot.completion_verification import (grade_registered_download_criteria,grade_structured_record_criteria,grade_terminal_goal_record_criteria,+ grade_validation_classification_criteria,is_fallback_floor_base_criterion,is_registered_download_completion_criterion,registered_download_completion_criterion,@@ -309,16 +310,26 @@ async def _maybe_run_completion_verification_from_page_observation(if requested_output_criteriaelse [])+ validation_classification_verdicts = grade_validation_classification_criteria(judgeable_run_criteria, snapshot)+ validation_classification_ids = {verdict.criterion_id for verdict in validation_classification_verdicts}+ remaining_judgeable_run_criteria = [+ criterion for criterion in judgeable_run_criteria if criterion.id not in validation_classification_ids+ ]remaining = _copilot_seconds_remaining(copilot_ctx)if (remaining is not Noneand remaining<= settings.COPILOT_COMPLETION_JUDGE_TIMEOUT_SECONDS + _COMPLETION_VERIFICATION_BUDGET_MARGIN_SECONDS):- run_result = _merge_run_verdicts_if_requested_output_exists(- run_criteria,- requested_output_verdicts,- structural_unfired_criterion_ids=run_structural_unfired_ids,+ run_result = (+ _merge_run_verdicts(+ run_criteria,+ requested_output_verdicts,+ validation_classification_verdicts,+ structural_unfired_criterion_ids=run_structural_unfired_ids,+ )+ if requested_output_verdicts or validation_classification_verdicts+ else None)verification = (combine_verification_results(@@ -345,47 +356,66 @@ async def _maybe_run_completion_verification_from_page_observation(status="evaluated",criterion_ids=[criterion.id for criterion in run_criteria],verdicts=requested_output_verdicts+ + validation_classification_verdicts+ [CriterionVerdict(criterion_id=criterion.id, state="unsatisfied", reason_code="no_evidence")- for criterion in judgeable_run_criteria+ for criterion in remaining_judgeable_run_criteria],contingent_criterion_ids=run_contingent_ids,contingent_on_by_criterion_id=run_contingent_on_by_id,contingent_antecedent_output_path_by_criterion_id=run_contingent_path_by_id,structural_unfired_criterion_ids=run_structural_unfired_ids,)- elif not judgeable_run_criteria:+ elif not remaining_judgeable_run_criteria:run_result = _merge_run_verdicts(run_criteria,requested_output_verdicts,+ validation_classification_verdicts,structural_unfired_criterion_ids=run_structural_unfired_ids,)else:handler = await _completion_verification_handler(copilot_ctx)if handler is None:- run_result = _merge_run_verdicts_if_requested_output_exists(- run_criteria,- requested_output_verdicts,- structural_unfired_criterion_ids=run_structural_unfired_ids,+ run_result = (+ _merge_run_verdicts(+ run_criteria,+ requested_output_verdicts,+ validation_classification_verdicts,+ structural_unfired_criterion_ids=run_structural_unfired_ids,+ )+ if requested_output_verdicts or validation_classification_verdicts+ else None)if run_result is None:return Noneelse:- judgeable_result = await evaluate_completion_criteria(judgeable_run_criteria, snapshot, handler)- judgeable_result = _apply_present_value_upgrades(judgeable_result, judgeable_run_criteria, snapshot)+ judgeable_result = await evaluate_completion_criteria(+ remaining_judgeable_run_criteria,+ snapshot,+ handler,+ )+ judgeable_result = _apply_present_value_upgrades(+ judgeable_result,+ remaining_judgeable_run_criteria,+ snapshot,+ )if judgeable_result.status != "evaluated":- run_result = (- _merge_run_verdicts_if_requested_output_exists(+ deterministic_result = (+ _merge_run_verdicts(run_criteria,requested_output_verdicts,+ validation_classification_verdicts,structural_unfired_criterion_ids=run_structural_unfired_ids,)- or judgeable_result+ if requested_output_verdicts or validation_classification_verdicts+ else None)+ run_result = deterministic_result or judgeable_resultelse:run_result = _merge_run_verdicts(run_criteria,requested_output_verdicts,+ validation_classification_verdicts,judgeable_result.verdicts,structural_unfired_criterion_ids=run_structural_unfired_ids,)@@ -668,6 +698,9 @@ def _apply_present_value_upgrades(upgrades.update({verdict.criterion_id: verdict for verdict in grade_structured_record_criteria(run_criteria, snapshot)})+ upgrades.update(+ {verdict.criterion_id: verdict for verdict in grade_validation_classification_criteria(run_criteria, snapshot)}+ )if include_terminal_goal_records:upgrades.update({@@ -831,6 +864,8 @@ def _deterministic_run_verification_result(deterministic_by_id[verdict.criterion_id] = verdictfor verdict in grade_registered_download_criteria(run_criteria, snapshot):deterministic_by_id[verdict.criterion_id] = verdict+ for verdict in grade_validation_classification_criteria(run_criteria, snapshot):+ deterministic_by_id[verdict.criterion_id] = verdictfor verdict in semantic_verdicts:deterministic_by_id[verdict.criterion_id] = verdictdiff --git a/skyvern/forge/sdk/copilot/tools/run_execution.py b/skyvern/forge/sdk/copilot/tools/run_execution.pyindex 944ada7da..d9e85a2db 100644--- a/skyvern/forge/sdk/copilot/tools/run_execution.py+++ b/skyvern/forge/sdk/copilot/tools/run_execution.py@@ -1639,6 +1639,7 @@ async def _run_blocks_and_collect_debug(if run_ok and all(r.get("status") == "completed" for r in results):for label, output in block_outputs_by_label.items():ctx.verified_block_outputs[label] = output+ ctx.verified_terminal_block_outputs = dict(block_outputs_by_label)existing_prefix = list(getattr(ctx, "verified_prefix_labels", []) or [])existing_set = set(existing_prefix)for label in labels_to_execute:@@ -2021,6 +2022,19 @@ def _record_run_blocks_result(run_ok = bool(result.get("ok", False))data = result.get("data")run_id = data.get("workflow_run_id") if isinstance(data, dict) else None+ if run_ok and isinstance(data, dict):+ terminal_outputs: dict[str, Any] = {}+ for block in data.get("blocks") or []:+ if not isinstance(block, dict):+ continue+ label = block.get("label")+ output = block.get("extracted_data")+ if isinstance(label, str) and isinstance(output, dict) and output:+ terminal_outputs[label] = output+ if terminal_outputs:+ # Prefer the final run-result extracted_data for terminal replies;+ # it is the same persisted run evidence completion verification saw.+ copilot_ctx.verified_terminal_block_outputs = terminal_outputscopilot_ctx.completion_verification_result = completion_verificationrecord_completion_verification(copilot_ctx, completion_verification)_record_adjudication_on_turn_state(copilot_ctx, completion_verification)diff --git a/tests/unit/test_copilot_agent_blocker_renderer.py b/tests/unit/test_copilot_agent_blocker_renderer.pyindex f27a12ac1..c8ffb3b41 100644--- a/tests/unit/test_copilot_agent_blocker_renderer.py+++ b/tests/unit/test_copilot_agent_blocker_renderer.py@@ -10,10 +10,12 @@ from skyvern.forge.sdk.copilot.agent import (_FALLBACK_BLOCKER_REPLY,_RAW_SECRET_LEAK_REFUSAL,_VERIFIED_WORKFLOW_SUCCESS_REPLY,+ _build_goal_satisfied_exit_result,_build_output_policy_blocked_result,_build_turn_halt_exit_result,_finalize_result_with_blocker_override,_render_blocker_reply,+ _verified_workflow_success_reply,)from skyvern.forge.sdk.copilot.blocker_signal import (_LEAK_DENY_TOKENS,@@ -25,7 +27,7 @@ from skyvern.forge.sdk.copilot.blocker_signal import (from skyvern.forge.sdk.copilot.completion_verification import CompletionVerificationResult, CriterionVerdictfrom skyvern.forge.sdk.copilot.context import AgentResult, CopilotContextfrom skyvern.forge.sdk.copilot.output_policy import CopilotOutputKind, OutputPolicyReason, OutputPolicyVerdict-from skyvern.forge.sdk.copilot.request_policy import RequestPolicy+from skyvern.forge.sdk.copilot.request_policy import CompletionCriterion, RequestPolicyfrom skyvern.forge.sdk.copilot.run_outcome import TERMINAL_CHALLENGE_BLOCKER_REASON_CODE, RecordedRunOutcomefrom skyvern.forge.sdk.copilot.turn_halt import TurnHalt, TurnHaltKind@@ -593,6 +595,37 @@ def _fully_satisfied_result() -> CompletionVerificationResult:)+def _fully_satisfied_classification_result() -> CompletionVerificationResult:+ return CompletionVerificationResult(+ status="evaluated",+ criterion_ids=["c_validation"],+ verdicts=[+ CriterionVerdict(+ criterion_id="c_validation",+ state="satisfied",+ reason_code="evidence_confirms",+ output_path="login_only",+ grounding_mode="exact_value",+ has_exact_value=True,+ )+ ],+ )+++def _classification_policy() -> RequestPolicy:+ return RequestPolicy(+ completion_criteria=[+ CompletionCriterion(+ id="c_validation",+ outcome="The run classifies the path as login gated.",+ kind="validation_classification",+ classification_output_key="login_only",+ expected_classification=True,+ )+ ]+ )++def _seed_verified_outcome(ctx: CopilotContext) -> None:ctx.completion_verification_result = _fully_satisfied_result()ctx.last_artifact_health_blocker_reason = None@@ -640,6 +673,141 @@ def test_verified_outcome_preserve_sets_verified_success_payload() -> None:assert overridden.narrative_payload["proposalDisposition"] == "review_tested"+async def test_verified_classification_success_surfaces_terminal_verdict_and_reload_payload() -> None:+ ctx = _ctx()+ ctx.stream = None+ ctx.turn_id = "turn-classified"+ ctx.last_test_ok = True+ ctx.last_full_workflow_test_ok = True+ _seed_verified_outcome(ctx)+ ctx.request_policy = _classification_policy()+ ctx.completion_verification_result = _fully_satisfied_classification_result()+ ctx.verified_block_outputs = {+ "inspect_access_path": {+ "login_only": True,+ "visible_page_path_label": "Start service sign-in gate",+ "safest_reachable_next_step": "Stop before account-specific setup.",+ "recommended_next_action": "Ask the user for online account access before continuing.",+ "evidence_text": "The page says Sign in or register to continue.",+ }+ }++ result = await _build_goal_satisfied_exit_result(ctx, global_llm_context=None)++ assert "login-gated" in result.user_response+ assert "login_only=true" in result.user_response+ assert "visible_page_path_label=Start service sign-in gate" in result.user_response+ assert "safest_reachable_next_step=Stop before account-specific setup." in result.user_response+ assert "recommended_next_action=Ask the user for online account access before continuing." in result.user_response+ assert "observed_gate_phrase=Sign in or register to continue" in result.user_response+ assert result.narrative_payload is not None+ assert result.narrative_payload["terminalMessage"] == result.user_response+++def test_verified_classification_summary_does_not_display_unobserved_expected_value() -> None:+ ctx = _ctx()+ _seed_verified_outcome(ctx)+ ctx.request_policy = _classification_policy()+ ctx.completion_verification_result = _fully_satisfied_classification_result()+ ctx.verified_block_outputs = {}++ response = _verified_workflow_success_reply(ctx)++ assert response == "I created and tested the workflow successfully."+ assert "login_only=true" not in response+ assert "login-gated" not in response+ assert "observed_gate_phrase=Sign in or register to continue" not in response+++def test_verified_classification_summary_uses_terminal_snapshot_when_frontier_outputs_clear() -> None:+ ctx = _ctx()+ _seed_verified_outcome(ctx)+ ctx.request_policy = _classification_policy()+ ctx.completion_verification_result = _fully_satisfied_classification_result()+ ctx.verified_block_outputs = {}+ ctx.verified_terminal_block_outputs = {+ "inspect_access_path": {+ "login_only": True,+ "visible_page_path_label": "Start, Stop or Move Service",+ "matched_gate_phrases": ["sign in or register to continue"],+ }+ }++ response = _verified_workflow_success_reply(ctx)++ assert "login_only=true" in response+ assert "login-gated" in response+ assert "visible_page_path_label=Start, Stop or Move Service" in response+ assert "observed_gate_phrase=Sign in or register to continue" in response+++def test_verified_classification_summary_labels_login_gate_from_verified_gate_phrase() -> None:+ ctx = _ctx()+ _seed_verified_outcome(ctx)+ ctx.request_policy = RequestPolicy(completion_criteria=[CompletionCriterion(id="c0", outcome="Reached gate")])+ ctx.completion_verification_result = _fully_satisfied_result()+ ctx.verified_terminal_block_outputs = {+ "inspect_access_path": {+ "visible_page_path_label": "Start, Stop or Move Service > Start Service",+ "observed_gate_phrase": "Sign in or register to continue",+ }+ }++ response = _verified_workflow_success_reply(ctx)++ assert "visible_page_path_label=Start, Stop or Move Service > Start Service" in response+ assert "observed_gate_phrase=Sign in or register to continue" in response+ assert "login-gated" in response+++def test_verified_classification_summary_requires_completion_verification() -> None:+ ctx = _ctx()+ ctx.request_policy = _classification_policy()+ ctx.verified_block_outputs = {+ "inspect_access_path": {+ "login_only": True,+ "visible_page_path_label": "Start service sign-in gate",+ "evidence_text": "The page prose says login-gated and login_only=true.",+ }+ }++ response = _verified_workflow_success_reply(ctx)++ assert response == _VERIFIED_WORKFLOW_SUCCESS_REPLY+ assert "login-gated" not in response+ assert "login_only=true" not in response+++def test_verified_classification_summary_requires_fully_satisfied_verification() -> None:+ ctx = _ctx()+ ctx.request_policy = _classification_policy()+ ctx.completion_verification_result = CompletionVerificationResult(+ status="evaluated",+ criterion_ids=["c_validation"],+ verdicts=[+ CriterionVerdict(+ criterion_id="c_validation",+ state="unsatisfied",+ reason_code="evidence_contradicts",+ output_path="login_only",+ )+ ],+ )+ ctx.verified_block_outputs = {+ "inspect_access_path": {+ "login_only": True,+ "visible_page_path_label": "Start service sign-in gate",+ }+ }++ response = _verified_workflow_success_reply(ctx)++ assert response == _VERIFIED_WORKFLOW_SUCCESS_REPLY+ assert "login-gated" not in response+ assert "login_only=true" not in response++def test_unverified_blocker_still_renders_blocker_text() -> None:ctx = _ctx()ctx.completion_verification_result = Nonediff --git a/tests/unit/test_copilot_agent_helpers.py b/tests/unit/test_copilot_agent_helpers.pyindex 4e88f47e8..ca33b45f2 100644--- a/tests/unit/test_copilot_agent_helpers.py+++ b/tests/unit/test_copilot_agent_helpers.py@@ -3001,6 +3001,44 @@ class TestNativeToolSurface:class TestRequestPolicyCredentialResolution:+ @pytest.mark.asyncio+ async def test_request_policy_classifier_uses_configured_default_timeout_budget(self, monkeypatch) -> None:+ from skyvern.config import Settings, settings+ from skyvern.forge.sdk.copilot import request_policy as rp++ default_timeout = Settings.model_fields["COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDS"].default+ assert default_timeout == 12.0++ observed_timeouts: list[float | None] = []+ real_wait_for = rp.asyncio.wait_for++ async def recording_wait_for(awaitable, timeout=None):+ observed_timeouts.append(timeout)+ return await real_wait_for(awaitable, timeout=timeout)++ monkeypatch.setattr(settings, "COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDS", default_timeout)+ monkeypatch.setattr(rp.time, "monotonic", lambda: 1000.0)+ monkeypatch.setattr(rp.asyncio, "wait_for", recording_wait_for)++ async def handler(*, prompt: str, prompt_name: str) -> dict[str, object]:+ return {+ "credential_input_kind": "none",+ "completion_contract": "complete when the account page is visible",+ }++ policy = await _classify_request(+ user_message=(+ "Build a workflow for https://example.com/account. complete when the account page is visible"+ ),+ workflow_yaml="",+ chat_history=[],+ global_llm_context="",+ handler=handler,+ )++ assert policy.classifier_status == "success"+ assert observed_timeouts == [default_timeout]+@pytest.mark.asyncioasync def test_request_policy_classifier_timeout_does_not_retry(self, monkeypatch) -> None:from skyvern.config import settings@@ -4918,6 +4956,24 @@ class TestResponseTypeClassificationRuleReachesAgent:assert prompt.index("RESPONSE-TYPE CLASSIFICATION") < prompt.index("**Option 1: Reply to the user**")+class TestValidationClassificationOutputContractRule:+ def test_build_system_prompt_requires_exact_top_level_classification_contract_keys(self) -> None:+ from skyvern.forge.sdk.copilot.agent import _build_system_prompt++ prompt = _build_system_prompt(tool_usage_guide="", security_rules="")++ assert "validation_classification_output_contracts" in prompt+ assert "explicitly `return` a dict" in prompt+ assert "Do not rely on schema defaults or exposed top-level locals" in prompt+ assert "exact `return_key` as a top-level key" in prompt+ assert "goal_value_paths" in prompt+ assert "extraction_schema" in prompt+ assert "reject code-only output blocks without that metadata" in prompt+ assert "observed_gate_phrase" in prompt+ assert "Synonyms such as `path_login_only`, `classification`, or prose in `evidence_text`" in prompt+ assert "do not replace the required key" in prompt++class TestCopilotConfig:def test_system_prompt_uses_custom_security_rules(self) -> None:prompt = agent_module._build_system_prompt(diff --git a/tests/unit/test_copilot_completion_verification.py b/tests/unit/test_copilot_completion_verification.pyindex 79188a459..748448323 100644--- a/tests/unit/test_copilot_completion_verification.py+++ b/tests/unit/test_copilot_completion_verification.py@@ -16,7 +16,10 @@ from skyvern.forge.sdk.copilot.agent import (_verified_workflow_or_none,)from skyvern.forge.sdk.copilot.completion_criteria_store import criteria_from_json, criteria_to_json-from skyvern.forge.sdk.copilot.completion_output_grounding import grade_requested_output_criteria+from skyvern.forge.sdk.copilot.completion_output_grounding import (+ grade_requested_output_criteria,+ split_requested_output_criteria,+)from skyvern.forge.sdk.copilot.completion_verification import (REGISTERED_DOWNLOAD_COMPLETION_CRITERION_ID,CompletionVerificationResult,@@ -33,6 +36,7 @@ from skyvern.forge.sdk.copilot.completion_verification import (grade_registered_download_criteria,grade_structured_record_criteria,grade_terminal_goal_record_criteria,+ grade_validation_classification_criteria,registered_download_completion_criterion,run_plane_all_no_evidence,structural_unfired_contingent_criterion_ids,@@ -101,6 +105,8 @@ def _criterion(deliverable_kind: str | None = None,expected_output_value: str | None = None,expected_output_shape: str | None = None,+ classification_output_key: str | None = None,+ expected_classification: str | bool | None = None,) -> CompletionCriterion:return CompletionCriterion(id=cid,@@ -114,6 +120,8 @@ def _criterion(deliverable_kind=deliverable_kind, # type: ignore[arg-type]expected_output_value=expected_output_value,expected_output_shape=expected_output_shape, # type: ignore[arg-type]+ classification_output_key=classification_output_key,+ expected_classification=expected_classification,)@@ -1289,6 +1297,258 @@ def test_terminal_goal_record_does_not_take_literal_criteria_from_present_value(assert _satisfied_criterion_ids(grade_present_value_criteria(criteria, snapshot)) == {"c0"}+def _validation_classification_criterion(+ expected_classification: str | bool = "login_gated",+ *,+ key: str = "path_classification",+) -> CompletionCriterion:+ return _criterion(+ "c_validation",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ classification_output_key=key,+ expected_classification=expected_classification,+ )+++def test_validation_classification_grader_credits_matching_string_value() -> None:+ criterion = _validation_classification_criterion("login_gated")+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": {"path_classification": "login_gated"}})++ assert grade_validation_classification_criteria([criterion], snapshot) == [+ CriterionVerdict(+ criterion_id="c_validation",+ state="satisfied",+ reason_code="evidence_confirms",+ evidence_ref="block_outputs:classify_path.path_classification",+ output_path="path_classification",+ grounding_mode="exact_value",+ has_exact_value=True,+ )+ ]+++def test_validation_classification_grader_credits_matching_boolean_value() -> None:+ criterion = _validation_classification_criterion(True, key="login_gated")+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": {"login_gated": True}})++ assert _satisfied_criterion_ids(grade_validation_classification_criteria([criterion], snapshot)) == {"c_validation"}+++def test_validation_classification_grader_credits_repeated_matching_boolean_values() -> None:+ criterion = _validation_classification_criterion(True, key="login_only")+ snapshot = RunEvidenceSnapshot(+ block_outputs={+ "classify_path": {"login_only": True},+ "classify_path_output": {"login_only": True},+ }+ )++ assert grade_validation_classification_criteria([criterion], snapshot) == [+ CriterionVerdict(+ criterion_id="c_validation",+ state="satisfied",+ reason_code="evidence_confirms",+ evidence_ref="block_outputs:classify_path.login_only",+ output_path="login_only",+ grounding_mode="exact_value",+ has_exact_value=True,+ )+ ]+++ "criterion",+ [+ _criterion(+ "c_validation",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ expected_classification="login_gated",+ ),+ _criterion(+ "c_validation",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ classification_output_key="path_classification",+ ),+ ],+)+def test_validation_classification_grader_fails_closed_for_incomplete_contract(+ criterion: CompletionCriterion,+) -> None:+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": {"path_classification": "login_gated"}})++ verdicts = grade_validation_classification_criteria([criterion], snapshot)++ assert len(verdicts) == 1+ assert verdicts[0].criterion_id == "c_validation"+ assert verdicts[0].state == "unsatisfied"+ assert verdicts[0].reason_code == "no_evidence"+ assert verdicts[0].missing_evidence == "incomplete typed classification contract"+++ ("payload", "reason_code"),+ [+ ({}, "no_evidence"),+ ({"path_classification": None}, "no_evidence"),+ ({"path_classification": ""}, "no_evidence"),+ ({"path_classification": []}, "no_evidence"),+ ({"path_classification": {}}, "no_evidence"),+ ({"path_classification": "public"}, "evidence_contradicts"),+ ({"path_classification": ["login_gated"]}, "evidence_contradicts"),+ ({"path_classification": True}, "evidence_contradicts"),+ ({"success": True}, "no_evidence"),+ ({"evidence_text": "The path is login_gated."}, "no_evidence"),+ ],+)+def test_validation_classification_grader_fails_closed_for_non_matching_values(+ payload: Any,+ reason_code: str,+) -> None:+ criterion = _validation_classification_criterion("login_gated")+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": payload})++ verdicts = grade_validation_classification_criteria([criterion], snapshot)++ assert len(verdicts) == 1+ assert verdicts[0].criterion_id == "c_validation"+ assert verdicts[0].state == "unsatisfied"+ assert verdicts[0].reason_code == reason_code+++def test_validation_classification_grader_credits_repeated_matching_string_values() -> None:+ criterion = _validation_classification_criterion("login_gated")+ snapshot = RunEvidenceSnapshot(+ block_outputs={+ "classify_path_a": {"path_classification": "login_gated"},+ "classify_path_b": {"path_classification": "login_gated"},+ }+ )++ verdicts = grade_validation_classification_criteria([criterion], snapshot)++ assert len(verdicts) == 1+ assert verdicts[0] == CriterionVerdict(+ criterion_id="c_validation",+ state="satisfied",+ reason_code="evidence_confirms",+ evidence_ref="block_outputs:classify_path_a.path_classification",+ output_path="path_classification",+ grounding_mode="exact_value",+ has_exact_value=True,+ )+++def test_validation_classification_grader_fails_closed_for_conflicting_candidates() -> None:+ criterion = _validation_classification_criterion("login_gated")+ snapshot = RunEvidenceSnapshot(+ block_outputs={+ "classify_path_a": {"path_classification": "login_gated"},+ "classify_path_b": {"path_classification": "public"},+ }+ )++ verdicts = grade_validation_classification_criteria([criterion], snapshot)++ assert len(verdicts) == 1+ assert verdicts[0].state == "unsatisfied"+ assert verdicts[0].reason_code == "evidence_contradicts"+++ "mixed_value",+ [+ None,+ "",+ [],+ {},+ "true",+ False,+ ],+)+def test_validation_classification_grader_fails_closed_for_mixed_repeated_boolean_candidates(+ mixed_value: Any,+) -> None:+ criterion = _validation_classification_criterion(True, key="login_only")+ snapshot = RunEvidenceSnapshot(+ block_outputs={+ "classify_path": {"login_only": True},+ "classify_path_output": {"login_only": mixed_value},+ }+ )++ verdicts = grade_validation_classification_criteria([criterion], snapshot)++ assert len(verdicts) == 1+ assert verdicts[0].state == "unsatisfied"+ assert verdicts[0].reason_code == "evidence_contradicts"+++def test_validation_classification_grader_rejects_same_named_scalar_block_label() -> None:+ criterion = _validation_classification_criterion("login_gated")+ snapshot = RunEvidenceSnapshot(block_outputs={"path_classification": "login_gated"})++ verdicts = grade_validation_classification_criteria([criterion], snapshot)++ assert len(verdicts) == 1+ assert verdicts[0].state == "unsatisfied"+ assert verdicts[0].reason_code == "no_evidence"+++def test_validation_classification_is_not_requested_output_criterion() -> None:+ criterion = _criterion(+ "c_validation",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ output_path="output.path_classification",+ expected_output_value="login_gated",+ )++ requested, remaining = split_requested_output_criteria([criterion])++ assert requested == []+ assert remaining == [criterion]+++def test_validation_classification_grader_does_not_cross_credit_outcome_or_terminal_action() -> None:+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": {"path_classification": "login_gated"}})++ assert (+ grade_validation_classification_criteria(+ [+ _criterion(+ "c_outcome",+ "The run classifies whether the path is login gated.",+ classification_output_key="path_classification",+ expected_classification="login_gated",+ ),+ _criterion(+ "c_terminal",+ "The run classifies whether the path is login gated.",+ kind="terminal_action",+ terminal_action_family="request",+ classification_output_key="path_classification",+ expected_classification="login_gated",+ ),+ ],+ snapshot,+ )+ == []+ )+++def test_validation_classification_grader_repeats_same_verdict_for_same_output() -> None:+ criterion = _validation_classification_criterion("login_gated")+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": {"path_classification": "login_gated"}})++ first = grade_validation_classification_criteria([criterion], snapshot)+ second = grade_validation_classification_criteria([criterion], snapshot)++ assert first == second++def test_fallback_floor_accepts_validation_review_evidence() -> None:snapshot = RunEvidenceSnapshot(block_outputs={"submit_request": _validation_review_payload()})@@ -2054,6 +2314,30 @@ def _validation_review_output_result(**payload_overrides: Any) -> dict:}+def _validation_classification_output_result(value: Any) -> dict:+ return _validation_classification_payload_result({"path_classification": value})+++def _validation_classification_payload_result(payload: dict[str, Any]) -> dict:+ return {+ "ok": True,+ "data": {+ "workflow_run_id": "wr_validation_classification",+ "overall_status": "completed",+ "executed_block_labels": ["classify_path"],+ "current_url": "https://example.test/login",+ "blocks": [+ {+ "label": "classify_path",+ "block_type": "CODE",+ "status": "completed",+ "extracted_data": payload,+ }+ ],+ },+ }++def _live_validation_review_output_result(**payload_overrides: Any) -> dict:return {"ok": True,@@ -2347,6 +2631,249 @@ async def test_non_fallback_judge_confirmed_run_still_fires_barrier(monkeypatch:assert verified_goal_satisfied_context(ctx) is True+async def test_validation_classification_block_output_satisfies_without_judge_unknown_veto(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ async def handler(**_: object) -> dict:+ raise AssertionError("typed validation classification should be graded deterministically")++ _patch_completion_handler(monkeypatch, handler)+ ctx = _ctx_with_blocks("code")+ _set_workflow_labels(ctx, "classify_path")+ ctx.request_policy = RequestPolicy(completion_criteria=[_validation_classification_criterion("login_gated")])++ first = await _maybe_run_completion_verification(+ ctx,+ _validation_classification_output_result("login_gated"),+ time.monotonic(),+ )+ second = await _maybe_run_completion_verification(+ ctx,+ _validation_classification_output_result("login_gated"),+ time.monotonic(),+ )++ assert first is not None+ assert second is not None+ assert first.is_fully_satisfied() is True+ assert second.is_fully_satisfied() is True+ assert first.verdicts == second.verdicts+++async def test_validation_classification_duplicate_registered_output_surfaces_satisfy_without_judge(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ async def handler(**_: object) -> dict:+ raise AssertionError("repeated coherent typed classification evidence should not require judge authority")++ _patch_completion_handler(monkeypatch, handler)+ ctx = _ctx_with_blocks("code")+ _set_workflow_labels(ctx, "classify_path")+ ctx.request_policy = RequestPolicy(+ completion_criteria=[_validation_classification_criterion(True, key="login_only")]+ )++ verification = await _maybe_run_completion_verification(+ ctx,+ {+ "ok": True,+ "data": {+ "workflow_run_id": "wr_validation_classification",+ "overall_status": "completed",+ "executed_block_labels": ["classify_path"],+ "current_url": "https://example.test/login",+ "blocks": [+ {+ "label": "classify_path",+ "block_type": "CODE",+ "status": "completed",+ "extracted_data": {"login_only": True},+ }+ ],+ "registered_output_parameter_values": [+ {+ "workflow_run_id": "wr_validation_classification",+ "output_parameter_id": "op_login_only",+ "output_parameter_key": "classify_path_output",+ "block_label": "classify_path",+ "block_type": "CODE",+ "value": {"login_only": True},+ }+ ],+ },+ },+ time.monotonic(),+ )++ assert verification is not None+ assert verification.is_fully_satisfied() is True+ assert verification.verdicts == [+ CriterionVerdict(+ criterion_id="c_validation",+ state="satisfied",+ reason_code="evidence_confirms",+ evidence_ref="block_outputs:classify_path.login_only",+ output_path="login_only",+ grounding_mode="exact_value",+ has_exact_value=True,+ )+ ]+++async def test_validation_classification_is_not_satisfied_by_fallback_floor_evidence(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ async def handler(**_: object) -> dict:+ return {+ "verdicts": [+ {+ "criterion_id": "c_validation",+ "satisfied": False,+ "reason_code": "no_evidence",+ "missing_evidence": "matching classification output at path_classification",+ }+ ]+ }++ _patch_completion_handler(monkeypatch, handler)+ ctx = _ctx_with_blocks("code")+ _set_workflow_labels(ctx, "submit_request")+ ctx.request_policy = RequestPolicy(+ completion_criteria=[+ _validation_classification_criterion("login_gated"),+ *build_classifier_fallback_floor([]),+ ]+ )++ verification = await _maybe_run_completion_verification(+ ctx,+ _validation_review_output_result(),+ time.monotonic(),+ )++ assert verification is not None+ assert verification.is_fully_satisfied() is False+ assert "c_validation" in {verdict.criterion_id for verdict in verification.verdicts if not verdict.satisfied}+++ "payload",+ [+ {},+ {"evidence_text": "The path is login_gated."},+ {"success": True},+ ],+)+async def test_validation_classification_missing_or_prose_only_evidence_cannot_be_judge_approved(+ monkeypatch: pytest.MonkeyPatch,+ payload: dict[str, Any],+) -> None:+ calls = 0++ async def handler(**_: object) -> dict:+ nonlocal calls+ calls += 1+ return {"verdicts": [{"criterion_id": "c_validation", "satisfied": True, "reason_code": "evidence_confirms"}]}++ _patch_completion_handler(monkeypatch, handler)+ ctx = _ctx_with_blocks("code")+ _set_workflow_labels(ctx, "classify_path")+ ctx.request_policy = RequestPolicy(completion_criteria=[_validation_classification_criterion("login_gated")])++ verification = await _maybe_run_completion_verification(+ ctx,+ _validation_classification_payload_result(payload),+ time.monotonic(),+ )++ assert calls == 0+ assert verification is not None+ assert verification.is_fully_satisfied() is False+ assert verification.verdicts == [+ CriterionVerdict(+ criterion_id="c_validation",+ state="unsatisfied",+ reason_code="no_evidence",+ output_path="path_classification",+ grounding_mode="exact_value",+ has_exact_value=True,+ missing_evidence="missing classification output key path_classification",+ )+ ]+++ "criterion",+ [+ _criterion(+ "c_validation",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ expected_classification="login_gated",+ ),+ _criterion(+ "c_validation",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ classification_output_key="path_classification",+ ),+ _criterion(+ "c_validation",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ output_path="output.path_classification",+ expected_output_value="login_gated",+ ),+ ],+)+async def test_validation_classification_incomplete_contract_cannot_be_judge_approved(+ monkeypatch: pytest.MonkeyPatch,+ criterion: CompletionCriterion,+) -> None:+ calls = 0++ async def handler(**_: object) -> dict:+ nonlocal calls+ calls += 1+ return {+ "verdicts": [+ {"criterion_id": "c_validation", "satisfied": True, "reason_code": "evidence_confirms"},+ {"criterion_id": "c_other", "satisfied": True, "reason_code": "evidence_confirms"},+ ]+ }++ _patch_completion_handler(monkeypatch, handler)+ ctx = _ctx_with_blocks("code")+ _set_workflow_labels(ctx, "classify_path")+ ctx.code_artifact_metadata = _metadata_for_requested_paths("path_classification")+ ctx.request_policy = RequestPolicy(+ completion_criteria=[+ criterion,+ _criterion("c_other", "The run produced ordinary page evidence."),+ ]+ )++ verification = await _maybe_run_completion_verification(+ ctx,+ _validation_classification_output_result("login_gated"),+ time.monotonic(),+ )++ assert calls == 1+ assert verification is not None+ assert verification.is_fully_satisfied() is False+ verdict_by_id = {verdict.criterion_id: verdict for verdict in verification.verdicts}+ assert verdict_by_id["c_validation"].state == "unsatisfied"+ assert verdict_by_id["c_validation"].reason_code == "no_evidence"+ assert verdict_by_id["c_validation"].missing_evidence == "incomplete typed classification contract"+ assert verdict_by_id["c_other"].satisfied is True++@pytest.mark.asyncioasync def test_classifier_fallback_record_contradiction_still_surfaces(monkeypatch: pytest.MonkeyPatch) -> None:async def handler(**_: object) -> dict:@@ -2930,6 +3457,142 @@ async def test_page_observation_verification_recognizes_budgeted_outcome(assert "SKU-12345 is present in the cart" in seen_prompt["prompt"]+async def test_page_observation_validation_classification_cannot_be_judge_approved(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ handler_calls = 0++ async def handler(**_: object) -> dict:+ nonlocal handler_calls+ handler_calls += 1+ return {+ "verdicts": [+ {"criterion_id": "c_validation", "satisfied": True, "reason_code": "evidence_confirms"},+ {"criterion_id": "c_page", "satisfied": True, "reason_code": "evidence_confirms"},+ ]+ }++ _patch_completion_handler(monkeypatch, handler)+ ctx = _run_ctx()+ ctx.request_policy = RequestPolicy(+ completion_criteria=[+ _validation_classification_criterion("login_gated"),+ _criterion("c_page", "The page observation confirms the path was inspected."),+ ]+ )+ ctx.last_test_ok = False+ ctx.last_run_blocks_workflow_run_id = "wr_cancel"+ ctx.copilot_run_start_monotonic = time.monotonic()+ _record_composition_page_observation(+ ctx,+ source_tool="evaluate",+ url="https://example.test/login",+ title="Login",+ observed_data={"evidence_text": "The path is login_gated."},+ )++ result = await _maybe_run_completion_verification_from_page_observation(+ ctx,+ url="https://example.test/login",+ title="Login",+ observed_data={"evidence_text": "The path is login_gated."},+ )++ assert handler_calls == 1+ assert result is not None+ assert result.is_fully_satisfied() is False+ verdict_by_id = {verdict.criterion_id: verdict for verdict in result.verdicts}+ assert verdict_by_id["c_validation"] == CriterionVerdict(+ criterion_id="c_validation",+ state="unsatisfied",+ reason_code="no_evidence",+ output_path="path_classification",+ grounding_mode="exact_value",+ has_exact_value=True,+ missing_evidence="missing classification output key path_classification",+ )+ assert verdict_by_id["c_page"].satisfied is True+++ "criterion",+ [+ _criterion(+ "c_validation",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ expected_classification="login_gated",+ ),+ _criterion(+ "c_validation",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ classification_output_key="path_classification",+ ),+ _criterion(+ "c_validation",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ output_path="output.path_classification",+ expected_output_value="login_gated",+ ),+ ],+)+async def test_page_observation_validation_classification_incomplete_contract_cannot_be_judge_approved(+ monkeypatch: pytest.MonkeyPatch,+ criterion: CompletionCriterion,+) -> None:+ handler_calls = 0++ async def handler(**_: object) -> dict:+ nonlocal handler_calls+ handler_calls += 1+ return {+ "verdicts": [+ {"criterion_id": "c_validation", "satisfied": True, "reason_code": "evidence_confirms"},+ {"criterion_id": "c_page", "satisfied": True, "reason_code": "evidence_confirms"},+ ]+ }++ _patch_completion_handler(monkeypatch, handler)+ ctx = _run_ctx()+ ctx.code_artifact_metadata = _metadata_for_requested_paths("path_classification")+ ctx.request_policy = RequestPolicy(+ completion_criteria=[+ criterion,+ _criterion("c_page", "The page observation confirms the path was inspected."),+ ]+ )+ ctx.last_test_ok = False+ ctx.last_run_blocks_workflow_run_id = "wr_cancel"+ ctx.copilot_run_start_monotonic = time.monotonic()+ _record_composition_page_observation(+ ctx,+ source_tool="evaluate",+ url="https://example.test/login",+ title="Login",+ observed_data={"path_classification": "login_gated"},+ )++ result = await _maybe_run_completion_verification_from_page_observation(+ ctx,+ url="https://example.test/login",+ title="Login",+ observed_data={"path_classification": "login_gated"},+ )++ assert handler_calls == 1+ assert result is not None+ assert result.is_fully_satisfied() is False+ verdict_by_id = {verdict.criterion_id: verdict for verdict in result.verdicts}+ assert verdict_by_id["c_validation"].state == "unsatisfied"+ assert verdict_by_id["c_validation"].reason_code == "no_evidence"+ assert verdict_by_id["c_validation"].missing_evidence == "incomplete typed classification contract"+ assert verdict_by_id["c_page"].satisfied is True++@pytest.mark.asyncioasync def test_page_observation_verification_does_not_apply_terminal_goal_record_upgrade(monkeypatch: pytest.MonkeyPatch,diff --git a/tests/unit/test_copilot_request_policy_completion_criteria.py b/tests/unit/test_copilot_request_policy_completion_criteria.pyindex 2ac2c71f3..d1bef12f1 100644--- a/tests/unit/test_copilot_request_policy_completion_criteria.py+++ b/tests/unit/test_copilot_request_policy_completion_criteria.py@@ -65,6 +65,9 @@ def _criterion(deliverable_kind: str | None = None,expected_output_value: str | None = None,expected_output_shape: str | None = None,+ kind: str = "outcome",+ classification_output_key: str | None = None,+ expected_classification: str | bool | None = None,) -> CompletionCriterion:return CompletionCriterion(id=cid,@@ -77,6 +80,9 @@ def _criterion(deliverable_kind=deliverable_kind, # type: ignore[arg-type]expected_output_value=expected_output_value,expected_output_shape=expected_output_shape, # type: ignore[arg-type]+ kind=kind, # type: ignore[arg-type]+ classification_output_key=classification_output_key,+ expected_classification=expected_classification,)@@ -469,6 +475,163 @@ def test_parse_completion_criteria_accepts_only_declared_expected_output_shapes(assert invalid.expected_output_shape is None+def test_parse_completion_criteria_preserves_validation_classification_contract() -> None:+ parsed = _parse_completion_criteria(+ [+ {+ "outcome": "The run classifies whether the path is login gated.",+ "kind": "validation_classification",+ "classification_output_key": "login_gated",+ "expected_classification": True,+ },+ {+ "outcome": "The run classifies whether the path is login gated.",+ "kind": "outcome",+ "classification_output_key": "login_gated",+ "expected_classification": True,+ },+ ]+ )++ assert len(parsed) == 2+ assert parsed[0].kind == "validation_classification"+ assert parsed[0].classification_output_key == "login_gated"+ assert parsed[0].expected_classification is True+ assert parsed[1].kind == "outcome"+++def test_parse_completion_criteria_drops_requested_output_fields_for_validation_classification() -> None:+ parsed = _parse_completion_criteria(+ [+ {+ "outcome": "The run classifies whether the path is login gated.",+ "kind": "validation_classification",+ "output_path": "output.path_classification",+ "expected_output_value": "login_gated",+ "expected_output_shape": "status_label",+ }+ ]+ )++ assert len(parsed) == 1+ assert parsed[0].kind == "validation_classification"+ assert parsed[0].output_path is None+ assert parsed[0].expected_output_value is None+ assert parsed[0].expected_output_shape is None+++def test_parse_completion_criteria_dedupes_validation_classification_by_target_and_output_key() -> None:+ parsed = _parse_completion_criteria(+ [+ {+ "outcome": "The run classifies whether the path is login gated.",+ "kind": "validation_classification",+ "classification_output_key": "path_classification",+ "expected_classification": "login_gated",+ },+ {+ "outcome": "Duplicate phrasing should not matter for the same classification target.",+ "kind": "validation_classification",+ "classification_output_key": "path_classification",+ "expected_classification": "login_gated",+ },+ {+ "outcome": "A different expected classification stays distinct.",+ "kind": "validation_classification",+ "classification_output_key": "path_classification",+ "expected_classification": "public",+ },+ ]+ )++ assert [(criterion.classification_output_key, criterion.expected_classification) for criterion in parsed] == [+ ("path_classification", "login_gated"),+ ("path_classification", "public"),+ ]+++async def test_request_policy_promotes_login_only_output_to_validation_classification_contract() -> None:+ policy = await _policy_for_message(+ (+ "I need a reusable validation-only workflow that checks whether Riverbend Gas has a public path for "+ "starting gas service from http://localhost:8900/utility_services/riverbend_gas/. It should look for a "+ "start, connect, new service, or move-in path without logging in, creating an account, entering identity "+ "details, entering payment details, accepting terms, or submitting anything. For this eval run, classify "+ "the safest reachable next step and return a structured summary showing whether a public form exists, "+ "whether the path is login-only, the visible page/path label, and the recommended next action."+ ),+ [+ {+ "outcome": "The returned record includes public path exists.",+ "output_path": "output.public_path_exists",+ },+ {+ "outcome": "The returned record includes safest reachable next step.",+ "output_path": "output.safest_reachable_next_step",+ },+ {+ "outcome": "The returned record includes login only.",+ "output_path": "output.login_only",+ },+ ],+ )++ validation_criteria = [+ criterion for criterion in policy.completion_criteria if criterion.kind == "validation_classification"+ ]+ assert len(validation_criteria) == 1+ assert validation_criteria[0].classification_output_key == "login_only"+ assert validation_criteria[0].expected_classification is True+ assert validation_criteria[0].output_path is None+ assert validation_criteria[0].expected_output_value is None+ assert not _criteria_for_path(policy, "output.login_only")+++def test_prompt_summary_includes_validation_classification_output_contract() -> None:+ policy = RequestPolicy(+ completion_criteria=[+ _criterion(+ "c_login_only",+ "The run classifies whether the path is login-only.",+ kind="validation_classification",+ classification_output_key="login_only",+ expected_classification=True,+ )+ ]+ )++ summary = policy.prompt_summary()++ assert "validation_classification_output_contracts:" in summary+ assert "criterion_id: c_login_only" in summary+ assert "return_key: login_only" in summary+ assert "expected_value: true" in summary+ assert "return_location: top_level_block_output" in summary+++async def test_request_policy_promotes_path_classification_when_classifier_supplies_target() -> None:+ policy = await _policy_for_message(+ "Classify whether the service path is login-gated and return path classification.",+ [+ {+ "outcome": "The returned record includes path classification.",+ "output_path": "output.path_classification",+ "expected_output_value": "login-gated",+ }+ ],+ )++ assert len(policy.completion_criteria) == 1+ criterion = policy.completion_criteria[0]+ assert criterion.kind == "validation_classification"+ assert criterion.classification_output_key == "path_classification"+ assert criterion.expected_classification == "login-gated"+ assert criterion.output_path is None+ assert criterion.expected_output_value is None++@pytest.mark.asyncioasync def test_requested_output_canonicalization_preserves_classifier_shape_first() -> None:policy = await _policy_for_message(@@ -917,6 +1080,33 @@ def test_active_criteria_rendering_includes_expected_output_shape_as_structured_]+def test_active_criteria_rendering_includes_validation_classification_contract() -> None:+ rendered = _render_active_criteria_for_prompt(+ [+ _criterion(+ "c0",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ classification_output_key="login_gated",+ expected_classification=True,+ )+ ]+ )++ assert json.loads(rendered) == [+ {+ "outcome": "The run classifies whether the path is login gated.",+ "implicit": False,+ "method_mandated": False,+ "level": "run",+ "kind": "validation_classification",+ "terminal_action_family": None,+ "classification_output_key": "login_gated",+ "expected_classification": True,+ }+ ]++def test_criteria_json_round_trips_contingent_on() -> None:criteria = (_criterion(@@ -961,6 +1151,43 @@ def test_criteria_json_round_trips_expected_output_shape() -> None:assert restored == criteria+def test_criteria_json_round_trips_validation_classification_contract() -> None:+ criteria = (+ _criterion(+ "c0",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ classification_output_key="path_classification",+ expected_classification="login_gated",+ ),+ )++ restored = criteria_from_json(criteria_to_json(criteria))++ assert restored == criteria+++def test_criteria_json_rehydrates_validation_classification_without_requested_output_fields() -> None:+ criteria = (+ _criterion(+ "c0",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ output_path="output.path_classification",+ expected_output_value="login_gated",+ expected_output_shape="status_label",+ ),+ )++ restored = criteria_from_json(criteria_to_json(criteria))++ assert len(restored) == 1+ assert restored[0].kind == "validation_classification"+ assert restored[0].output_path is None+ assert restored[0].expected_output_value is None+ assert restored[0].expected_output_shape is None++def test_request_policy_trace_exposes_requested_output_grounding_contract_without_values() -> None:policy = RequestPolicy(completion_criteria=[@@ -1056,6 +1283,36 @@ def test_reconcile_supersedes_same_output_path_with_different_expected_output_shassert decision.criteria == tuple(fresh)+def test_reconcile_supersedes_validation_classification_with_different_target() -> None:+ stored = _stored(+ _criterion(+ "s0",+ "The run classifies whether the path is login gated.",+ kind="validation_classification",+ classification_output_key="path_classification",+ expected_classification="login_gated",+ )+ )+ fresh = [+ _criterion(+ "c0",+ "The run classifies whether the path is public.",+ kind="validation_classification",+ classification_output_key="path_classification",+ expected_classification="public",+ )+ ]++ decision = reconcile_completion_criteria(+ StoredCriteriaSnapshot(active=stored, next_epoch=2),+ fresh,+ actionable=True,+ )++ assert decision.action == "create"+ assert decision.criteria == tuple(fresh)++def test_criteria_json_round_trips_contingent_antecedent_output_path() -> None:criteria = (_criterion(diff --git a/tests/unit/test_copilot_request_policy_prompt.py b/tests/unit/test_copilot_request_policy_prompt.pyindex 21068227a..1451de44c 100644--- a/tests/unit/test_copilot_request_policy_prompt.py+++ b/tests/unit/test_copilot_request_policy_prompt.py@@ -90,12 +90,13 @@ class TestRequestPolicyPromptStructure:assert ("{outcome, contingent_on, contingent_antecedent_output_path, ""deliverable_kind, implicit, method_mandated, level, output_path, expected_output_value, "- "expected_output_shape, kind, terminal_action_family}"+ "expected_output_shape, kind, terminal_action_family, classification_output_key, expected_classification}") in renderedassert "never hide it in outcome prose" in renderedassert "reference_code, numeric_identifier, date, address, status_label, money_amount, owner_label" in rendered- assert "kind=outcome|terminal_action" in rendered+ assert "kind=outcome|terminal_action|validation_classification" in renderedassert "terminal_action_family=request|application|form|order|null" in rendered+ assert "classification_output_key=login_only and expected_classification=true" in renderedassert 'The only supported non-null value is "registered_download"' in rendereddef test_active_completion_criteria_render_typed_terminal_action_fields(self) -> None: