Skip to content

Automate browser based workflows with AI

SKY-11559: Verify validation classifications deterministically (#6902)

Andrew Neilsone1f190a98597
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
index 0da6518db..b0391b3f2 100644
--- a/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-agent.j2
@@ -164,6 +164,7 @@ IMPORTANT WORKFLOW RULES:
* When extracting table-like or list-like records (locations, affiliations, prices, inventory, line items, statuses), iterate the actual row/item elements (`tr`, `[role="row"]`, or equivalent repeated item containers) and read fields from that same row/item. Do NOT scan broad containers such as `section`, `.card`, `article`, or `li` and then infer fields from mixed text; that can combine several rows into one record. Derive summary fields only from the parsed row objects, not from broad page text.
* When a successful search/open-details block reaches a details or profile page, include the visible details/profile page label in the block's returned `evidence_text`. Keep structured record fields focused on the requested data; use the page label only as supporting evidence.
* When a validation-only workflow must stop on a pre-submit Review page, the terminal Review block must return page-grounded values, not only booleans. Include an explicit validation-only marker such as `validation_only: true` or `submit_mode: "validation_only"`, include `review_values` or `review_fields` as visible Review-page label/value strings, include `evidence_text` containing the visible Review-page text that verbatim contains those values, include an explicit false submit/finalize-click signal, and do not click Submit/Finalize.
+* When REQUEST POLICY includes `validation_classification_output_contracts`, those contracts are mandatory code-block return fields. For each contract, the code block that classifies the run MUST explicitly `return` a dict containing the exact `return_key` as a top-level key, with the exact JSON type/value shown by `expected_value` when the classification is true for the reached page. Do not rely on schema defaults or exposed top-level locals for these values. The matching `code_artifact_metadata` row for that block must declare the same `return_key` under `goal_value_paths` and in `extraction_schema`; update_and_run_blocks will reject code-only output blocks without that metadata. For login-gated validation, also return page-grounded display fields such as `visible_page_path_label` with the full visible section/breadcrumb label and `observed_gate_phrase` with the exact visible sign-in/register gate phrase when present. Synonyms such as `path_login_only`, `classification`, or prose in `evidence_text` do not replace the required key unless that exact key is the contract's `return_key`.
* Save the complete reusable workflow, but run only a small verified frontier when the workflow has several page-changing stages. `update_and_run_blocks` accepts the full YAML plus a subset of `block_labels`; for form/search workflows, first run the entry/form-state frontier, then continue with submit/results, then expansion/extraction. After a verified prefix or a budgeted run that leaves the browser on a later page, preserve the upstream workflow text and run only the missing downstream labels. Do not remove later blocks from the YAML just to shorten a test; do not submit a long chain of all blocks at once when each stage can be verified independently.
* Set `complete_criterion` on every decomposed `navigation` block to the observable post-action page state — without it, smaller blocks still return vague "continue" verdicts.
* Use descriptive, unique labels for blocks (snake_case format).
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2
index dd75a36d3..ef0590b24 100644
--- a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2
+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2
@@ -25,15 +25,18 @@ completion_contract:
- Set to null when the user does not provide an explicit completion condition.
completion_criteria:
-- An array of the END-STATE OUTCOMES that must be true for the user's goal to be done. Each item is {outcome, contingent_on, contingent_antecedent_output_path, deliverable_kind, implicit, method_mandated, level, output_path, expected_output_value, expected_output_shape, kind, terminal_action_family}.
+- An array of the END-STATE OUTCOMES that must be true for the user's goal to be done. Each item is {outcome, contingent_on, contingent_antecedent_output_path, deliverable_kind, implicit, method_mandated, level, output_path, expected_output_value, expected_output_shape, kind, terminal_action_family, classification_output_key, expected_classification}.
- outcome: a short statement of an end state the workflow must achieve (an END, not a step). Extract the goal as outcomes the user wants to be true, not the recipe they describe.
- contingent_on: null for ordinary criteria. Use a short antecedent string only when this criterion should be graded only if that antecedent happens, for example {"contingent_on": "the provider site blocks online submission", "contingent_antecedent_output_path": "output.blocker", "outcome": "the blocker is reported to the user"}. The antecedent goes in contingent_on; the consequent to grade goes in outcome. Do not encode if/then criteria only in prose.
- contingent_antecedent_output_path: null unless the antecedent is represented by a structured run output field. The only valid form is output.<field> (for example output.blocker). Do not emit transcript paths, selectors, regexes, JSONPath, screenshots, nested paths, or prose conditions here.
- deliverable_kind: null unless this criterion's requested output is satisfied by execution-owned browser-download registration rather than a user-authored extraction field. The only supported non-null value is "registered_download".
- expected_output_value: null unless this criterion requires an exact structured output value the user explicitly provided. Put that value here as data; never hide it in outcome prose.
- expected_output_shape: null unless this criterion requires a generated/provider structured output value whose exact value is not known before the run. Use only one of: reference_code, numeric_identifier, date, address, status_label, money_amount, owner_label. Do not use this as a non-empty-value proxy; leave null when the field's shape is not one of those declared types.
-- kind=outcome|terminal_action. Use terminal_action only when the criterion's end state is a terminal action in the external site for one of the supported families below. Use outcome for all other outcomes.
-- terminal_action_family=request|application|form|order|null. Set the matching family when kind=terminal_action; set null when kind=outcome.
+- kind=outcome|terminal_action|validation_classification. Use validation_classification only when the user's no-side-effect goal is itself a judgment/classification returned by the workflow, such as whether a path is login-gated. Use terminal_action only when the criterion's end state is a terminal action in the external site for one of the supported families below. Use outcome for all other outcomes.
+- terminal_action_family=request|application|form|order|null. Set the matching family when kind=terminal_action; set null otherwise.
+- classification_output_key: null unless kind=validation_classification. Set it to the exact top-level block output field/key that will carry the classification scalar. Do not use JSONPath, regex, transcript paths, screenshots, prose conditions, or nested paths.
+- expected_classification: null unless kind=validation_classification. Set it to the exact expected classification target as a boolean or canonical enum/string value. Do not hide this target in outcome prose.
+- For validation-only utility/path checks whose goal is to classify whether the reachable path is login-only/login-gated, prefer kind=validation_classification with classification_output_key=login_only and expected_classification=true. Do not represent that classification target as a requested-output criterion such as output.login_only.
- Capture ALL distinct outcomes the goal requires, including implicit constraints the user did not state as a step (for example "added exactly once / not duplicated", "the item is present in the cart"). Mark such items implicit=true.
- A prescribed HOW step is NOT a criterion. "search for X", "open the result lightbox", "click add to cart", "go to the cart page" are methods/navigation, not outcomes — do not list them. A different or shorter path that reaches the same end state must still satisfy the goal.
- Telling/showing/reporting results to the user is NOT a criterion. "tell me what it extracted", "show me the data", "confirm to me it worked" describe the chat reply, which no workflow definition or run can evidence — capture only the underlying workflow outcome (the data is extracted into the run output, the form is submitted) and list nothing for the reporting itself.
diff --git a/skyvern/forge/sdk/copilot/agent.py b/skyvern/forge/sdk/copilot/agent.py
index 91c50777a..355d954bf 100644
--- a/skyvern/forge/sdk/copilot/agent.py
+++ b/skyvern/forge/sdk/copilot/agent.py
@@ -1491,7 +1491,7 @@ async def _build_goal_satisfied_exit_result(ctx: CopilotContext, global_llm_cont
verified_workflow, verified_yaml = _verified_workflow_or_none(ctx)
clean_test = ctx.last_test_ok is True and ctx.last_full_workflow_test_ok is True
if clean_test and verified_goal_claim_authorized(ctx):
- user_response = _VERIFIED_WORKFLOW_SUCCESS_REPLY
+ user_response = _verified_workflow_success_reply(ctx)
elif clean_test:
user_response = (
"I built the workflow and the test run completed, but the goal outcome was not "
@@ -1687,6 +1687,120 @@ _UNVALIDATED_PROPOSAL_AFFORDANCE = (
"I have a draft workflow proposal. Use Review to inspect it, Accept to save it, or Reject to discard it. "
"It has not been tested or verified end-to-end."
)
+_VERIFIED_CLASSIFICATION_CONTEXT_KEYS = (
+ "visible_page_path_label",
+ "safest_reachable_next_step",
+ "recommended_next_action",
+)
+_VERIFIED_CLASSIFICATION_GATE_KEYS = (
+ "observed_gate_phrase",
+ "gate_phrase",
+ "gate_summary",
+ "blocked_by",
+)
+_VERIFIED_CLASSIFICATION_GATE_PHRASES = ("Sign in or register to continue",)
+_VERIFIED_TERMINAL_VALUE_MAX_CHARS = 180
+
+
+def _terminal_summary_scalar(value: Any) -> str | None:
+ if isinstance(value, bool):
+ return "true" if value else "false"
+ if isinstance(value, int | float) and not isinstance(value, bool):
+ return str(value)
+ if not isinstance(value, str):
+ return None
+ cleaned = " ".join(value.strip().split())
+ if not cleaned or contains_internal_machinery_leak(cleaned):
+ return None
+ if len(cleaned) > _VERIFIED_TERMINAL_VALUE_MAX_CHARS:
+ cleaned = cleaned[: _VERIFIED_TERMINAL_VALUE_MAX_CHARS - 1].rstrip() + "..."
+ return cleaned
+
+
+def _verified_output_value(ctx: CopilotContext, output_key: str | None) -> Any:
+ if not output_key:
+ return None
+ block_outputs = ctx.verified_terminal_block_outputs or ctx.verified_block_outputs or {}
+ for output in block_outputs.values():
+ if isinstance(output, dict) and output_key in output:
+ return output.get(output_key)
+ return None
+
+
+def _verified_adjacent_output_parts(ctx: CopilotContext) -> list[str]:
+ parts: list[str] = []
+ block_outputs = ctx.verified_terminal_block_outputs or ctx.verified_block_outputs or {}
+ login_gate_phrase_found = False
+
+ def append_gate_phrase(value: Any) -> None:
+ nonlocal login_gate_phrase_found
+ if isinstance(value, str):
+ for phrase in _VERIFIED_CLASSIFICATION_GATE_PHRASES:
+ if phrase.lower() in value.lower():
+ login_gate_phrase_found = True
+ parts.append(f"observed_gate_phrase={phrase}")
+ return
+ if isinstance(value, list | tuple | set):
+ for item in value:
+ append_gate_phrase(item)
+
+ for output in block_outputs.values():
+ if not isinstance(output, dict):
+ continue
+ for key in (*_VERIFIED_CLASSIFICATION_CONTEXT_KEYS, *_VERIFIED_CLASSIFICATION_GATE_KEYS):
+ value = _terminal_summary_scalar(output.get(key))
+ if value is not None:
+ parts.append(f"{key}={value}")
+ for value in output.values():
+ append_gate_phrase(value)
+ if login_gate_phrase_found:
+ parts.append("login-gated")
+ return list(dict.fromkeys(parts))
+
+
+def _verified_classification_summary(ctx: CopilotContext) -> str | None:
+ if not verified_goal_claim_authorized(ctx) or not outcome_fully_verified(ctx):
+ return None
+ result = ctx.completion_verification_result
+ if result is None or not result.is_fully_satisfied():
+ return None
+ policy = ctx.request_policy if isinstance(ctx.request_policy, RequestPolicy) else None
+ if policy is None:
+ return None
+ verdict_by_id = {verdict.criterion_id: verdict for verdict in result.verdicts}
+ parts: list[str] = []
+ login_gated_confirmed = False
+ for criterion in policy.completion_criteria:
+ if criterion.kind != "validation_classification":
+ continue
+ verdict = verdict_by_id.get(criterion.id)
+ if verdict is None or not verdict.satisfied:
+ continue
+ output_key = criterion.classification_output_key or verdict.output_path
+ actual = _verified_output_value(ctx, output_key)
+ display_value = _terminal_summary_scalar(actual)
+ if output_key and display_value is not None:
+ parts.append(f"{output_key}={display_value}")
+ actual_text = actual.strip().lower() if isinstance(actual, str) else None
+ if (
+ isinstance(output_key, str)
+ and output_key in {"login_only", "login_gated", "path_login_only"}
+ and actual is True
+ ) or actual_text in {"login-gated", "login_gated"}:
+ login_gated_confirmed = True
+ if login_gated_confirmed:
+ parts.append("login-gated")
+ parts.extend(_verified_adjacent_output_parts(ctx))
+ if not parts:
+ return None
+ return "; ".join(dict.fromkeys(parts))
+
+
+def _verified_workflow_success_reply(ctx: CopilotContext) -> str:
+ summary = _verified_classification_summary(ctx)
+ if summary is None:
+ return _VERIFIED_WORKFLOW_SUCCESS_REPLY
+ return f"{_VERIFIED_WORKFLOW_SUCCESS_REPLY} Verified result: {summary}."
@dataclass(frozen=True)
@@ -1722,7 +1836,7 @@ def _render_typed_run_outcome_reply(
if not has_verified_workflow or not verified_goal_claim_authorized(ctx):
return None
return _TypedRunOutcomeReply(
- user_response=f"{_VERIFIED_WORKFLOW_SUCCESS_REPLY} The latest run demonstrated the requested outcome.",
+ user_response=f"{_verified_workflow_success_reply(ctx)} The latest run demonstrated the requested outcome.",
demonstrated=True,
)
@@ -1785,7 +1899,7 @@ def _verified_terminal_preserve_result(
if verified_workflow is None:
return None
final_text, outcome = apply_repeated_reply_guard(
- final_text=_VERIFIED_WORKFLOW_SUCCESS_REPLY,
+ final_text=_verified_workflow_success_reply(ctx),
attempted_kind=derive_response_kind(ctx.turn_intent),
blocked_signatures=ctx.blocked_reply_signatures,
terminal_reason="verified_goal_satisfied",
@@ -2715,7 +2829,7 @@ async def _translate_to_agent_result(
)
if verified_terminal_ready:
resp_type = "REPLY"
- user_response = _VERIFIED_WORKFLOW_SUCCESS_REPLY
+ user_response = _verified_workflow_success_reply(ctx)
agent_admits_incomplete = False
else:
# Default-true preserves backwards-compat with stale prompts and missing fields.
diff --git a/skyvern/forge/sdk/copilot/completion_criteria_store.py b/skyvern/forge/sdk/copilot/completion_criteria_store.py
index c4ac27145..aabae500d 100644
--- a/skyvern/forge/sdk/copilot/completion_criteria_store.py
+++ b/skyvern/forge/sdk/copilot/completion_criteria_store.py
@@ -20,6 +20,8 @@ from skyvern.forge.sdk.copilot.request_policy import (
CriterionKind,
ExpectedOutputShape,
TerminalActionFamily,
+ _coerce_classification_output_key,
+ _coerce_expected_classification,
_coerce_expected_output_shape,
_normalize_contingent_antecedent_output_path,
_normalize_deliverable_kind,
@@ -36,7 +38,7 @@ CRITERIA_SET_STATUS_ACTIVE = "active"
CRITERIA_SET_STATUS_SUPERSEDED = "superseded"
TRIPWIRE_CONSECUTIVE_ALL_NO_EVIDENCE = 2
_CRITERION_LEVELS = ("definition", "run")
-_CRITERION_KINDS = ("outcome", "terminal_action")
+_CRITERION_KINDS = ("outcome", "terminal_action", "validation_classification")
_TERMINAL_ACTION_FAMILIES = ("request", "application", "form", "order")
@@ -129,6 +131,8 @@ def criteria_to_json(criteria: tuple[CompletionCriterion, ...] | list[Completion
"expected_output_shape": criterion.expected_output_shape,
"kind": criterion.kind,
"terminal_action_family": criterion.terminal_action_family,
+ "classification_output_key": criterion.classification_output_key,
+ "expected_classification": criterion.expected_classification,
}
for criterion in criteria
]
@@ -149,6 +153,8 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:
output_path = item.get("output_path")
expected_output_value = item.get("expected_output_value")
expected_output_shape = _coerce_expected_output_shape(item.get("expected_output_shape"))
+ classification_output_key = _coerce_classification_output_key(item.get("classification_output_key"))
+ expected_classification = _coerce_expected_classification(item.get("expected_classification"))
contingent_on = item.get("contingent_on")
contingent_antecedent_output_path = _normalize_contingent_antecedent_output_path(
item.get("contingent_antecedent_output_path")
@@ -159,6 +165,17 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:
terminal_action_family = (
family_raw if kind == "terminal_action" and family_raw in _TERMINAL_ACTION_FAMILIES else None
)
+ stored_output_path = output_path.strip() if isinstance(output_path, str) and output_path.strip() else None
+ stored_expected_output_value = (
+ expected_output_value.strip()
+ if isinstance(expected_output_value, str) and expected_output_value.strip()
+ else None
+ )
+ stored_expected_output_shape = cast(ExpectedOutputShape | None, expected_output_shape)
+ if kind == "validation_classification":
+ stored_output_path = None
+ stored_expected_output_value = None
+ stored_expected_output_shape = None
criteria.append(
CompletionCriterion(
id=criterion_id,
@@ -171,13 +188,13 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:
implicit=bool(item.get("implicit")),
method_mandated=bool(item.get("method_mandated")),
level=level if isinstance(level, str) and level in _CRITERION_LEVELS else "run", # type: ignore[arg-type]
- output_path=output_path.strip() if isinstance(output_path, str) and output_path.strip() else None,
- expected_output_value=expected_output_value.strip()
- if isinstance(expected_output_value, str) and expected_output_value.strip()
- else None,
- expected_output_shape=cast(ExpectedOutputShape | None, expected_output_shape),
+ output_path=stored_output_path,
+ expected_output_value=stored_expected_output_value,
+ expected_output_shape=stored_expected_output_shape,
kind=cast(CriterionKind, kind),
terminal_action_family=cast(TerminalActionFamily | None, terminal_action_family),
+ classification_output_key=classification_output_key,
+ expected_classification=expected_classification,
)
)
return tuple(criteria)
@@ -189,6 +206,10 @@ def _criterion_reconcile_key(criterion: CompletionCriterion) -> str:
deliverable_kind_key = criterion.deliverable_kind or ""
expected_output_value_key = criterion.expected_output_value or ""
expected_output_shape_key = criterion.expected_output_shape or ""
+ classification_output_key = criterion.classification_output_key or ""
+ expected_classification_key = (
+ str(criterion.expected_classification) if criterion.expected_classification is not None else ""
+ )
if criterion.output_path:
return (
f"contingent:{contingent_key}\x1fantecedent_path:{contingent_path_key}"
@@ -196,10 +217,22 @@ def _criterion_reconcile_key(criterion: CompletionCriterion) -> str:
f"\x1foutput_path:{criterion.output_path}"
f"\x1fexpected_output_value:{expected_output_value_key}"
f"\x1fexpected_output_shape:{expected_output_shape_key}"
+ f"\x1fkind:{criterion.kind}"
+ f"\x1fclassification_output_key:{classification_output_key}"
+ f"\x1fexpected_classification:{expected_classification_key}"
+ )
+ if criterion.kind == "validation_classification":
+ return (
+ f"contingent:{contingent_key}\x1fantecedent_path:{contingent_path_key}"
+ f"\x1fdeliverable_kind:{deliverable_kind_key}"
+ f"\x1fkind:{criterion.kind}"
+ f"\x1fclassification_output_key:{classification_output_key}"
+ f"\x1fexpected_classification:{expected_classification_key}"
)
return (
f"contingent:{contingent_key}\x1fantecedent_path:{contingent_path_key}"
f"\x1fdeliverable_kind:{deliverable_kind_key}"
+ f"\x1fkind:{criterion.kind}"
f"\x1foutcome:{normalized_criterion_outcome_key(criterion.outcome)}"
)
diff --git a/skyvern/forge/sdk/copilot/completion_output_grounding.py b/skyvern/forge/sdk/copilot/completion_output_grounding.py
index 8f6461e25..030a60c3a 100644
--- a/skyvern/forge/sdk/copilot/completion_output_grounding.py
+++ b/skyvern/forge/sdk/copilot/completion_output_grounding.py
@@ -40,6 +40,7 @@ def split_requested_output_criteria(
criterion.output_path
and criterion.level != "definition"
and not criterion.method_mandated
+ and criterion.kind != "validation_classification"
and _normalize_output_path(criterion.output_path)
):
requested.append(criterion)
diff --git a/skyvern/forge/sdk/copilot/completion_verification.py b/skyvern/forge/sdk/copilot/completion_verification.py
index b35862840..8569879f9 100644
--- a/skyvern/forge/sdk/copilot/completion_verification.py
+++ b/skyvern/forge/sdk/copilot/completion_verification.py
@@ -282,6 +282,7 @@ class RunEvidenceSnapshot:
_UNAVAILABLE = CompletionVerificationResult(status="unavailable")
_MISSING_VERDICT_EVIDENCE = "judge did not return a verdict for this criterion"
+_INCOMPLETE_VALIDATION_CLASSIFICATION_CONTRACT = "incomplete typed classification contract"
_MISSING_REGISTERED_DOWNLOAD_EVIDENCE = "run output did not include a non-empty registered browser download"
@@ -395,8 +396,21 @@ def _render_criteria(criteria: list[CompletionCriterion]) -> str:
)
deliverable_kind = f" [deliverable_kind={criterion.deliverable_kind}]" if criterion.deliverable_kind else ""
output_path = f" [required_output_path={criterion.output_path}]" if criterion.output_path else ""
+ classification_output_key = (
+ f" [classification_output_key={criterion.classification_output_key}]"
+ if criterion.classification_output_key
+ else ""
+ )
+ expected_classification = (
+ f" [expected_classification={criterion.expected_classification}]"
+ if criterion.expected_classification is not None
+ else ""
+ )
parts.append(
- f"- {criterion.id}: {criterion.outcome}{contingent_on}{antecedent_output_path}{deliverable_kind}{output_path}{flags}"
+ f"- {criterion.id}: {criterion.outcome}"
+ f" [kind={criterion.kind}]"
+ f"{contingent_on}{antecedent_output_path}{deliverable_kind}{output_path}"
+ f"{classification_output_key}{expected_classification}{flags}"
)
return "\n".join(parts)
@@ -837,6 +851,166 @@ def grade_present_value_criteria(
return verdicts
+def _classification_string(value: str) -> str:
+ return " ".join(value.casefold().split())
+
+
+def _classification_value_matches(expected: str | bool, actual: Any) -> bool:
+ if isinstance(expected, bool):
+ return isinstance(actual, bool) and actual is expected
+ if not isinstance(actual, str):
+ return False
+ normalized_expected = _classification_string(expected)
+ normalized_actual = _classification_string(actual)
+ return bool(normalized_expected) and normalized_actual == normalized_expected
+
+
+def _is_empty_classification_value(value: Any) -> bool:
+ return value is None or value == "" or value == [] or value == {}
+
+
+def _classification_canonical_value(expected: str | bool, actual: Any) -> str | bool | None:
+ if not _classification_value_matches(expected, actual):
+ return None
+ if isinstance(expected, bool):
+ return actual
+ return _classification_string(actual)
+
+
+def _classification_output_candidates(
+ snapshot: RunEvidenceSnapshot,
+ output_key: str,
+) -> list[tuple[str, Any]]:
+ candidates: list[tuple[str, Any]] = []
+ for label, payload in snapshot.block_outputs.items():
+ if isinstance(payload, dict) and output_key in payload:
+ candidates.append((f"block_outputs:{label}.{output_key}", payload[output_key]))
+ return sorted(candidates, key=lambda candidate: candidate[0])
+
+
+def _validation_classification_unsatisfied(
+ criterion: CompletionCriterion,
+ *,
+ reason_code: Literal["no_evidence", "evidence_contradicts"],
+ output_key: str,
+ evidence_ref: str | None = None,
+ missing_evidence: str | None = None,
+) -> CriterionVerdict:
+ return CriterionVerdict(
+ criterion_id=criterion.id,
+ state="unsatisfied",
+ reason_code=reason_code,
+ evidence_ref=evidence_ref,
+ missing_evidence=missing_evidence,
+ output_path=output_key,
+ grounding_mode="exact_value",
+ has_exact_value=True,
+ )
+
+
+def grade_validation_classification_criteria(
+ criteria: list[CompletionCriterion], snapshot: RunEvidenceSnapshot
+) -> list[CriterionVerdict]:
+ verdicts: list[CriterionVerdict] = []
+ for criterion in criteria:
+ if criterion.kind != "validation_classification":
+ continue
+ output_key = criterion.classification_output_key
+ expected = criterion.expected_classification
+ if output_key is None or expected is None:
+ verdicts.append(
+ _validation_classification_unsatisfied(
+ criterion,
+ reason_code="no_evidence",
+ output_key=output_key or "",
+ missing_evidence=_INCOMPLETE_VALIDATION_CLASSIFICATION_CONTRACT,
+ )
+ )
+ continue
+ candidates = _classification_output_candidates(snapshot, output_key)
+ if not candidates:
+ verdicts.append(
+ _validation_classification_unsatisfied(
+ criterion,
+ reason_code="no_evidence",
+ output_key=output_key,
+ missing_evidence=f"missing classification output key {output_key}",
+ )
+ )
+ continue
+ if len(candidates) == 1:
+ evidence_ref, actual = candidates[0]
+ if _is_empty_classification_value(actual):
+ verdicts.append(
+ _validation_classification_unsatisfied(
+ criterion,
+ reason_code="no_evidence",
+ output_key=output_key,
+ evidence_ref=evidence_ref,
+ missing_evidence=f"empty classification output key {output_key}",
+ )
+ )
+ continue
+ if not _classification_value_matches(expected, actual):
+ verdicts.append(
+ _validation_classification_unsatisfied(
+ criterion,
+ reason_code="evidence_contradicts",
+ output_key=output_key,
+ evidence_ref=evidence_ref,
+ missing_evidence=f"classification output key {output_key} did not match expected value",
+ )
+ )
+ continue
+ verdicts.append(
+ CriterionVerdict(
+ criterion_id=criterion.id,
+ state="satisfied",
+ reason_code="evidence_confirms",
+ evidence_ref=evidence_ref,
+ output_path=output_key,
+ grounding_mode="exact_value",
+ has_exact_value=True,
+ )
+ )
+ continue
+ matching_canonical_values: set[str | bool] = set()
+ contradiction_evidence_ref: str | None = None
+ for evidence_ref, actual in candidates:
+ if _is_empty_classification_value(actual):
+ contradiction_evidence_ref = evidence_ref
+ break
+ canonical_value = _classification_canonical_value(expected, actual)
+ if canonical_value is None:
+ contradiction_evidence_ref = evidence_ref
+ break
+ matching_canonical_values.add(canonical_value)
+ if contradiction_evidence_ref is not None or len(matching_canonical_values) != 1:
+ verdicts.append(
+ _validation_classification_unsatisfied(
+ criterion,
+ reason_code="evidence_contradicts",
+ output_key=output_key,
+ evidence_ref=contradiction_evidence_ref,
+ missing_evidence=f"classification output key {output_key} had contradictory evidence",
+ )
+ )
+ continue
+ evidence_ref = candidates[0][0]
+ verdicts.append(
+ CriterionVerdict(
+ criterion_id=criterion.id,
+ state="satisfied",
+ reason_code="evidence_confirms",
+ evidence_ref=evidence_ref,
+ output_path=output_key,
+ grounding_mode="exact_value",
+ has_exact_value=True,
+ )
+ )
+ return verdicts
+
+
def grade_record_semantic_consistency(
criteria: list[CompletionCriterion], snapshot: RunEvidenceSnapshot
) -> list[CriterionVerdict]:
diff --git a/skyvern/forge/sdk/copilot/context.py b/skyvern/forge/sdk/copilot/context.py
index e991fad6a..9f164121a 100644
--- a/skyvern/forge/sdk/copilot/context.py
+++ b/skyvern/forge/sdk/copilot/context.py
@@ -597,6 +597,7 @@ class CopilotContext(AgentContext):
# state untouched, because the browser session is now in post-failure
# state and the prefix labels can no longer be trusted as an anchor.
verified_block_outputs: dict[str, Any] = field(default_factory=dict)
+ verified_terminal_block_outputs: dict[str, Any] = field(default_factory=dict)
verified_prefix_labels: list[str] = field(default_factory=list)
verified_prefix_current_url: str | None = None
last_requested_block_labels: list[str] = field(default_factory=list)
diff --git a/skyvern/forge/sdk/copilot/request_policy.py b/skyvern/forge/sdk/copilot/request_policy.py
index 03a4b30fc..01956dfc1 100644
--- a/skyvern/forge/sdk/copilot/request_policy.py
+++ b/skyvern/forge/sdk/copilot/request_policy.py
@@ -4,7 +4,7 @@ import asyncio
import json
import re
import time
-from dataclasses import dataclass, field
+from dataclasses import dataclass, field, replace
from typing import Any, Literal, cast, get_args
from urllib.parse import urlparse
@@ -155,8 +155,17 @@ _MAX_TRACE_COMPLETION_CRITERIA = 8
_COMPLETION_CRITERION_OUTCOME_MAX_CHARS = 200
_COMPLETION_CRITERION_CONTINGENT_ON_MAX_CHARS = 200
_COMPLETION_CRITERION_EXPECTED_VALUE_MAX_CHARS = 500
+_COMPLETION_CRITERION_CLASSIFICATION_TARGET_MAX_CHARS = 120
+_CLASSIFICATION_OUTPUT_KEY_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
_CONTINGENT_ANTECEDENT_OUTPUT_PATH_RE = re.compile(r"^output\.[A-Za-z_][A-Za-z0-9_]*$")
_REQUESTED_OUTPUT_CRITERION_ID_PREFIX = "__copilot_requested_output__"
+_VALIDATION_CLASSIFICATION_BOOLEAN_OUTPUT_TARGETS: dict[str, tuple[str, bool]] = {
+ "output.login_only": ("login_only", True),
+ "output.login_gated": ("login_gated", True),
+}
+_VALIDATION_CLASSIFICATION_LABEL_OUTPUT_TARGETS: dict[str, str] = {
+ "output.path_classification": "path_classification",
+}
FALLBACK_FLOOR_CRITERION_ID_PREFIX = "__copilot_fallback_floor__"
_FALLBACK_FLOOR_BASE_ID = f"{FALLBACK_FLOOR_CRITERION_ID_PREFIX}run"
@@ -166,8 +175,9 @@ _FALLBACK_FLOOR_CREDENTIAL_OUTCOME = "The credentialed step authenticates and re
CriterionLevel = Literal["definition", "run"]
_CRITERION_LEVELS: frozenset[str] = frozenset({"definition", "run"})
-CriterionKind = Literal["outcome", "terminal_action"]
+CriterionKind = Literal["outcome", "terminal_action", "validation_classification"]
TerminalActionFamily = Literal["request", "application", "form", "order"]
+ClassificationTarget = str | bool
ExpectedOutputShape = Literal[
"reference_code",
"numeric_identifier",
@@ -177,7 +187,7 @@ ExpectedOutputShape = Literal[
"money_amount",
"owner_label",
]
-_CRITERION_KINDS: frozenset[str] = frozenset({"outcome", "terminal_action"})
+_CRITERION_KINDS: frozenset[str] = frozenset({"outcome", "terminal_action", "validation_classification"})
_TERMINAL_ACTION_FAMILIES: frozenset[str] = frozenset({"request", "application", "form", "order"})
_EXPECTED_OUTPUT_SHAPES: frozenset[str] = frozenset(get_args(ExpectedOutputShape))
@@ -234,6 +244,8 @@ class CompletionCriterion:
expected_output_shape: ExpectedOutputShape | None = None
kind: CriterionKind = "outcome"
terminal_action_family: TerminalActionFamily | None = None
+ classification_output_key: str | None = None
+ expected_classification: ClassificationTarget | None = None
@dataclass
@@ -329,6 +341,22 @@ class RequestPolicy:
lines.append(f"completion_contract: {self.completion_contract}")
if self.raw_secret_detected:
lines.append(f"raw_secret_detected: {self.raw_secret_detected}")
+ validation_classification_criteria = [
+ criterion
+ for criterion in self.graded_completion_criteria()
+ if criterion.kind == "validation_classification"
+ and criterion.classification_output_key
+ and criterion.expected_classification is not None
+ ]
+ if validation_classification_criteria:
+ lines.append("validation_classification_output_contracts:")
+ for criterion in validation_classification_criteria:
+ lines.append(
+ f"- criterion_id: {criterion.id}; "
+ f"return_key: {criterion.classification_output_key}; "
+ f"expected_value: {json.dumps(criterion.expected_classification)}; "
+ "return_location: top_level_block_output"
+ )
if self.resolved_credentials:
lines += [
"resolved_credentials:",
@@ -560,6 +588,22 @@ def _coerce_expected_output_shape(value: Any) -> ExpectedOutputShape | None:
return None
+def _coerce_classification_output_key(value: Any) -> str | None:
+ if not isinstance(value, str):
+ return None
+ key = value.strip()
+ return key if _CLASSIFICATION_OUTPUT_KEY_RE.fullmatch(key) else None
+
+
+def _coerce_expected_classification(value: Any) -> ClassificationTarget | None:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ collapsed = " ".join(value.split())[:_COMPLETION_CRITERION_CLASSIFICATION_TARGET_MAX_CHARS].strip()
+ return collapsed or None
+ return None
+
+
def _coerce_classifier_payload(raw: Any) -> dict[str, Any] | None:
if isinstance(raw, str):
raw = parse_final_response(raw)
@@ -633,7 +677,7 @@ def _parse_completion_criteria(raw: Any) -> list[CompletionCriterion]:
if not isinstance(raw, list):
return []
criteria: list[CompletionCriterion] = []
- seen: set[tuple[str, str, str, str]] = set()
+ seen: set[tuple[str, str, str, str, str, str]] = set()
for item in raw:
if not isinstance(item, dict):
continue
@@ -664,17 +708,33 @@ def _parse_completion_criteria(raw: Any) -> list[CompletionCriterion]:
item.get("contingent_antecedent_output_path")
)
deliverable_kind = _normalize_deliverable_kind(item.get("deliverable_kind"))
+ kind = _coerce_criterion_kind(item.get("kind"))
+ classification_output_key = (
+ _coerce_classification_output_key(item.get("classification_output_key"))
+ if kind == "validation_classification"
+ else None
+ )
+ expected_classification = (
+ _coerce_expected_classification(item.get("expected_classification"))
+ if kind == "validation_classification"
+ else None
+ )
+ if kind == "validation_classification":
+ output_path = None
+ expected_output_value = None
+ expected_output_shape = None
key = (
contingent_on or "",
contingent_antecedent_output_path or "",
- output_path or normalized_criterion_outcome_key(outcome),
+ output_path or classification_output_key or normalized_criterion_outcome_key(outcome),
deliverable_kind or "",
+ kind,
+ str(expected_classification) if expected_classification is not None else "",
)
if key in seen:
continue
seen.add(key)
level_raw = item.get("level")
- kind = _coerce_criterion_kind(item.get("kind"))
criteria.append(
CompletionCriterion(
id=f"c{len(criteria)}",
@@ -692,6 +752,8 @@ def _parse_completion_criteria(raw: Any) -> list[CompletionCriterion]:
expected_output_shape=expected_output_shape,
kind=kind,
terminal_action_family=_coerce_terminal_action_family(item.get("terminal_action_family"), kind),
+ classification_output_key=classification_output_key,
+ expected_classification=expected_classification,
)
)
if len(criteria) >= _MAX_COMPLETION_CRITERIA:
@@ -1125,6 +1187,54 @@ def _apply_requested_output_completion_criteria(
)
+def _validation_classification_target_for_legacy_criterion(
+ criterion: CompletionCriterion,
+) -> tuple[str, ClassificationTarget] | None:
+ if criterion.kind != "outcome" or criterion.level == "definition" or criterion.method_mandated:
+ return None
+ if not criterion.output_path:
+ return None
+ boolean_target = _VALIDATION_CLASSIFICATION_BOOLEAN_OUTPUT_TARGETS.get(criterion.output_path)
+ if boolean_target is not None:
+ return boolean_target
+ label_key = _VALIDATION_CLASSIFICATION_LABEL_OUTPUT_TARGETS.get(criterion.output_path)
+ if label_key is None or criterion.expected_output_value is None:
+ return None
+ return (label_key, criterion.expected_output_value)
+
+
+def _apply_validation_classification_completion_criteria(policy: RequestPolicy) -> None:
+ """Promote legacy classifier output carriers into typed validation-classification contracts."""
+
+ criteria: list[CompletionCriterion] = []
+ seen_classification_targets: set[tuple[str, str]] = set()
+ for criterion in policy.completion_criteria:
+ target = _validation_classification_target_for_legacy_criterion(criterion)
+ if target is not None:
+ output_key, expected = target
+ criterion = replace(
+ criterion,
+ kind="validation_classification",
+ output_path=None,
+ expected_output_value=None,
+ expected_output_shape=None,
+ deliverable_kind=None,
+ terminal_action_family=None,
+ classification_output_key=output_key,
+ expected_classification=expected,
+ )
+ if criterion.kind == "validation_classification":
+ target_key = (
+ criterion.classification_output_key or "",
+ str(criterion.expected_classification) if criterion.expected_classification is not None else "",
+ )
+ if target_key in seen_classification_targets:
+ continue
+ seen_classification_targets.add(target_key)
+ criteria.append(criterion)
+ policy.completion_criteria = criteria
+
+
def _render_active_criteria_for_prompt(criteria: list[CompletionCriterion] | None) -> str:
if not criteria:
return ""
@@ -1150,6 +1260,10 @@ def _render_active_criteria_for_prompt(criteria: list[CompletionCriterion] | Non
item["expected_output_value"] = criterion.expected_output_value
if criterion.expected_output_shape:
item["expected_output_shape"] = criterion.expected_output_shape
+ if criterion.classification_output_key:
+ item["classification_output_key"] = criterion.classification_output_key
+ if criterion.expected_classification is not None:
+ item["expected_classification"] = criterion.expected_classification
rendered.append(item)
return json.dumps(rendered)
@@ -1466,6 +1580,7 @@ async def _classify_request(
policy.requires_user_clarification = False
policy.raw_secret_evidence = None
_apply_requested_output_completion_criteria(policy, user_message, requested_output_path_aliases)
+ _apply_validation_classification_completion_criteria(policy)
policy.completion_contract_status = (
"present" if policy.completion_contract or policy.graded_completion_criteria() else "absent"
)
diff --git a/skyvern/forge/sdk/copilot/tools/completion.py b/skyvern/forge/sdk/copilot/tools/completion.py
index 878d54527..9a5de96d7 100644
--- a/skyvern/forge/sdk/copilot/tools/completion.py
+++ b/skyvern/forge/sdk/copilot/tools/completion.py
@@ -24,6 +24,7 @@ from skyvern.forge.sdk.copilot.completion_verification import (
grade_registered_download_criteria,
grade_structured_record_criteria,
grade_terminal_goal_record_criteria,
+ grade_validation_classification_criteria,
is_fallback_floor_base_criterion,
is_registered_download_completion_criterion,
registered_download_completion_criterion,
@@ -309,16 +310,26 @@ async def _maybe_run_completion_verification_from_page_observation(
if requested_output_criteria
else []
)
+ validation_classification_verdicts = grade_validation_classification_criteria(judgeable_run_criteria, snapshot)
+ validation_classification_ids = {verdict.criterion_id for verdict in validation_classification_verdicts}
+ remaining_judgeable_run_criteria = [
+ criterion for criterion in judgeable_run_criteria if criterion.id not in validation_classification_ids
+ ]
remaining = _copilot_seconds_remaining(copilot_ctx)
if (
remaining is not None
and remaining
<= settings.COPILOT_COMPLETION_JUDGE_TIMEOUT_SECONDS + _COMPLETION_VERIFICATION_BUDGET_MARGIN_SECONDS
):
- run_result = _merge_run_verdicts_if_requested_output_exists(
- run_criteria,
- requested_output_verdicts,
- structural_unfired_criterion_ids=run_structural_unfired_ids,
+ run_result = (
+ _merge_run_verdicts(
+ run_criteria,
+ requested_output_verdicts,
+ validation_classification_verdicts,
+ structural_unfired_criterion_ids=run_structural_unfired_ids,
+ )
+ if requested_output_verdicts or validation_classification_verdicts
+ else None
)
verification = (
combine_verification_results(
@@ -345,47 +356,66 @@ async def _maybe_run_completion_verification_from_page_observation(
status="evaluated",
criterion_ids=[criterion.id for criterion in run_criteria],
verdicts=requested_output_verdicts
+ + validation_classification_verdicts
+ [
CriterionVerdict(criterion_id=criterion.id, state="unsatisfied", reason_code="no_evidence")
- for criterion in judgeable_run_criteria
+ for criterion in remaining_judgeable_run_criteria
],
contingent_criterion_ids=run_contingent_ids,
contingent_on_by_criterion_id=run_contingent_on_by_id,
contingent_antecedent_output_path_by_criterion_id=run_contingent_path_by_id,
structural_unfired_criterion_ids=run_structural_unfired_ids,
)
- elif not judgeable_run_criteria:
+ elif not remaining_judgeable_run_criteria:
run_result = _merge_run_verdicts(
run_criteria,
requested_output_verdicts,
+ validation_classification_verdicts,
structural_unfired_criterion_ids=run_structural_unfired_ids,
)
else:
handler = await _completion_verification_handler(copilot_ctx)
if handler is None:
- run_result = _merge_run_verdicts_if_requested_output_exists(
- run_criteria,
- requested_output_verdicts,
- structural_unfired_criterion_ids=run_structural_unfired_ids,
+ run_result = (
+ _merge_run_verdicts(
+ run_criteria,
+ requested_output_verdicts,
+ validation_classification_verdicts,
+ structural_unfired_criterion_ids=run_structural_unfired_ids,
+ )
+ if requested_output_verdicts or validation_classification_verdicts
+ else None
)
if run_result is None:
return None
else:
- judgeable_result = await evaluate_completion_criteria(judgeable_run_criteria, snapshot, handler)
- judgeable_result = _apply_present_value_upgrades(judgeable_result, judgeable_run_criteria, snapshot)
+ judgeable_result = await evaluate_completion_criteria(
+ remaining_judgeable_run_criteria,
+ snapshot,
+ handler,
+ )
+ judgeable_result = _apply_present_value_upgrades(
+ judgeable_result,
+ remaining_judgeable_run_criteria,
+ snapshot,
+ )
if judgeable_result.status != "evaluated":
- run_result = (
- _merge_run_verdicts_if_requested_output_exists(
+ deterministic_result = (
+ _merge_run_verdicts(
run_criteria,
requested_output_verdicts,
+ validation_classification_verdicts,
structural_unfired_criterion_ids=run_structural_unfired_ids,
)
- or judgeable_result
+ if requested_output_verdicts or validation_classification_verdicts
+ else None
)
+ run_result = deterministic_result or judgeable_result
else:
run_result = _merge_run_verdicts(
run_criteria,
requested_output_verdicts,
+ validation_classification_verdicts,
judgeable_result.verdicts,
structural_unfired_criterion_ids=run_structural_unfired_ids,
)
@@ -668,6 +698,9 @@ def _apply_present_value_upgrades(
upgrades.update(
{verdict.criterion_id: verdict for verdict in grade_structured_record_criteria(run_criteria, snapshot)}
)
+ upgrades.update(
+ {verdict.criterion_id: verdict for verdict in grade_validation_classification_criteria(run_criteria, snapshot)}
+ )
if include_terminal_goal_records:
upgrades.update(
{
@@ -831,6 +864,8 @@ def _deterministic_run_verification_result(
deterministic_by_id[verdict.criterion_id] = verdict
for verdict in grade_registered_download_criteria(run_criteria, snapshot):
deterministic_by_id[verdict.criterion_id] = verdict
+ for verdict in grade_validation_classification_criteria(run_criteria, snapshot):
+ deterministic_by_id[verdict.criterion_id] = verdict
for verdict in semantic_verdicts:
deterministic_by_id[verdict.criterion_id] = verdict
diff --git a/skyvern/forge/sdk/copilot/tools/run_execution.py b/skyvern/forge/sdk/copilot/tools/run_execution.py
index 944ada7da..d9e85a2db 100644
--- a/skyvern/forge/sdk/copilot/tools/run_execution.py
+++ b/skyvern/forge/sdk/copilot/tools/run_execution.py
@@ -1639,6 +1639,7 @@ async def _run_blocks_and_collect_debug(
if run_ok and all(r.get("status") == "completed" for r in results):
for label, output in block_outputs_by_label.items():
ctx.verified_block_outputs[label] = output
+ ctx.verified_terminal_block_outputs = dict(block_outputs_by_label)
existing_prefix = list(getattr(ctx, "verified_prefix_labels", []) or [])
existing_set = set(existing_prefix)
for label in labels_to_execute:
@@ -2021,6 +2022,19 @@ def _record_run_blocks_result(
run_ok = bool(result.get("ok", False))
data = result.get("data")
run_id = data.get("workflow_run_id") if isinstance(data, dict) else None
+ if run_ok and isinstance(data, dict):
+ terminal_outputs: dict[str, Any] = {}
+ for block in data.get("blocks") or []:
+ if not isinstance(block, dict):
+ continue
+ label = block.get("label")
+ output = block.get("extracted_data")
+ if isinstance(label, str) and isinstance(output, dict) and output:
+ terminal_outputs[label] = output
+ if terminal_outputs:
+ # Prefer the final run-result extracted_data for terminal replies;
+ # it is the same persisted run evidence completion verification saw.
+ copilot_ctx.verified_terminal_block_outputs = terminal_outputs
copilot_ctx.completion_verification_result = completion_verification
record_completion_verification(copilot_ctx, completion_verification)
_record_adjudication_on_turn_state(copilot_ctx, completion_verification)
diff --git a/tests/unit/test_copilot_agent_blocker_renderer.py b/tests/unit/test_copilot_agent_blocker_renderer.py
index f27a12ac1..c8ffb3b41 100644
--- a/tests/unit/test_copilot_agent_blocker_renderer.py
+++ b/tests/unit/test_copilot_agent_blocker_renderer.py
@@ -10,10 +10,12 @@ from skyvern.forge.sdk.copilot.agent import (
_FALLBACK_BLOCKER_REPLY,
_RAW_SECRET_LEAK_REFUSAL,
_VERIFIED_WORKFLOW_SUCCESS_REPLY,
+ _build_goal_satisfied_exit_result,
_build_output_policy_blocked_result,
_build_turn_halt_exit_result,
_finalize_result_with_blocker_override,
_render_blocker_reply,
+ _verified_workflow_success_reply,
)
from skyvern.forge.sdk.copilot.blocker_signal import (
_LEAK_DENY_TOKENS,
@@ -25,7 +27,7 @@ from skyvern.forge.sdk.copilot.blocker_signal import (
from skyvern.forge.sdk.copilot.completion_verification import CompletionVerificationResult, CriterionVerdict
from skyvern.forge.sdk.copilot.context import AgentResult, CopilotContext
from skyvern.forge.sdk.copilot.output_policy import CopilotOutputKind, OutputPolicyReason, OutputPolicyVerdict
-from skyvern.forge.sdk.copilot.request_policy import RequestPolicy
+from skyvern.forge.sdk.copilot.request_policy import CompletionCriterion, RequestPolicy
from skyvern.forge.sdk.copilot.run_outcome import TERMINAL_CHALLENGE_BLOCKER_REASON_CODE, RecordedRunOutcome
from skyvern.forge.sdk.copilot.turn_halt import TurnHalt, TurnHaltKind
@@ -593,6 +595,37 @@ def _fully_satisfied_result() -> CompletionVerificationResult:
)
+def _fully_satisfied_classification_result() -> CompletionVerificationResult:
+ return CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=["c_validation"],
+ verdicts=[
+ CriterionVerdict(
+ criterion_id="c_validation",
+ state="satisfied",
+ reason_code="evidence_confirms",
+ output_path="login_only",
+ grounding_mode="exact_value",
+ has_exact_value=True,
+ )
+ ],
+ )
+
+
+def _classification_policy() -> RequestPolicy:
+ return RequestPolicy(
+ completion_criteria=[
+ CompletionCriterion(
+ id="c_validation",
+ outcome="The run classifies the path as login gated.",
+ kind="validation_classification",
+ classification_output_key="login_only",
+ expected_classification=True,
+ )
+ ]
+ )
+
+
def _seed_verified_outcome(ctx: CopilotContext) -> None:
ctx.completion_verification_result = _fully_satisfied_result()
ctx.last_artifact_health_blocker_reason = None
@@ -640,6 +673,141 @@ def test_verified_outcome_preserve_sets_verified_success_payload() -> None:
assert overridden.narrative_payload["proposalDisposition"] == "review_tested"
+async def test_verified_classification_success_surfaces_terminal_verdict_and_reload_payload() -> None:
+ ctx = _ctx()
+ ctx.stream = None
+ ctx.turn_id = "turn-classified"
+ ctx.last_test_ok = True
+ ctx.last_full_workflow_test_ok = True
+ _seed_verified_outcome(ctx)
+ ctx.request_policy = _classification_policy()
+ ctx.completion_verification_result = _fully_satisfied_classification_result()
+ ctx.verified_block_outputs = {
+ "inspect_access_path": {
+ "login_only": True,
+ "visible_page_path_label": "Start service sign-in gate",
+ "safest_reachable_next_step": "Stop before account-specific setup.",
+ "recommended_next_action": "Ask the user for online account access before continuing.",
+ "evidence_text": "The page says Sign in or register to continue.",
+ }
+ }
+
+ result = await _build_goal_satisfied_exit_result(ctx, global_llm_context=None)
+
+ assert "login-gated" in result.user_response
+ assert "login_only=true" in result.user_response
+ assert "visible_page_path_label=Start service sign-in gate" in result.user_response
+ assert "safest_reachable_next_step=Stop before account-specific setup." in result.user_response
+ assert "recommended_next_action=Ask the user for online account access before continuing." in result.user_response
+ assert "observed_gate_phrase=Sign in or register to continue" in result.user_response
+ assert result.narrative_payload is not None
+ assert result.narrative_payload["terminalMessage"] == result.user_response
+
+
+def test_verified_classification_summary_does_not_display_unobserved_expected_value() -> None:
+ ctx = _ctx()
+ _seed_verified_outcome(ctx)
+ ctx.request_policy = _classification_policy()
+ ctx.completion_verification_result = _fully_satisfied_classification_result()
+ ctx.verified_block_outputs = {}
+
+ response = _verified_workflow_success_reply(ctx)
+
+ assert response == "I created and tested the workflow successfully."
+ assert "login_only=true" not in response
+ assert "login-gated" not in response
+ assert "observed_gate_phrase=Sign in or register to continue" not in response
+
+
+def test_verified_classification_summary_uses_terminal_snapshot_when_frontier_outputs_clear() -> None:
+ ctx = _ctx()
+ _seed_verified_outcome(ctx)
+ ctx.request_policy = _classification_policy()
+ ctx.completion_verification_result = _fully_satisfied_classification_result()
+ ctx.verified_block_outputs = {}
+ ctx.verified_terminal_block_outputs = {
+ "inspect_access_path": {
+ "login_only": True,
+ "visible_page_path_label": "Start, Stop or Move Service",
+ "matched_gate_phrases": ["sign in or register to continue"],
+ }
+ }
+
+ response = _verified_workflow_success_reply(ctx)
+
+ assert "login_only=true" in response
+ assert "login-gated" in response
+ assert "visible_page_path_label=Start, Stop or Move Service" in response
+ assert "observed_gate_phrase=Sign in or register to continue" in response
+
+
+def test_verified_classification_summary_labels_login_gate_from_verified_gate_phrase() -> None:
+ ctx = _ctx()
+ _seed_verified_outcome(ctx)
+ ctx.request_policy = RequestPolicy(completion_criteria=[CompletionCriterion(id="c0", outcome="Reached gate")])
+ ctx.completion_verification_result = _fully_satisfied_result()
+ ctx.verified_terminal_block_outputs = {
+ "inspect_access_path": {
+ "visible_page_path_label": "Start, Stop or Move Service > Start Service",
+ "observed_gate_phrase": "Sign in or register to continue",
+ }
+ }
+
+ response = _verified_workflow_success_reply(ctx)
+
+ assert "visible_page_path_label=Start, Stop or Move Service > Start Service" in response
+ assert "observed_gate_phrase=Sign in or register to continue" in response
+ assert "login-gated" in response
+
+
+def test_verified_classification_summary_requires_completion_verification() -> None:
+ ctx = _ctx()
+ ctx.request_policy = _classification_policy()
+ ctx.verified_block_outputs = {
+ "inspect_access_path": {
+ "login_only": True,
+ "visible_page_path_label": "Start service sign-in gate",
+ "evidence_text": "The page prose says login-gated and login_only=true.",
+ }
+ }
+
+ response = _verified_workflow_success_reply(ctx)
+
+ assert response == _VERIFIED_WORKFLOW_SUCCESS_REPLY
+ assert "login-gated" not in response
+ assert "login_only=true" not in response
+
+
+def test_verified_classification_summary_requires_fully_satisfied_verification() -> None:
+ ctx = _ctx()
+ ctx.request_policy = _classification_policy()
+ ctx.completion_verification_result = CompletionVerificationResult(
+ status="evaluated",
+ criterion_ids=["c_validation"],
+ verdicts=[
+ CriterionVerdict(
+ criterion_id="c_validation",
+ state="unsatisfied",
+ reason_code="evidence_contradicts",
+ output_path="login_only",
+ )
+ ],
+ )
+ ctx.verified_block_outputs = {
+ "inspect_access_path": {
+ "login_only": True,
+ "visible_page_path_label": "Start service sign-in gate",
+ }
+ }
+
+ response = _verified_workflow_success_reply(ctx)
+
+ assert response == _VERIFIED_WORKFLOW_SUCCESS_REPLY
+ assert "login-gated" not in response
+ assert "login_only=true" not in response
+
+
def test_unverified_blocker_still_renders_blocker_text() -> None:
ctx = _ctx()
ctx.completion_verification_result = None
diff --git a/tests/unit/test_copilot_agent_helpers.py b/tests/unit/test_copilot_agent_helpers.py
index 4e88f47e8..ca33b45f2 100644
--- a/tests/unit/test_copilot_agent_helpers.py
+++ b/tests/unit/test_copilot_agent_helpers.py
@@ -3001,6 +3001,44 @@ class TestNativeToolSurface:
class TestRequestPolicyCredentialResolution:
+ @pytest.mark.asyncio
+ async def test_request_policy_classifier_uses_configured_default_timeout_budget(self, monkeypatch) -> None:
+ from skyvern.config import Settings, settings
+ from skyvern.forge.sdk.copilot import request_policy as rp
+
+ default_timeout = Settings.model_fields["COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDS"].default
+ assert default_timeout == 12.0
+
+ observed_timeouts: list[float | None] = []
+ real_wait_for = rp.asyncio.wait_for
+
+ async def recording_wait_for(awaitable, timeout=None):
+ observed_timeouts.append(timeout)
+ return await real_wait_for(awaitable, timeout=timeout)
+
+ monkeypatch.setattr(settings, "COPILOT_REQUEST_POLICY_CLASSIFIER_TIMEOUT_SECONDS", default_timeout)
+ monkeypatch.setattr(rp.time, "monotonic", lambda: 1000.0)
+ monkeypatch.setattr(rp.asyncio, "wait_for", recording_wait_for)
+
+ async def handler(*, prompt: str, prompt_name: str) -> dict[str, object]:
+ return {
+ "credential_input_kind": "none",
+ "completion_contract": "complete when the account page is visible",
+ }
+
+ policy = await _classify_request(
+ user_message=(
+ "Build a workflow for https://example.com/account. complete when the account page is visible"
+ ),
+ workflow_yaml="",
+ chat_history=[],
+ global_llm_context="",
+ handler=handler,
+ )
+
+ assert policy.classifier_status == "success"
+ assert observed_timeouts == [default_timeout]
+
@pytest.mark.asyncio
async def test_request_policy_classifier_timeout_does_not_retry(self, monkeypatch) -> None:
from skyvern.config import settings
@@ -4918,6 +4956,24 @@ class TestResponseTypeClassificationRuleReachesAgent:
assert prompt.index("RESPONSE-TYPE CLASSIFICATION") < prompt.index("**Option 1: Reply to the user**")
+class TestValidationClassificationOutputContractRule:
+ def test_build_system_prompt_requires_exact_top_level_classification_contract_keys(self) -> None:
+ from skyvern.forge.sdk.copilot.agent import _build_system_prompt
+
+ prompt = _build_system_prompt(tool_usage_guide="", security_rules="")
+
+ assert "validation_classification_output_contracts" in prompt
+ assert "explicitly `return` a dict" in prompt
+ assert "Do not rely on schema defaults or exposed top-level locals" in prompt
+ assert "exact `return_key` as a top-level key" in prompt
+ assert "goal_value_paths" in prompt
+ assert "extraction_schema" in prompt
+ assert "reject code-only output blocks without that metadata" in prompt
+ assert "observed_gate_phrase" in prompt
+ assert "Synonyms such as `path_login_only`, `classification`, or prose in `evidence_text`" in prompt
+ assert "do not replace the required key" in prompt
+
+
class TestCopilotConfig:
def test_system_prompt_uses_custom_security_rules(self) -> None:
prompt = agent_module._build_system_prompt(
diff --git a/tests/unit/test_copilot_completion_verification.py b/tests/unit/test_copilot_completion_verification.py
index 79188a459..748448323 100644
--- a/tests/unit/test_copilot_completion_verification.py
+++ b/tests/unit/test_copilot_completion_verification.py
@@ -16,7 +16,10 @@ from skyvern.forge.sdk.copilot.agent import (
_verified_workflow_or_none,
)
from skyvern.forge.sdk.copilot.completion_criteria_store import criteria_from_json, criteria_to_json
-from skyvern.forge.sdk.copilot.completion_output_grounding import grade_requested_output_criteria
+from skyvern.forge.sdk.copilot.completion_output_grounding import (
+ grade_requested_output_criteria,
+ split_requested_output_criteria,
+)
from skyvern.forge.sdk.copilot.completion_verification import (
REGISTERED_DOWNLOAD_COMPLETION_CRITERION_ID,
CompletionVerificationResult,
@@ -33,6 +36,7 @@ from skyvern.forge.sdk.copilot.completion_verification import (
grade_registered_download_criteria,
grade_structured_record_criteria,
grade_terminal_goal_record_criteria,
+ grade_validation_classification_criteria,
registered_download_completion_criterion,
run_plane_all_no_evidence,
structural_unfired_contingent_criterion_ids,
@@ -101,6 +105,8 @@ def _criterion(
deliverable_kind: str | None = None,
expected_output_value: str | None = None,
expected_output_shape: str | None = None,
+ classification_output_key: str | None = None,
+ expected_classification: str | bool | None = None,
) -> CompletionCriterion:
return CompletionCriterion(
id=cid,
@@ -114,6 +120,8 @@ def _criterion(
deliverable_kind=deliverable_kind, # type: ignore[arg-type]
expected_output_value=expected_output_value,
expected_output_shape=expected_output_shape, # type: ignore[arg-type]
+ classification_output_key=classification_output_key,
+ expected_classification=expected_classification,
)
@@ -1289,6 +1297,258 @@ def test_terminal_goal_record_does_not_take_literal_criteria_from_present_value(
assert _satisfied_criterion_ids(grade_present_value_criteria(criteria, snapshot)) == {"c0"}
+def _validation_classification_criterion(
+ expected_classification: str | bool = "login_gated",
+ *,
+ key: str = "path_classification",
+) -> CompletionCriterion:
+ return _criterion(
+ "c_validation",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ classification_output_key=key,
+ expected_classification=expected_classification,
+ )
+
+
+def test_validation_classification_grader_credits_matching_string_value() -> None:
+ criterion = _validation_classification_criterion("login_gated")
+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": {"path_classification": "login_gated"}})
+
+ assert grade_validation_classification_criteria([criterion], snapshot) == [
+ CriterionVerdict(
+ criterion_id="c_validation",
+ state="satisfied",
+ reason_code="evidence_confirms",
+ evidence_ref="block_outputs:classify_path.path_classification",
+ output_path="path_classification",
+ grounding_mode="exact_value",
+ has_exact_value=True,
+ )
+ ]
+
+
+def test_validation_classification_grader_credits_matching_boolean_value() -> None:
+ criterion = _validation_classification_criterion(True, key="login_gated")
+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": {"login_gated": True}})
+
+ assert _satisfied_criterion_ids(grade_validation_classification_criteria([criterion], snapshot)) == {"c_validation"}
+
+
+def test_validation_classification_grader_credits_repeated_matching_boolean_values() -> None:
+ criterion = _validation_classification_criterion(True, key="login_only")
+ snapshot = RunEvidenceSnapshot(
+ block_outputs={
+ "classify_path": {"login_only": True},
+ "classify_path_output": {"login_only": True},
+ }
+ )
+
+ assert grade_validation_classification_criteria([criterion], snapshot) == [
+ CriterionVerdict(
+ criterion_id="c_validation",
+ state="satisfied",
+ reason_code="evidence_confirms",
+ evidence_ref="block_outputs:classify_path.login_only",
+ output_path="login_only",
+ grounding_mode="exact_value",
+ has_exact_value=True,
+ )
+ ]
+
+
+ "criterion",
+ [
+ _criterion(
+ "c_validation",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ expected_classification="login_gated",
+ ),
+ _criterion(
+ "c_validation",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ classification_output_key="path_classification",
+ ),
+ ],
+)
+def test_validation_classification_grader_fails_closed_for_incomplete_contract(
+ criterion: CompletionCriterion,
+) -> None:
+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": {"path_classification": "login_gated"}})
+
+ verdicts = grade_validation_classification_criteria([criterion], snapshot)
+
+ assert len(verdicts) == 1
+ assert verdicts[0].criterion_id == "c_validation"
+ assert verdicts[0].state == "unsatisfied"
+ assert verdicts[0].reason_code == "no_evidence"
+ assert verdicts[0].missing_evidence == "incomplete typed classification contract"
+
+
+ ("payload", "reason_code"),
+ [
+ ({}, "no_evidence"),
+ ({"path_classification": None}, "no_evidence"),
+ ({"path_classification": ""}, "no_evidence"),
+ ({"path_classification": []}, "no_evidence"),
+ ({"path_classification": {}}, "no_evidence"),
+ ({"path_classification": "public"}, "evidence_contradicts"),
+ ({"path_classification": ["login_gated"]}, "evidence_contradicts"),
+ ({"path_classification": True}, "evidence_contradicts"),
+ ({"success": True}, "no_evidence"),
+ ({"evidence_text": "The path is login_gated."}, "no_evidence"),
+ ],
+)
+def test_validation_classification_grader_fails_closed_for_non_matching_values(
+ payload: Any,
+ reason_code: str,
+) -> None:
+ criterion = _validation_classification_criterion("login_gated")
+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": payload})
+
+ verdicts = grade_validation_classification_criteria([criterion], snapshot)
+
+ assert len(verdicts) == 1
+ assert verdicts[0].criterion_id == "c_validation"
+ assert verdicts[0].state == "unsatisfied"
+ assert verdicts[0].reason_code == reason_code
+
+
+def test_validation_classification_grader_credits_repeated_matching_string_values() -> None:
+ criterion = _validation_classification_criterion("login_gated")
+ snapshot = RunEvidenceSnapshot(
+ block_outputs={
+ "classify_path_a": {"path_classification": "login_gated"},
+ "classify_path_b": {"path_classification": "login_gated"},
+ }
+ )
+
+ verdicts = grade_validation_classification_criteria([criterion], snapshot)
+
+ assert len(verdicts) == 1
+ assert verdicts[0] == CriterionVerdict(
+ criterion_id="c_validation",
+ state="satisfied",
+ reason_code="evidence_confirms",
+ evidence_ref="block_outputs:classify_path_a.path_classification",
+ output_path="path_classification",
+ grounding_mode="exact_value",
+ has_exact_value=True,
+ )
+
+
+def test_validation_classification_grader_fails_closed_for_conflicting_candidates() -> None:
+ criterion = _validation_classification_criterion("login_gated")
+ snapshot = RunEvidenceSnapshot(
+ block_outputs={
+ "classify_path_a": {"path_classification": "login_gated"},
+ "classify_path_b": {"path_classification": "public"},
+ }
+ )
+
+ verdicts = grade_validation_classification_criteria([criterion], snapshot)
+
+ assert len(verdicts) == 1
+ assert verdicts[0].state == "unsatisfied"
+ assert verdicts[0].reason_code == "evidence_contradicts"
+
+
+ "mixed_value",
+ [
+ None,
+ "",
+ [],
+ {},
+ "true",
+ False,
+ ],
+)
+def test_validation_classification_grader_fails_closed_for_mixed_repeated_boolean_candidates(
+ mixed_value: Any,
+) -> None:
+ criterion = _validation_classification_criterion(True, key="login_only")
+ snapshot = RunEvidenceSnapshot(
+ block_outputs={
+ "classify_path": {"login_only": True},
+ "classify_path_output": {"login_only": mixed_value},
+ }
+ )
+
+ verdicts = grade_validation_classification_criteria([criterion], snapshot)
+
+ assert len(verdicts) == 1
+ assert verdicts[0].state == "unsatisfied"
+ assert verdicts[0].reason_code == "evidence_contradicts"
+
+
+def test_validation_classification_grader_rejects_same_named_scalar_block_label() -> None:
+ criterion = _validation_classification_criterion("login_gated")
+ snapshot = RunEvidenceSnapshot(block_outputs={"path_classification": "login_gated"})
+
+ verdicts = grade_validation_classification_criteria([criterion], snapshot)
+
+ assert len(verdicts) == 1
+ assert verdicts[0].state == "unsatisfied"
+ assert verdicts[0].reason_code == "no_evidence"
+
+
+def test_validation_classification_is_not_requested_output_criterion() -> None:
+ criterion = _criterion(
+ "c_validation",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ output_path="output.path_classification",
+ expected_output_value="login_gated",
+ )
+
+ requested, remaining = split_requested_output_criteria([criterion])
+
+ assert requested == []
+ assert remaining == [criterion]
+
+
+def test_validation_classification_grader_does_not_cross_credit_outcome_or_terminal_action() -> None:
+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": {"path_classification": "login_gated"}})
+
+ assert (
+ grade_validation_classification_criteria(
+ [
+ _criterion(
+ "c_outcome",
+ "The run classifies whether the path is login gated.",
+ classification_output_key="path_classification",
+ expected_classification="login_gated",
+ ),
+ _criterion(
+ "c_terminal",
+ "The run classifies whether the path is login gated.",
+ kind="terminal_action",
+ terminal_action_family="request",
+ classification_output_key="path_classification",
+ expected_classification="login_gated",
+ ),
+ ],
+ snapshot,
+ )
+ == []
+ )
+
+
+def test_validation_classification_grader_repeats_same_verdict_for_same_output() -> None:
+ criterion = _validation_classification_criterion("login_gated")
+ snapshot = RunEvidenceSnapshot(block_outputs={"classify_path": {"path_classification": "login_gated"}})
+
+ first = grade_validation_classification_criteria([criterion], snapshot)
+ second = grade_validation_classification_criteria([criterion], snapshot)
+
+ assert first == second
+
+
def test_fallback_floor_accepts_validation_review_evidence() -> None:
snapshot = RunEvidenceSnapshot(block_outputs={"submit_request": _validation_review_payload()})
@@ -2054,6 +2314,30 @@ def _validation_review_output_result(**payload_overrides: Any) -> dict:
}
+def _validation_classification_output_result(value: Any) -> dict:
+ return _validation_classification_payload_result({"path_classification": value})
+
+
+def _validation_classification_payload_result(payload: dict[str, Any]) -> dict:
+ return {
+ "ok": True,
+ "data": {
+ "workflow_run_id": "wr_validation_classification",
+ "overall_status": "completed",
+ "executed_block_labels": ["classify_path"],
+ "current_url": "https://example.test/login",
+ "blocks": [
+ {
+ "label": "classify_path",
+ "block_type": "CODE",
+ "status": "completed",
+ "extracted_data": payload,
+ }
+ ],
+ },
+ }
+
+
def _live_validation_review_output_result(**payload_overrides: Any) -> dict:
return {
"ok": True,
@@ -2347,6 +2631,249 @@ async def test_non_fallback_judge_confirmed_run_still_fires_barrier(monkeypatch:
assert verified_goal_satisfied_context(ctx) is True
+async def test_validation_classification_block_output_satisfies_without_judge_unknown_veto(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ async def handler(**_: object) -> dict:
+ raise AssertionError("typed validation classification should be graded deterministically")
+
+ _patch_completion_handler(monkeypatch, handler)
+ ctx = _ctx_with_blocks("code")
+ _set_workflow_labels(ctx, "classify_path")
+ ctx.request_policy = RequestPolicy(completion_criteria=[_validation_classification_criterion("login_gated")])
+
+ first = await _maybe_run_completion_verification(
+ ctx,
+ _validation_classification_output_result("login_gated"),
+ time.monotonic(),
+ )
+ second = await _maybe_run_completion_verification(
+ ctx,
+ _validation_classification_output_result("login_gated"),
+ time.monotonic(),
+ )
+
+ assert first is not None
+ assert second is not None
+ assert first.is_fully_satisfied() is True
+ assert second.is_fully_satisfied() is True
+ assert first.verdicts == second.verdicts
+
+
+async def test_validation_classification_duplicate_registered_output_surfaces_satisfy_without_judge(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ async def handler(**_: object) -> dict:
+ raise AssertionError("repeated coherent typed classification evidence should not require judge authority")
+
+ _patch_completion_handler(monkeypatch, handler)
+ ctx = _ctx_with_blocks("code")
+ _set_workflow_labels(ctx, "classify_path")
+ ctx.request_policy = RequestPolicy(
+ completion_criteria=[_validation_classification_criterion(True, key="login_only")]
+ )
+
+ verification = await _maybe_run_completion_verification(
+ ctx,
+ {
+ "ok": True,
+ "data": {
+ "workflow_run_id": "wr_validation_classification",
+ "overall_status": "completed",
+ "executed_block_labels": ["classify_path"],
+ "current_url": "https://example.test/login",
+ "blocks": [
+ {
+ "label": "classify_path",
+ "block_type": "CODE",
+ "status": "completed",
+ "extracted_data": {"login_only": True},
+ }
+ ],
+ "registered_output_parameter_values": [
+ {
+ "workflow_run_id": "wr_validation_classification",
+ "output_parameter_id": "op_login_only",
+ "output_parameter_key": "classify_path_output",
+ "block_label": "classify_path",
+ "block_type": "CODE",
+ "value": {"login_only": True},
+ }
+ ],
+ },
+ },
+ time.monotonic(),
+ )
+
+ assert verification is not None
+ assert verification.is_fully_satisfied() is True
+ assert verification.verdicts == [
+ CriterionVerdict(
+ criterion_id="c_validation",
+ state="satisfied",
+ reason_code="evidence_confirms",
+ evidence_ref="block_outputs:classify_path.login_only",
+ output_path="login_only",
+ grounding_mode="exact_value",
+ has_exact_value=True,
+ )
+ ]
+
+
+async def test_validation_classification_is_not_satisfied_by_fallback_floor_evidence(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ async def handler(**_: object) -> dict:
+ return {
+ "verdicts": [
+ {
+ "criterion_id": "c_validation",
+ "satisfied": False,
+ "reason_code": "no_evidence",
+ "missing_evidence": "matching classification output at path_classification",
+ }
+ ]
+ }
+
+ _patch_completion_handler(monkeypatch, handler)
+ ctx = _ctx_with_blocks("code")
+ _set_workflow_labels(ctx, "submit_request")
+ ctx.request_policy = RequestPolicy(
+ completion_criteria=[
+ _validation_classification_criterion("login_gated"),
+ *build_classifier_fallback_floor([]),
+ ]
+ )
+
+ verification = await _maybe_run_completion_verification(
+ ctx,
+ _validation_review_output_result(),
+ time.monotonic(),
+ )
+
+ assert verification is not None
+ assert verification.is_fully_satisfied() is False
+ assert "c_validation" in {verdict.criterion_id for verdict in verification.verdicts if not verdict.satisfied}
+
+
+ "payload",
+ [
+ {},
+ {"evidence_text": "The path is login_gated."},
+ {"success": True},
+ ],
+)
+async def test_validation_classification_missing_or_prose_only_evidence_cannot_be_judge_approved(
+ monkeypatch: pytest.MonkeyPatch,
+ payload: dict[str, Any],
+) -> None:
+ calls = 0
+
+ async def handler(**_: object) -> dict:
+ nonlocal calls
+ calls += 1
+ return {"verdicts": [{"criterion_id": "c_validation", "satisfied": True, "reason_code": "evidence_confirms"}]}
+
+ _patch_completion_handler(monkeypatch, handler)
+ ctx = _ctx_with_blocks("code")
+ _set_workflow_labels(ctx, "classify_path")
+ ctx.request_policy = RequestPolicy(completion_criteria=[_validation_classification_criterion("login_gated")])
+
+ verification = await _maybe_run_completion_verification(
+ ctx,
+ _validation_classification_payload_result(payload),
+ time.monotonic(),
+ )
+
+ assert calls == 0
+ assert verification is not None
+ assert verification.is_fully_satisfied() is False
+ assert verification.verdicts == [
+ CriterionVerdict(
+ criterion_id="c_validation",
+ state="unsatisfied",
+ reason_code="no_evidence",
+ output_path="path_classification",
+ grounding_mode="exact_value",
+ has_exact_value=True,
+ missing_evidence="missing classification output key path_classification",
+ )
+ ]
+
+
+ "criterion",
+ [
+ _criterion(
+ "c_validation",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ expected_classification="login_gated",
+ ),
+ _criterion(
+ "c_validation",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ classification_output_key="path_classification",
+ ),
+ _criterion(
+ "c_validation",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ output_path="output.path_classification",
+ expected_output_value="login_gated",
+ ),
+ ],
+)
+async def test_validation_classification_incomplete_contract_cannot_be_judge_approved(
+ monkeypatch: pytest.MonkeyPatch,
+ criterion: CompletionCriterion,
+) -> None:
+ calls = 0
+
+ async def handler(**_: object) -> dict:
+ nonlocal calls
+ calls += 1
+ return {
+ "verdicts": [
+ {"criterion_id": "c_validation", "satisfied": True, "reason_code": "evidence_confirms"},
+ {"criterion_id": "c_other", "satisfied": True, "reason_code": "evidence_confirms"},
+ ]
+ }
+
+ _patch_completion_handler(monkeypatch, handler)
+ ctx = _ctx_with_blocks("code")
+ _set_workflow_labels(ctx, "classify_path")
+ ctx.code_artifact_metadata = _metadata_for_requested_paths("path_classification")
+ ctx.request_policy = RequestPolicy(
+ completion_criteria=[
+ criterion,
+ _criterion("c_other", "The run produced ordinary page evidence."),
+ ]
+ )
+
+ verification = await _maybe_run_completion_verification(
+ ctx,
+ _validation_classification_output_result("login_gated"),
+ time.monotonic(),
+ )
+
+ assert calls == 1
+ assert verification is not None
+ assert verification.is_fully_satisfied() is False
+ verdict_by_id = {verdict.criterion_id: verdict for verdict in verification.verdicts}
+ assert verdict_by_id["c_validation"].state == "unsatisfied"
+ assert verdict_by_id["c_validation"].reason_code == "no_evidence"
+ assert verdict_by_id["c_validation"].missing_evidence == "incomplete typed classification contract"
+ assert verdict_by_id["c_other"].satisfied is True
+
+
@pytest.mark.asyncio
async def test_classifier_fallback_record_contradiction_still_surfaces(monkeypatch: pytest.MonkeyPatch) -> None:
async def handler(**_: object) -> dict:
@@ -2930,6 +3457,142 @@ async def test_page_observation_verification_recognizes_budgeted_outcome(
assert "SKU-12345 is present in the cart" in seen_prompt["prompt"]
+async def test_page_observation_validation_classification_cannot_be_judge_approved(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ handler_calls = 0
+
+ async def handler(**_: object) -> dict:
+ nonlocal handler_calls
+ handler_calls += 1
+ return {
+ "verdicts": [
+ {"criterion_id": "c_validation", "satisfied": True, "reason_code": "evidence_confirms"},
+ {"criterion_id": "c_page", "satisfied": True, "reason_code": "evidence_confirms"},
+ ]
+ }
+
+ _patch_completion_handler(monkeypatch, handler)
+ ctx = _run_ctx()
+ ctx.request_policy = RequestPolicy(
+ completion_criteria=[
+ _validation_classification_criterion("login_gated"),
+ _criterion("c_page", "The page observation confirms the path was inspected."),
+ ]
+ )
+ ctx.last_test_ok = False
+ ctx.last_run_blocks_workflow_run_id = "wr_cancel"
+ ctx.copilot_run_start_monotonic = time.monotonic()
+ _record_composition_page_observation(
+ ctx,
+ source_tool="evaluate",
+ url="https://example.test/login",
+ title="Login",
+ observed_data={"evidence_text": "The path is login_gated."},
+ )
+
+ result = await _maybe_run_completion_verification_from_page_observation(
+ ctx,
+ url="https://example.test/login",
+ title="Login",
+ observed_data={"evidence_text": "The path is login_gated."},
+ )
+
+ assert handler_calls == 1
+ assert result is not None
+ assert result.is_fully_satisfied() is False
+ verdict_by_id = {verdict.criterion_id: verdict for verdict in result.verdicts}
+ assert verdict_by_id["c_validation"] == CriterionVerdict(
+ criterion_id="c_validation",
+ state="unsatisfied",
+ reason_code="no_evidence",
+ output_path="path_classification",
+ grounding_mode="exact_value",
+ has_exact_value=True,
+ missing_evidence="missing classification output key path_classification",
+ )
+ assert verdict_by_id["c_page"].satisfied is True
+
+
+ "criterion",
+ [
+ _criterion(
+ "c_validation",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ expected_classification="login_gated",
+ ),
+ _criterion(
+ "c_validation",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ classification_output_key="path_classification",
+ ),
+ _criterion(
+ "c_validation",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ output_path="output.path_classification",
+ expected_output_value="login_gated",
+ ),
+ ],
+)
+async def test_page_observation_validation_classification_incomplete_contract_cannot_be_judge_approved(
+ monkeypatch: pytest.MonkeyPatch,
+ criterion: CompletionCriterion,
+) -> None:
+ handler_calls = 0
+
+ async def handler(**_: object) -> dict:
+ nonlocal handler_calls
+ handler_calls += 1
+ return {
+ "verdicts": [
+ {"criterion_id": "c_validation", "satisfied": True, "reason_code": "evidence_confirms"},
+ {"criterion_id": "c_page", "satisfied": True, "reason_code": "evidence_confirms"},
+ ]
+ }
+
+ _patch_completion_handler(monkeypatch, handler)
+ ctx = _run_ctx()
+ ctx.code_artifact_metadata = _metadata_for_requested_paths("path_classification")
+ ctx.request_policy = RequestPolicy(
+ completion_criteria=[
+ criterion,
+ _criterion("c_page", "The page observation confirms the path was inspected."),
+ ]
+ )
+ ctx.last_test_ok = False
+ ctx.last_run_blocks_workflow_run_id = "wr_cancel"
+ ctx.copilot_run_start_monotonic = time.monotonic()
+ _record_composition_page_observation(
+ ctx,
+ source_tool="evaluate",
+ url="https://example.test/login",
+ title="Login",
+ observed_data={"path_classification": "login_gated"},
+ )
+
+ result = await _maybe_run_completion_verification_from_page_observation(
+ ctx,
+ url="https://example.test/login",
+ title="Login",
+ observed_data={"path_classification": "login_gated"},
+ )
+
+ assert handler_calls == 1
+ assert result is not None
+ assert result.is_fully_satisfied() is False
+ verdict_by_id = {verdict.criterion_id: verdict for verdict in result.verdicts}
+ assert verdict_by_id["c_validation"].state == "unsatisfied"
+ assert verdict_by_id["c_validation"].reason_code == "no_evidence"
+ assert verdict_by_id["c_validation"].missing_evidence == "incomplete typed classification contract"
+ assert verdict_by_id["c_page"].satisfied is True
+
+
@pytest.mark.asyncio
async def test_page_observation_verification_does_not_apply_terminal_goal_record_upgrade(
monkeypatch: pytest.MonkeyPatch,
diff --git a/tests/unit/test_copilot_request_policy_completion_criteria.py b/tests/unit/test_copilot_request_policy_completion_criteria.py
index 2ac2c71f3..d1bef12f1 100644
--- a/tests/unit/test_copilot_request_policy_completion_criteria.py
+++ b/tests/unit/test_copilot_request_policy_completion_criteria.py
@@ -65,6 +65,9 @@ def _criterion(
deliverable_kind: str | None = None,
expected_output_value: str | None = None,
expected_output_shape: str | None = None,
+ kind: str = "outcome",
+ classification_output_key: str | None = None,
+ expected_classification: str | bool | None = None,
) -> CompletionCriterion:
return CompletionCriterion(
id=cid,
@@ -77,6 +80,9 @@ def _criterion(
deliverable_kind=deliverable_kind, # type: ignore[arg-type]
expected_output_value=expected_output_value,
expected_output_shape=expected_output_shape, # type: ignore[arg-type]
+ kind=kind, # type: ignore[arg-type]
+ classification_output_key=classification_output_key,
+ expected_classification=expected_classification,
)
@@ -469,6 +475,163 @@ def test_parse_completion_criteria_accepts_only_declared_expected_output_shapes(
assert invalid.expected_output_shape is None
+def test_parse_completion_criteria_preserves_validation_classification_contract() -> None:
+ parsed = _parse_completion_criteria(
+ [
+ {
+ "outcome": "The run classifies whether the path is login gated.",
+ "kind": "validation_classification",
+ "classification_output_key": "login_gated",
+ "expected_classification": True,
+ },
+ {
+ "outcome": "The run classifies whether the path is login gated.",
+ "kind": "outcome",
+ "classification_output_key": "login_gated",
+ "expected_classification": True,
+ },
+ ]
+ )
+
+ assert len(parsed) == 2
+ assert parsed[0].kind == "validation_classification"
+ assert parsed[0].classification_output_key == "login_gated"
+ assert parsed[0].expected_classification is True
+ assert parsed[1].kind == "outcome"
+
+
+def test_parse_completion_criteria_drops_requested_output_fields_for_validation_classification() -> None:
+ parsed = _parse_completion_criteria(
+ [
+ {
+ "outcome": "The run classifies whether the path is login gated.",
+ "kind": "validation_classification",
+ "output_path": "output.path_classification",
+ "expected_output_value": "login_gated",
+ "expected_output_shape": "status_label",
+ }
+ ]
+ )
+
+ assert len(parsed) == 1
+ assert parsed[0].kind == "validation_classification"
+ assert parsed[0].output_path is None
+ assert parsed[0].expected_output_value is None
+ assert parsed[0].expected_output_shape is None
+
+
+def test_parse_completion_criteria_dedupes_validation_classification_by_target_and_output_key() -> None:
+ parsed = _parse_completion_criteria(
+ [
+ {
+ "outcome": "The run classifies whether the path is login gated.",
+ "kind": "validation_classification",
+ "classification_output_key": "path_classification",
+ "expected_classification": "login_gated",
+ },
+ {
+ "outcome": "Duplicate phrasing should not matter for the same classification target.",
+ "kind": "validation_classification",
+ "classification_output_key": "path_classification",
+ "expected_classification": "login_gated",
+ },
+ {
+ "outcome": "A different expected classification stays distinct.",
+ "kind": "validation_classification",
+ "classification_output_key": "path_classification",
+ "expected_classification": "public",
+ },
+ ]
+ )
+
+ assert [(criterion.classification_output_key, criterion.expected_classification) for criterion in parsed] == [
+ ("path_classification", "login_gated"),
+ ("path_classification", "public"),
+ ]
+
+
+async def test_request_policy_promotes_login_only_output_to_validation_classification_contract() -> None:
+ policy = await _policy_for_message(
+ (
+ "I need a reusable validation-only workflow that checks whether Riverbend Gas has a public path for "
+ "starting gas service from http://localhost:8900/utility_services/riverbend_gas/. It should look for a "
+ "start, connect, new service, or move-in path without logging in, creating an account, entering identity "
+ "details, entering payment details, accepting terms, or submitting anything. For this eval run, classify "
+ "the safest reachable next step and return a structured summary showing whether a public form exists, "
+ "whether the path is login-only, the visible page/path label, and the recommended next action."
+ ),
+ [
+ {
+ "outcome": "The returned record includes public path exists.",
+ "output_path": "output.public_path_exists",
+ },
+ {
+ "outcome": "The returned record includes safest reachable next step.",
+ "output_path": "output.safest_reachable_next_step",
+ },
+ {
+ "outcome": "The returned record includes login only.",
+ "output_path": "output.login_only",
+ },
+ ],
+ )
+
+ validation_criteria = [
+ criterion for criterion in policy.completion_criteria if criterion.kind == "validation_classification"
+ ]
+ assert len(validation_criteria) == 1
+ assert validation_criteria[0].classification_output_key == "login_only"
+ assert validation_criteria[0].expected_classification is True
+ assert validation_criteria[0].output_path is None
+ assert validation_criteria[0].expected_output_value is None
+ assert not _criteria_for_path(policy, "output.login_only")
+
+
+def test_prompt_summary_includes_validation_classification_output_contract() -> None:
+ policy = RequestPolicy(
+ completion_criteria=[
+ _criterion(
+ "c_login_only",
+ "The run classifies whether the path is login-only.",
+ kind="validation_classification",
+ classification_output_key="login_only",
+ expected_classification=True,
+ )
+ ]
+ )
+
+ summary = policy.prompt_summary()
+
+ assert "validation_classification_output_contracts:" in summary
+ assert "criterion_id: c_login_only" in summary
+ assert "return_key: login_only" in summary
+ assert "expected_value: true" in summary
+ assert "return_location: top_level_block_output" in summary
+
+
+async def test_request_policy_promotes_path_classification_when_classifier_supplies_target() -> None:
+ policy = await _policy_for_message(
+ "Classify whether the service path is login-gated and return path classification.",
+ [
+ {
+ "outcome": "The returned record includes path classification.",
+ "output_path": "output.path_classification",
+ "expected_output_value": "login-gated",
+ }
+ ],
+ )
+
+ assert len(policy.completion_criteria) == 1
+ criterion = policy.completion_criteria[0]
+ assert criterion.kind == "validation_classification"
+ assert criterion.classification_output_key == "path_classification"
+ assert criterion.expected_classification == "login-gated"
+ assert criterion.output_path is None
+ assert criterion.expected_output_value is None
+
+
@pytest.mark.asyncio
async def test_requested_output_canonicalization_preserves_classifier_shape_first() -> None:
policy = await _policy_for_message(
@@ -917,6 +1080,33 @@ def test_active_criteria_rendering_includes_expected_output_shape_as_structured_
]
+def test_active_criteria_rendering_includes_validation_classification_contract() -> None:
+ rendered = _render_active_criteria_for_prompt(
+ [
+ _criterion(
+ "c0",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ classification_output_key="login_gated",
+ expected_classification=True,
+ )
+ ]
+ )
+
+ assert json.loads(rendered) == [
+ {
+ "outcome": "The run classifies whether the path is login gated.",
+ "implicit": False,
+ "method_mandated": False,
+ "level": "run",
+ "kind": "validation_classification",
+ "terminal_action_family": None,
+ "classification_output_key": "login_gated",
+ "expected_classification": True,
+ }
+ ]
+
+
def test_criteria_json_round_trips_contingent_on() -> None:
criteria = (
_criterion(
@@ -961,6 +1151,43 @@ def test_criteria_json_round_trips_expected_output_shape() -> None:
assert restored == criteria
+def test_criteria_json_round_trips_validation_classification_contract() -> None:
+ criteria = (
+ _criterion(
+ "c0",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ classification_output_key="path_classification",
+ expected_classification="login_gated",
+ ),
+ )
+
+ restored = criteria_from_json(criteria_to_json(criteria))
+
+ assert restored == criteria
+
+
+def test_criteria_json_rehydrates_validation_classification_without_requested_output_fields() -> None:
+ criteria = (
+ _criterion(
+ "c0",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ output_path="output.path_classification",
+ expected_output_value="login_gated",
+ expected_output_shape="status_label",
+ ),
+ )
+
+ restored = criteria_from_json(criteria_to_json(criteria))
+
+ assert len(restored) == 1
+ assert restored[0].kind == "validation_classification"
+ assert restored[0].output_path is None
+ assert restored[0].expected_output_value is None
+ assert restored[0].expected_output_shape is None
+
+
def test_request_policy_trace_exposes_requested_output_grounding_contract_without_values() -> None:
policy = RequestPolicy(
completion_criteria=[
@@ -1056,6 +1283,36 @@ def test_reconcile_supersedes_same_output_path_with_different_expected_output_sh
assert decision.criteria == tuple(fresh)
+def test_reconcile_supersedes_validation_classification_with_different_target() -> None:
+ stored = _stored(
+ _criterion(
+ "s0",
+ "The run classifies whether the path is login gated.",
+ kind="validation_classification",
+ classification_output_key="path_classification",
+ expected_classification="login_gated",
+ )
+ )
+ fresh = [
+ _criterion(
+ "c0",
+ "The run classifies whether the path is public.",
+ kind="validation_classification",
+ classification_output_key="path_classification",
+ expected_classification="public",
+ )
+ ]
+
+ decision = reconcile_completion_criteria(
+ StoredCriteriaSnapshot(active=stored, next_epoch=2),
+ fresh,
+ actionable=True,
+ )
+
+ assert decision.action == "create"
+ assert decision.criteria == tuple(fresh)
+
+
def test_criteria_json_round_trips_contingent_antecedent_output_path() -> None:
criteria = (
_criterion(
diff --git a/tests/unit/test_copilot_request_policy_prompt.py b/tests/unit/test_copilot_request_policy_prompt.py
index 21068227a..1451de44c 100644
--- a/tests/unit/test_copilot_request_policy_prompt.py
+++ b/tests/unit/test_copilot_request_policy_prompt.py
@@ -90,12 +90,13 @@ class TestRequestPolicyPromptStructure:
assert (
"{outcome, contingent_on, contingent_antecedent_output_path, "
"deliverable_kind, implicit, method_mandated, level, output_path, expected_output_value, "
- "expected_output_shape, kind, terminal_action_family}"
+ "expected_output_shape, kind, terminal_action_family, classification_output_key, expected_classification}"
) in rendered
assert "never hide it in outcome prose" in rendered
assert "reference_code, numeric_identifier, date, address, status_label, money_amount, owner_label" in rendered
- assert "kind=outcome|terminal_action" in rendered
+ assert "kind=outcome|terminal_action|validation_classification" in rendered
assert "terminal_action_family=request|application|form|order|null" in rendered
+ assert "classification_output_key=login_only and expected_classification=true" in rendered
assert 'The only supported non-null value is "registered_download"' in rendered
def test_active_completion_criteria_render_typed_terminal_action_fields(self) -> None: