Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
feat(SKY-11897): typed judgment-evidence grounding — confirm/refute/abstain from independent packet (#7224)
Andrew Neilson6a5226d19728
diff --git a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2 b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2index 3de9fb10b..b7462c12a 100644--- a/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2+++ b/skyvern/forge/prompts/skyvern/workflow-copilot-request-policy.j2@@ -25,13 +25,15 @@ completion_contract:- Set to null when the user does not provide an explicit completion condition.completion_criteria:-- An array of the END-STATE OUTCOMES that must be true for the user's goal to be done. Each item is {outcome, contingent_on, contingent_antecedent_output_path, deliverable_kind, implicit, method_mandated, level, output_path, expected_output_value, expected_output_shape, requested_output_evidence_source, kind, terminal_action_family, classification_output_key, expected_classification}.+- An array of the END-STATE OUTCOMES that must be true for the user's goal to be done. Each item is {outcome, contingent_on, contingent_antecedent_output_path, deliverable_kind, implicit, method_mandated, level, output_path, expected_output_value, expected_output_shape, requested_output_evidence_source, kind, terminal_action_family, classification_output_key, expected_classification, judgment_predicate, judgment_polarity_when_holds}.- outcome: a short statement of an end state the workflow must achieve (an END, not a step). Extract the goal as outcomes the user wants to be true, not the recipe they describe.- contingent_on: null for ordinary criteria. Use a short antecedent string only when this criterion should be graded only if that antecedent happens, for example {"contingent_on": "the provider site blocks online submission", "contingent_antecedent_output_path": "output.blocker", "outcome": "the blocker is reported to the user"}. The antecedent goes in contingent_on; the consequent to grade goes in outcome. Do not encode if/then criteria only in prose.- contingent_antecedent_output_path: null unless the antecedent is represented by a structured run output field. The only valid form is output.<field> (for example output.blocker). Do not emit transcript paths, selectors, regexes, JSONPath, screenshots, nested paths, or prose conditions here.- deliverable_kind: null unless this criterion's requested output is satisfied by execution-owned browser-download registration rather than a user-authored extraction field. The only supported non-null value is "registered_download".- expected_output_value: null unless this criterion requires an exact structured output value the user explicitly provided. Put that value here as data; never hide it in outcome prose. For a goal-judgment field whose expected answer is a boolean (e.g. selected_highest_priority), provide the JSON true or false here.- expected_output_shape: null unless this criterion requires a generated/provider structured output value whose exact value is not known before the run. Use only one of: reference_code, numeric_identifier, date, address, status_label, money_amount, owner_label, goal_judgment_boolean. goal_judgment_boolean declares that the field is an artifact-computed true/false judgment; the typed boolean expected_output_value is the deterministic backstop. Do not use this as a non-empty-value proxy; leave null when the field's shape is not one of those declared types.+- judgment_predicate: null unless expected_output_shape=goal_judgment_boolean and requested_output_evidence_source=independent_run_evidence. Use only the closed-vocabulary value login_gate_blocks_target, and only when the requested judgment is whether login gating blocks the target path.+- judgment_polarity_when_holds: null unless judgment_predicate is non-null. Set it to the boolean output value that means the predicate holds; for login_gate_blocks_target this is true.- requested_output_evidence_source: runtime_output|independent_run_evidence|registered_output_parameter|registered_artifact_content. Use runtime_output for site-produced values the code block directly reads from the page or browser state, such as dates, amounts, statuses, identifiers, names, or downloaded-file descriptors. Use independent_run_evidence only for artifact-computed judgments, selections, superlatives, rankings, validation booleans, or "best/correct/cheapest/highest" claims that cannot prove themselves by returning their own answer. Use registered_output_parameter or registered_artifact_content only when the requested datum must be proven by execution-owned registered output parameters or registered artifact content. Do not choose this from output_path names, fixture names, domain keywords, or prose labels; choose it from the user's requested evidence contract.- When a selection, priority, ranking, or superlative criterion governs a returned requested-output field, set output_path for that returned field and requested_output_evidence_source=independent_run_evidence; the selected value cannot prove its own correctness.- Example: if the user asks to return the selected or highest-priority document name, emit the criterion with output_path=output.document_name and requested_output_evidence_source=independent_run_evidence.@@ -40,6 +42,7 @@ completion_criteria:- classification_output_key: null unless kind=validation_classification. Set it to the exact top-level block output field/key that will carry the classification scalar. Do not use JSONPath, regex, transcript paths, screenshots, prose conditions, or nested paths.- expected_classification: null unless kind=validation_classification. Set it to the exact expected classification target as a boolean or canonical enum/string value. Do not hide this target in outcome prose.- For validation-only utility/path checks whose goal is to classify whether the reachable path is login-only/login-gated, prefer kind=validation_classification with classification_output_key=login_only and expected_classification=true. Do not represent that classification target as a requested-output criterion such as output.login_only.+- For validation-only utility/path checks whose goal is to return whether the target path is blocked by a login gate, emit this as a requested-output judgment with kind=outcome, output_path=output.login_gate_blocks_target, expected_output_shape=goal_judgment_boolean, requested_output_evidence_source=independent_run_evidence, judgment_predicate=login_gate_blocks_target, and judgment_polarity_when_holds=true. Do not use kind=validation_classification for this returned field.- Capture ALL distinct outcomes the goal requires, including implicit constraints the user did not state as a step (for example "added exactly once / not duplicated", "the item is present in the cart"). Mark such items implicit=true.- A prescribed HOW step is NOT a criterion. "search for X", "open the result lightbox", "click add to cart", "go to the cart page" are methods/navigation, not outcomes — do not list them. A different or shorter path that reaches the same end state must still satisfy the goal.- Telling/showing/reporting results to the user is NOT a criterion. "tell me what it extracted", "show me the data", "confirm to me it worked" describe the chat reply, which no workflow definition or run can evidence — capture only the underlying workflow outcome (the data is extracted into the run output, the form is submitted) and list nothing for the reporting itself.diff --git a/skyvern/forge/sdk/copilot/code_block_synthesis.py b/skyvern/forge/sdk/copilot/code_block_synthesis.pyindex fc3b0380a..4ba8d5c5e 100644--- a/skyvern/forge/sdk/copilot/code_block_synthesis.py+++ b/skyvern/forge/sdk/copilot/code_block_synthesis.py@@ -1073,6 +1073,8 @@ def build_artifact_metadata_skeleton("text": _FILL_CRITERION_TEXT,"level": "terminal","terminal": True,+ "judgment_predicate": None,+ "judgment_polarity_when_holds": None,}],"terminal_verifier_expectations": [diff --git a/skyvern/forge/sdk/copilot/completion_criteria_store.py b/skyvern/forge/sdk/copilot/completion_criteria_store.pyindex 230dd15b7..840c9c890 100644--- a/skyvern/forge/sdk/copilot/completion_criteria_store.py+++ b/skyvern/forge/sdk/copilot/completion_criteria_store.py@@ -28,10 +28,12 @@ from skyvern.forge.sdk.copilot.request_policy import (_coerce_expected_classification,_coerce_expected_output_shape,_coerce_expected_output_value,+ _coerce_judgment_truth_condition,_coerce_requested_output_evidence_source,_normalize_contingent_antecedent_output_path,_normalize_deliverable_kind,is_fallback_floor_criterion,+ judgment_truth_condition_key,normalized_criterion_outcome_key,requested_output_path_for_field,typed_expected_output_value_key,@@ -147,6 +149,9 @@ def criteria_to_json(criteria: tuple[CompletionCriterion, ...] | list[Completionitem["requested_output_corroborator"] = Trueif criterion.mint_degrade is not None:item["mint_degrade"] = criterion.mint_degrade+ if criterion.judgment_truth_condition is not None:+ item["judgment_predicate"] = criterion.judgment_truth_condition.predicate+ item["judgment_polarity_when_holds"] = criterion.judgment_truth_condition.polarity_when_holdsitems.append(item)return items@@ -222,6 +227,9 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:mint_degrade="turn_unsatisfiable_fallback"if item.get("mint_degrade") == "turn_unsatisfiable_fallback"else None,+ judgment_truth_condition=_coerce_judgment_truth_condition(+ item.get("judgment_predicate"), item.get("judgment_polarity_when_holds")+ ),))return tuple(criteria)@@ -230,7 +238,10 @@ def criteria_from_json(raw: Any) -> tuple[CompletionCriterion, ...]:def _criterion_reconcile_key(criterion: CompletionCriterion) -> str:contingent_key = criterion.contingent_on or ""contingent_path_key = criterion.contingent_antecedent_output_path or ""- deliverable_kind_key = f"{criterion.deliverable_kind or ''}\x1fmint_degrade:{criterion.mint_degrade or ''}"+ deliverable_kind_key = (+ f"{criterion.deliverable_kind or ''}\x1fmint_degrade:{criterion.mint_degrade or ''}"+ f"\x1fjudgment:{judgment_truth_condition_key(criterion.judgment_truth_condition)}"+ )expected_output_value_key = typed_expected_output_value_key(criterion.expected_output_value)expected_output_shape_key = criterion.expected_output_shape or ""requested_output_evidence_source_key = criterion.requested_output_evidence_sourcediff --git a/skyvern/forge/sdk/copilot/completion_output_grounding.py b/skyvern/forge/sdk/copilot/completion_output_grounding.pyindex 8131af96f..85229a3cb 100644--- a/skyvern/forge/sdk/copilot/completion_output_grounding.py+++ b/skyvern/forge/sdk/copilot/completion_output_grounding.py@@ -7,19 +7,25 @@ import textwrapfrom collections.abc import Mappingfrom typing import Any, Literal, Protocol+import structlogimport yamlfrom skyvern.forge.sdk.copilot.completion_verification import CriterionVerdict, EvidenceSourceKind, RunEvidenceSnapshotfrom skyvern.forge.sdk.copilot.request_policy import (CompletionCriterion,ExpectedOutputValue,+ JudgmentPredicate,+ JudgmentTruthCondition,RequestedOutputEvidenceSource,+ _coerce_judgment_truth_condition,_is_judgment_boolean_criterion,lookup_requested_output_path_alias,schema_output_path_aliases_from_criteria,)from skyvern.utils.yaml_loader import safe_load_no_dates+LOG = structlog.get_logger()+_GOAL_PATH_INDEX_PATTERN = re.compile(r"\[\d+\]")_REQUESTED_OUTPUT_PREFIX = "output."_STRUCTURAL_ABSTENTION_REASON_CODE = "structurally_abstained"@@ -28,6 +34,9 @@ _IGNORED_RUNTIME_FIELD_NAMES = frozenset({"evidence_text"})_INDEPENDENT_EVIDENCE_SOURCES: frozenset[EvidenceSourceKind] = frozenset({"independent_page_evidence", "registered_output_parameter", "registered_artifact_content"})+# A judgment boolean may only be decided by page evidence observed after the run. The other+# "independent" sources are registered by the generated code, so they can carry a fabricated packet.+_JUDGMENT_EVIDENCE_SOURCES: frozenset[EvidenceSourceKind] = frozenset({"independent_page_evidence"})_POST_RUN_PAGE_OBSERVATION_LABEL = "post_run_page_observation"_REGISTERED_ARTIFACT_OBSERVATION_LABEL = "registered_artifact_observation"_MIN_CARRIER_VALUE_CHARS = 4@@ -37,10 +46,15 @@ _PAGE_EVIDENCE_STAMP_KEYS = frozenset({"workflow_run_id", "observed_after_workflclass _GroundingCtx(Protocol):code_artifact_metadata: object+ workflow_verification_evidence: _WorkflowVerificationEvidenceCtx | Nonelast_workflow_yaml: str | Noneworkflow_yaml: str | None+class _WorkflowVerificationEvidenceCtx(Protocol):+ code_artifact_metadata: object++def split_requested_output_criteria(criteria: list[CompletionCriterion],) -> tuple[list[CompletionCriterion], list[CompletionCriterion]]:@@ -49,7 +63,7 @@ def split_requested_output_criteria(for criterion in criteria:if (criterion.output_path- and criterion.level != "definition"+ and (criterion.level != "definition" or _is_judgment_boolean_criterion(criterion))and not criterion.method_mandatedand criterion.kind != "validation_classification"and _normalize_output_path(criterion.output_path)@@ -66,16 +80,20 @@ def grade_requested_output_criteria(snapshot: RunEvidenceSnapshot,) -> list[CriterionVerdict]:requested_path_aliases = schema_output_path_aliases_from_criteria(criteria)- raw_authored_paths_by_label = _authored_output_contract_paths_by_label(copilot_ctx)- authored_paths_by_label = _authored_output_contract_paths_by_label(copilot_ctx, requested_path_aliases)- declared_judgment_output_paths = _producer_declared_judgment_output_paths(copilot_ctx.code_artifact_metadata)+ metadata = _grounding_code_artifact_metadata(copilot_ctx)+ raw_authored_paths_by_label = _authored_output_contract_paths_by_label(copilot_ctx, metadata=metadata)+ authored_paths_by_label = _authored_output_contract_paths_by_label(+ copilot_ctx, requested_path_aliases, metadata=metadata+ )+ declared_judgment_output_paths = _producer_declared_judgment_output_paths(metadata)+ declared_judgment_truth_conditions = _producer_declared_judgment_truth_conditions(metadata)verdicts: list[CriterionVerdict] = []for criterion in criteria:path = _normalize_output_path(criterion.output_path)- # A carrier verdict is already sourced from independent evidence and only returned on a- # positive confirmation, so the early return skips the self-emission veto path that no- # longer applies; abstentions fall through to normal authored-path grading.- carrier_verdict = _carrier_confirmation(criterion, path, snapshot)+ # A carrier verdict is sourced from independent evidence. The judgment arm may return an+ # unsatisfied evidence_contradicts verdict, which must reach the grade loop unswallowed;+ # scalar-carrier abstentions and non-confirmations fall through to normal authored-path grading.+ carrier_verdict = _carrier_confirmation(criterion, path, snapshot, declared_judgment_truth_conditions)if carrier_verdict is not None:verdicts.append(carrier_verdict)continue@@ -92,12 +110,31 @@ def grade_requested_output_criteria())continue+ expected_value = criterion.expected_output_value+ expected_shape = criterion.expected_output_shape+ grounding_mode = _criterion_grounding_mode(criterion)+ requires_declared_independent_evidence = bool(path) and path in declared_judgment_output_paths+ effective_evidence_source: RequestedOutputEvidenceSource = (+ "independent_run_evidence"+ if requires_declared_independent_evidence+ else criterion.requested_output_evidence_source+ )+ judgment_grounding_bars_self_emission = (+ _is_judgment_boolean_criterion(criterion) or requires_declared_independent_evidence+ )+ trace_fields = _criterion_trace_fields(criterion, grounding_mode, effective_evidence_source)independent_labels = {label for label, source in snapshot.block_output_sources.items() if source in _INDEPENDENT_EVIDENCE_SOURCES}- accepted_runtime_outputs = dict(- _accepted_runtime_outputs(snapshot.block_outputs, accepted_labels, independent_labels)+ accepted_runtime_output_items = _accepted_runtime_outputs(+ snapshot.block_outputs, accepted_labels, independent_labels)+ if judgment_grounding_bars_self_emission:+ accepted_runtime_output_items = sorted(+ accepted_runtime_output_items,+ key=lambda item: 0 if item[0] in independent_labels else 1,+ )+ accepted_runtime_outputs = dict(accepted_runtime_output_items)accepted_output_sources = {label: sourcefor label, source in snapshot.block_output_sources.items()@@ -108,19 +145,6 @@ def grade_requested_output_criteria(*(raw_authored_paths_by_label.get(label, set()) for label in accepted_labels))projection_roots = _runtime_projection_roots(accepted_authored_paths | accepted_raw_authored_paths)- expected_value = criterion.expected_output_value- expected_shape = criterion.expected_output_shape- grounding_mode = _criterion_grounding_mode(criterion)- requires_declared_independent_evidence = bool(path) and path in declared_judgment_output_paths- effective_evidence_source: RequestedOutputEvidenceSource = (- "independent_run_evidence"- if requires_declared_independent_evidence- else criterion.requested_output_evidence_source- )- judgment_grounding_bars_self_emission = (- _is_judgment_boolean_criterion(criterion) or requires_declared_independent_evidence- )- trace_fields = _criterion_trace_fields(criterion, grounding_mode, effective_evidence_source)if expected_value is not None:refutation = _independent_evidence_text_refutation(accepted_runtime_outputs,@@ -266,10 +290,13 @@ def grade_requested_output_criteria(def _carrier_confirmation(- criterion: CompletionCriterion, path: str, snapshot: RunEvidenceSnapshot+ criterion: CompletionCriterion,+ path: str,+ snapshot: RunEvidenceSnapshot,+ declared_judgment_truth_conditions: Mapping[str, JudgmentTruthCondition],) -> CriterionVerdict | None:- if _is_judgment_boolean_criterion(criterion):- return None+ if _is_judgment_boolean_criterion(criterion) or path in declared_judgment_truth_conditions:+ return _judgment_carrier_verdict(criterion, path, snapshot, declared_judgment_truth_conditions)value_text = _carrier_scalar_text(criterion.expected_output_value)if value_text is None or len(value_text) < _MIN_CARRIER_VALUE_CHARS:return None@@ -509,6 +536,233 @@ def _producer_declared_judgment_output_paths(metadata: object) -> set[str]:return paths+def _producer_declared_judgment_truth_conditions(metadata: object) -> dict[str, JudgmentTruthCondition]:+ if not isinstance(metadata, Mapping):+ return {}+ conditions: dict[str, JudgmentTruthCondition] = {}+ for artifact in metadata.values():+ if not isinstance(artifact, Mapping):+ continue+ declared_criteria = artifact.get("completion_criteria")+ if not isinstance(declared_criteria, list):+ continue+ for declared in declared_criteria:+ if not isinstance(declared, Mapping):+ continue+ normalized = _normalize_output_path(declared.get("output_path"))+ condition = _coerce_judgment_truth_condition(+ declared.get("judgment_predicate"), declared.get("judgment_polarity_when_holds")+ )+ if condition is None and declared.get("requested_output_evidence_source") == "independent_run_evidence":+ condition = _producer_declared_path_judgment_truth_condition(normalized)+ if normalized and condition is not None:+ conditions.setdefault(normalized, condition)+ return conditions+++def _producer_declared_path_judgment_truth_condition(output_path: str) -> JudgmentTruthCondition | None:+ if output_path == "login_gate_blocks_target":+ return JudgmentTruthCondition(predicate="login_gate_blocks_target", polarity_when_holds=True)+ return None+++def _grounding_code_artifact_metadata(copilot_ctx: _GroundingCtx) -> object:+ metadata = copilot_ctx.code_artifact_metadata+ if _metadata_has_authored_output_contract(metadata):+ return metadata+ evidence = copilot_ctx.workflow_verification_evidence+ if evidence is not None and _metadata_has_authored_output_contract(evidence.code_artifact_metadata):+ return evidence.code_artifact_metadata+ return metadata+++def _metadata_has_authored_output_contract(metadata: object) -> bool:+ if not isinstance(metadata, Mapping):+ return False+ return any(+ isinstance(artifact, Mapping) and bool(_artifact_contract_paths(artifact)) for artifact in metadata.values()+ )+++def _judgment_carrier_verdict(+ criterion: CompletionCriterion,+ path: str,+ snapshot: RunEvidenceSnapshot,+ declared_judgment_truth_conditions: Mapping[str, JudgmentTruthCondition],+) -> CriterionVerdict | None:+ expected_value = _bool_canonical(criterion.expected_output_value)+ if expected_value is None:+ expected_value = _emitted_judgment_boolean(snapshot, path)+ if expected_value is None:+ return None+ condition = criterion.judgment_truth_condition or declared_judgment_truth_conditions.get(path)+ if condition is None:+ return None+ origin = "criterion" if criterion.judgment_truth_condition is not None else "producer_metadata"+ trace_fields = _criterion_trace_fields(+ criterion,+ "judgment_boolean",+ "independent_run_evidence"+ if path in declared_judgment_truth_conditions+ else criterion.requested_output_evidence_source,+ )+ confirmed: CriterionVerdict | None = None+ confirmed_label: str | None = None+ abstention_label: str | None = None+ abstention_source: EvidenceSourceKind | None = None+ for label, payload in snapshot.block_outputs.items():+ source = snapshot.block_output_sources.get(label)+ if source not in _JUDGMENT_EVIDENCE_SOURCES or not isinstance(payload, Mapping):+ continue+ if abstention_label is None:+ abstention_label = label+ abstention_source = source+ predicate_holds = _judgment_packet_predicate_holds(condition.predicate, payload)+ if predicate_holds is None:+ continue+ observed_value = condition.polarity_when_holds if predicate_holds else not condition.polarity_when_holds+ if observed_value != expected_value:+ _log_judgment_evidence_verdict(condition, label, "evidence_contradicts", origin)+ return CriterionVerdict(+ criterion_id=criterion.id,+ state="unsatisfied",+ reason_code="evidence_contradicts",+ evidence_ref=f"block_outputs:{label}",+ missing_evidence=f"independent evidence contradicted output.{path}",+ evidence_source=source,+ **trace_fields,+ )+ if confirmed is None:+ confirmed_label = label+ confirmed = CriterionVerdict(+ criterion_id=criterion.id,+ state="satisfied",+ reason_code="evidence_confirms",+ evidence_ref=f"block_outputs:{label}",+ evidence_source=source,+ **trace_fields,+ )+ if confirmed is not None and confirmed_label is not None:+ _log_judgment_evidence_verdict(condition, confirmed_label, "evidence_confirms", origin)+ if confirmed is None and abstention_label is not None:+ return CriterionVerdict(+ criterion_id=criterion.id,+ state="unsatisfied",+ reason_code=_STRUCTURAL_ABSTENTION_REASON_CODE,+ evidence_ref=f"block_outputs:{abstention_label}",+ missing_evidence=f"independent evidence did not structurally decide output.{path}",+ evidence_source=abstention_source,+ **trace_fields,+ )+ return confirmed+++def _emitted_judgment_boolean(snapshot: RunEvidenceSnapshot, path: str) -> bool | None:+ parts = _path_parts(path)+ observed_values: set[bool] = set()+ for label, payload in snapshot.block_outputs.items():+ if snapshot.block_output_sources.get(label) in _INDEPENDENT_EVIDENCE_SOURCES:+ continue+ values, _source_path = _runtime_path_values(payload, parts, path, set())+ for value in values:+ canonical = _bool_canonical(value)+ if canonical is None:+ return None+ observed_values.add(canonical)+ if len(observed_values) == 1:+ return next(iter(observed_values))+ return None+++def _log_judgment_evidence_verdict(+ condition: JudgmentTruthCondition, packet_label: str, verdict: str, origin: str+) -> None:+ LOG.info(+ "copilot_judgment_evidence_verdict",+ predicate=condition.predicate,+ packet_label=packet_label,+ verdict=verdict,+ origin=origin,+ )+++def _judgment_packet_predicate_holds(predicate: JudgmentPredicate, packet: Mapping[str, object]) -> bool | None:+ if predicate == "login_gate_blocks_target":+ return _login_gate_blocks_target(packet)+ return None+++def _login_gate_blocks_target(packet: Mapping[str, object]) -> bool | None:+ forms = _packet_mapping_list(packet.get("forms"))+ result_containers = _packet_mapping_list(packet.get("result_containers"))+ navigation_targets = _packet_mapping_list(packet.get("navigation_targets"))+ if _has_structural_result_container(result_containers):+ return False+ if _has_enabled_password_field(forms) and (_has_submit_control(forms) or _has_gated_submit_control(packet, forms)):+ return True+ if not forms and not result_containers and not navigation_targets:+ return None+ return None+++def _packet_mapping_list(value: object) -> list[Mapping[str, object]]:+ if not isinstance(value, list):+ return []+ return [item for item in value if isinstance(item, Mapping)]+++def _has_enabled_password_field(forms: list[Mapping[str, object]]) -> bool:+ for form in forms:+ for field in _packet_mapping_list(form.get("fields")):+ if field.get("disabled") is True:+ continue+ field_type = field.get("type")+ if isinstance(field_type, str) and field_type.casefold() == "password":+ return True+ return False+++def _has_submit_control(forms: list[Mapping[str, object]]) -> bool:+ for form in forms:+ for control in _packet_mapping_list(form.get("submit_controls")):+ control_type = control.get("type")+ selector = control.get("selector")+ if isinstance(control_type, str) and control_type.casefold() in {"", "button", "submit"}:+ return True+ if isinstance(selector, str) and selector.strip():+ return True+ return False+++def _has_gated_submit_control(packet: Mapping[str, object], forms: list[Mapping[str, object]]) -> bool:+ for form in forms:+ for control in _packet_mapping_list(form.get("submit_controls")):+ if control.get("disabled") is True:+ return True+ challenge_state = packet.get("challenge_state")+ if isinstance(challenge_state, Mapping):+ if challenge_state.get("gates_submit_controls") is True:+ return True+ if _packet_mapping_list(challenge_state.get("gated_submit_controls")):+ return True+ return False+++def _has_structural_result_container(containers: list[Mapping[str, object]]) -> bool:+ for container in containers:+ row_count = container.get("row_count")+ if isinstance(row_count, int) and row_count > 0:+ return True+ for key in ("selector", "id", "tag", "row_selector"):+ value = container.get(key)+ if isinstance(value, str) and value.strip():+ return True+ rows = container.get("sample_rows")+ if isinstance(rows, list) and rows:+ return True+ return False++def _schema_boolean_output_paths(schema: Mapping[str, Any] | None, prefix: str = "") -> set[str]:if not isinstance(schema, Mapping):return set()@@ -534,9 +788,9 @@ def _schema_boolean_output_paths(schema: Mapping[str, Any] | None, prefix: str =def _authored_output_contract_paths_by_label(- copilot_ctx: _GroundingCtx, aliases: dict[str, str] | None = None+ copilot_ctx: _GroundingCtx, aliases: dict[str, str] | None = None, *, metadata: object | None = None) -> dict[str, set[str]]:- metadata = copilot_ctx.code_artifact_metadata+ metadata = metadata if metadata is not None else copilot_ctx.code_artifact_metadatacode_by_label = _code_blocks_by_label(copilot_ctx)paths_by_label: dict[str, set[str]] = {}if isinstance(metadata, Mapping):diff --git a/skyvern/forge/sdk/copilot/request_policy.py b/skyvern/forge/sdk/copilot/request_policy.pyindex 6fee80510..454e950a6 100644--- a/skyvern/forge/sdk/copilot/request_policy.py+++ b/skyvern/forge/sdk/copilot/request_policy.py@@ -203,6 +203,8 @@ RequestedOutputEvidenceSource = Literal["registered_output_parameter","registered_artifact_content",]+JudgmentPredicate = Literal["login_gate_blocks_target"]+_JUDGMENT_PREDICATES: frozenset[str] = frozenset(get_args(JudgmentPredicate))_CRITERION_KINDS: frozenset[str] = frozenset({"outcome", "terminal_action", "validation_classification"})_TERMINAL_ACTION_FAMILIES: frozenset[str] = frozenset({"request", "application", "form", "order"})_EXPECTED_OUTPUT_SHAPES: frozenset[str] = frozenset(get_args(ExpectedOutputShape))@@ -230,8 +232,11 @@ _OUTPUT_NEGATED_INTENT_PREFIX_RE = re.compile()_OUTPUT_EXPLICIT_FIELD_KEY_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")_OUTPUT_NAMED_FIELD_RE = re.compile(- r"\b(?:(?:requested[-\s]+output|output|return(?:ed)?|final|structured)\s+)?fields?\s+"- r"(?:named|called)\s+([A-Za-z_][A-Za-z0-9_]*)\b",+ r"\b(?:"+ r"(?:(?:requested[-\s]+output|output|return(?:ed)?|final|structured)\s+)?fields?\s+(?:named|called)"+ r"|(?:requested[-\s]+)?output\s+(?:named|called)"+ r"|named\s+(?:requested[-\s]+)?output"+ r")\s+`?([A-Za-z_][A-Za-z0-9_]*)`?\b",re.I,)_OUTPUT_ACRONYM_RE = re.compile(r"\b[A-Z][A-Z0-9]{1,10}\b")@@ -254,6 +259,15 @@ _OUTPUT_OUTCOME_WORDS = frozenset()+@dataclass(frozen=True)+class JudgmentTruthCondition:+ """Per-polarity page-evidence predicate a judgment boolean is decidable by. ``polarity_when_holds``+ is the emitted boolean value that corresponds to the predicate holding on the independent packet."""++ predicate: JudgmentPredicate+ polarity_when_holds: bool++@dataclass(frozen=True)class CompletionCriterion:id: str@@ -276,6 +290,7 @@ class CompletionCriterion:expected_classification: ClassificationTarget | None = Nonerequested_output_corroborator: bool = Falsemint_degrade: Literal["turn_unsatisfiable_fallback"] | None = None+ judgment_truth_condition: JudgmentTruthCondition | None = None@dataclass@@ -447,6 +462,12 @@ def typed_expected_output_value_key(value: ExpectedOutputValue | None) -> str:return ""+def judgment_truth_condition_key(condition: JudgmentTruthCondition | None) -> str:+ if condition is None:+ return ""+ return f"{condition.predicate}:{'t' if condition.polarity_when_holds else 'f'}"++def _criterion_grounding_mode(criterion: CompletionCriterion,) -> Literal["exact_value", "shape", "missing", "judgment_boolean"]:@@ -684,6 +705,14 @@ def _coerce_expected_output_shape(value: Any) -> ExpectedOutputShape | None:return None+def _coerce_judgment_truth_condition(predicate: Any, polarity: Any) -> JudgmentTruthCondition | None:+ if not isinstance(predicate, str) or predicate not in _JUDGMENT_PREDICATES:+ return None+ if not isinstance(polarity, bool):+ return None+ return JudgmentTruthCondition(predicate=cast(JudgmentPredicate, predicate), polarity_when_holds=polarity)++def _coerce_requested_output_evidence_source(value: Any) -> RequestedOutputEvidenceSource:if isinstance(value, str) and value in _REQUESTED_OUTPUT_EVIDENCE_SOURCES:return cast(RequestedOutputEvidenceSource, value)@@ -805,6 +834,9 @@ def _parse_completion_criteria(raw: Any) -> list[CompletionCriterion]:coerced_judgment_bool = _canonical_bool_string(expected_output_value)if coerced_judgment_bool is not None:expected_output_value = coerced_judgment_bool+ judgment_truth_condition = _coerce_judgment_truth_condition(+ item.get("judgment_predicate"), item.get("judgment_polarity_when_holds")+ )contingent_on_raw = item.get("contingent_on")contingent_on = (" ".join(contingent_on_raw.split())[:_COMPLETION_CRITERION_CONTINGENT_ON_MAX_CHARS].strip()@@ -844,6 +876,7 @@ def _parse_completion_criteria(raw: Any) -> list[CompletionCriterion]:typed_expected_output_value_key(expected_output_value),expected_output_shape or "",requested_output_evidence_source,+ judgment_truth_condition_key(judgment_truth_condition),)if key in seen:continue@@ -869,8 +902,17 @@ def _parse_completion_criteria(raw: Any) -> list[CompletionCriterion]:terminal_action_family=_coerce_terminal_action_family(item.get("terminal_action_family"), kind),classification_output_key=classification_output_key,expected_classification=expected_classification,+ judgment_truth_condition=judgment_truth_condition,))+ if judgment_truth_condition is not None:+ LOG.info(+ "copilot_judgment_truth_condition_minted",+ mint_source="classifier",+ predicate=judgment_truth_condition.predicate,+ polarity_when_holds=judgment_truth_condition.polarity_when_holds,+ criterion_id=criteria[-1].id,+ )if len(criteria) >= _MAX_COMPLETION_CRITERIA:breakreturn criteria@@ -1208,6 +1250,23 @@ def _requested_output_evidence_sources_from_criteria(return sources+def _requested_output_judgment_conditions_from_criteria(+ criteria: list[CompletionCriterion],+ requested_specs: list[tuple[str, str, str]],+) -> dict[str, JudgmentTruthCondition]:+ conditions: dict[str, JudgmentTruthCondition] = {}+ for criterion in criteria:+ if criterion.level == "definition" or criterion.method_mandated or criterion.judgment_truth_condition is None:+ continue+ for field_name, output_path, _field_label in requested_specs:+ if criterion.output_path != output_path and not _criterion_text_covers_requested_output(+ criterion, field_name+ ):+ continue+ conditions.setdefault(output_path, criterion.judgment_truth_condition)+ return conditions++def _requested_output_criterion_id(output_path: str) -> str:slug = "_".join(_word_tokens(output_path)) or "field"return f"{_REQUESTED_OUTPUT_CRITERION_ID_PREFIX}{slug}"@@ -1429,6 +1488,9 @@ def _apply_requested_output_completion_criteria(source_by_output_path = _requested_output_evidence_sources_from_criteria(policy.completion_criteria, requested_specs)+ judgment_condition_by_output_path = _requested_output_judgment_conditions_from_criteria(+ policy.completion_criteria, requested_specs+ )metadata_by_output_path: dict[str, tuple[str | None, str | None, Literal["registered_download"] | None]] = {}for criterion in policy.completion_criteria:@@ -1477,6 +1539,7 @@ def _apply_requested_output_completion_criteria(contingent_on=metadata_by_output_path.get(output_path, (None, None, None))[0],contingent_antecedent_output_path=metadata_by_output_path.get(output_path, (None, None, None))[1],deliverable_kind=metadata_by_output_path.get(output_path, (None, None, None))[2],+ judgment_truth_condition=judgment_condition_by_output_path.get(output_path),)for _field_name, output_path, field_label in requested_specs]@@ -1571,6 +1634,9 @@ def _render_active_criteria_for_prompt(criteria: list[CompletionCriterion] | Nonitem["expected_output_shape"] = criterion.expected_output_shapeif criterion.requested_output_evidence_source != "runtime_output":item["requested_output_evidence_source"] = criterion.requested_output_evidence_source+ if criterion.judgment_truth_condition is not None:+ item["judgment_predicate"] = criterion.judgment_truth_condition.predicate+ item["judgment_polarity_when_holds"] = criterion.judgment_truth_condition.polarity_when_holdsif criterion.classification_output_key:item["classification_output_key"] = criterion.classification_output_keyif criterion.expected_classification is not None:diff --git a/skyvern/forge/sdk/copilot/tools/completion.py b/skyvern/forge/sdk/copilot/tools/completion.pyindex 5f37e460e..ec3b00c52 100644--- a/skyvern/forge/sdk/copilot/tools/completion.py+++ b/skyvern/forge/sdk/copilot/tools/completion.py@@ -51,7 +51,11 @@ from skyvern.forge.sdk.copilot.reached_download_target import (ReachedDownloadTarget,derive_from_block_outputs,)-from skyvern.forge.sdk.copilot.request_policy import CompletionCriterion, is_fallback_floor_criterion+from skyvern.forge.sdk.copilot.request_policy import (+ CompletionCriterion,+ _is_judgment_boolean_criterion,+ is_fallback_floor_criterion,+)from skyvern.forge.sdk.copilot.runtime import PreRunPageReference, RegisteredArtifactEvidencefrom skyvern.forge.sdk.copilot.terminal_predicates import outcome_fully_verifiedfrom skyvern.forge.sdk.copilot.tracing_setup import copilot_span@@ -294,9 +298,16 @@ def _authored_output_contract_path(value: object) -> str:return f"output.{path}"-def _split_criteria_by_plane(criteria: list[Any]) -> tuple[list[CompletionCriterion], list[CompletionCriterion]]:- run_criteria = [c for c in criteria if getattr(c, "level", "run") != "definition"]- definition_criteria = [c for c in criteria if getattr(c, "level", "run") == "definition"]+def _split_criteria_by_plane(+ criteria: list[CompletionCriterion],+) -> tuple[list[CompletionCriterion], list[CompletionCriterion]]:+ run_criteria: list[CompletionCriterion] = []+ definition_criteria: list[CompletionCriterion] = []+ for criterion in criteria:+ if criterion.level != "definition" or (criterion.output_path and _is_judgment_boolean_criterion(criterion)):+ run_criteria.append(criterion)+ else:+ definition_criteria.append(criterion)return run_criteria, definition_criteria@@ -792,6 +803,15 @@ def _bind_independent_post_run_page_evidence(key: value for key, value in evidence.items() if key not in _POST_RUN_PAGE_EVIDENCE_STAMP_KEYS}block_output_sources[_POST_RUN_PAGE_OBSERVATION_LABEL] = "independent_page_evidence"+ LOG.info(+ "copilot_post_run_page_evidence_snapshot_binding",+ workflow_run_id=run_id,+ packet_label=_POST_RUN_PAGE_OBSERVATION_LABEL,+ evidence_source="independent_page_evidence",+ evidence_workflow_run_id=evidence.get("workflow_run_id"),+ evidence_observed_after_workflow_run=evidence.get("observed_after_workflow_run"),+ evidence_source_tool=evidence.get("source_tool"),+ )def _bind_registered_artifact_evidence(diff --git a/skyvern/forge/sdk/copilot/tools/workflow_update.py b/skyvern/forge/sdk/copilot/tools/workflow_update.pyindex f3cd77416..04a69bfce 100644--- a/skyvern/forge/sdk/copilot/tools/workflow_update.py+++ b/skyvern/forge/sdk/copilot/tools/workflow_update.py@@ -110,8 +110,11 @@ from skyvern.forge.sdk.copilot.reached_download_target import (code_is_download_intent,)from skyvern.forge.sdk.copilot.request_policy import (+ CompletionCriterion,+ JudgmentPredicate,RequestedOutputEvidenceSource,_coerce_requested_output_evidence_source,+ _is_judgment_boolean_criterion,)from skyvern.forge.sdk.copilot.runtime import (AgentContext,@@ -252,6 +255,17 @@ class CodeArtifactCompletionCriterion(BaseModel):terminal: bool | None = Noneoutput_path: str | None = Nonerequested_output_evidence_source: RequestedOutputEvidenceSource | None = None+ judgment_predicate: JudgmentPredicate | None = Field(+ default=None,+ description=(+ "For a judgment-boolean criterion, the closed-vocabulary page-evidence predicate the "+ "independent post-run packet decides this boolean by (e.g. `login_gate_blocks_target`)."+ ),+ )+ judgment_polarity_when_holds: bool | None = Field(+ default=None,+ description="The emitted boolean value that corresponds to `judgment_predicate` holding on the packet.",+ )class CodeArtifactScopedRef(BaseModel):@@ -1656,13 +1670,10 @@ def _recorded_outcome_missing_output_paths(ctx: AgentContext) -> set[str]:def _requested_output_child_paths(ctx: AgentContext) -> set[str]:if _copilot_block_authoring_policy(ctx) != BlockAuthoringPolicy.CODE_ONLY_BROWSER:return set()- request_policy = getattr(ctx, "request_policy", None)- criteria_source = getattr(request_policy, "graded_completion_criteria", None)- criteria = criteria_source() if callable(criteria_source) else getattr(request_policy, "completion_criteria", [])- if not isinstance(criteria, list):- return set()paths: set[str] = set()- for criterion in criteria:+ for criterion in _active_completion_criteria(ctx):+ if isinstance(criterion, CompletionCriterion) and _independent_judgment_output_criterion(criterion):+ continueif getattr(criterion, "level", None) == "definition":continueif getattr(criterion, "method_mandated", False):@@ -1675,6 +1686,32 @@ def _requested_output_child_paths(ctx: AgentContext) -> set[str]:return paths+def _active_completion_criteria(ctx: AgentContext) -> list[CompletionCriterion]:+ request_policy = ctx.request_policy+ if request_policy is None:+ return []+ return request_policy.graded_completion_criteria()+++def _independent_judgment_output_criterion(criterion: CompletionCriterion) -> bool:+ return criterion.requested_output_evidence_source == "independent_run_evidence" and _is_judgment_boolean_criterion(+ criterion+ )+++def _independent_judgment_output_paths(ctx: AgentContext) -> set[str]:+ paths: set[str] = set()+ for criterion in _active_completion_criteria(ctx):+ if not isinstance(criterion, CompletionCriterion):+ continue+ if not _independent_judgment_output_criterion(criterion):+ continue+ path = _canonical_requested_output_path(criterion.output_path)+ if path and _output_path_has_child(path):+ paths.add(path)+ return paths++def _canonical_requested_output_path(value: object) -> str:if not isinstance(value, str):return ""@@ -2296,7 +2333,11 @@ def _runtime_output_repair_contract_from_recorded_outcome(ctx: AgentContext) ->def _output_contract_required_paths_source(ctx: AgentContext) -> tuple[set[str], str, str]:runtime_contract = _runtime_output_repair_contract_from_recorded_outcome(ctx)if runtime_contract is not None:- return runtime_contract.required_paths, runtime_contract.source, runtime_contract.reason_code+ return (+ runtime_contract.required_paths - _independent_judgment_output_paths(ctx),+ runtime_contract.source,+ runtime_contract.reason_code,+ )required_paths, source, reason_code = _required_child_output_paths_for_authoring(ctx)repair_context = getattr(ctx, "last_code_authoring_repair_context", None)if required_paths or not isinstance(repair_context, CodeAuthoringRepairContext):@@ -2310,6 +2351,7 @@ def _output_contract_required_paths_source(ctx: AgentContext) -> tuple[set[str],*repair_context.required_code_return_paths,])+ required_paths -= _independent_judgment_output_paths(ctx)source = str(repair_context.metadata_contract_source or "").strip() or "metadata_reject"reason_code = (str(repair_context.metadata_contract_reason_code or "").strip()diff --git a/tests/unit/test_copilot_code_block_persist_seam.py b/tests/unit/test_copilot_code_block_persist_seam.pyindex 6643c3afd..d1f4d2e5b 100644--- a/tests/unit/test_copilot_code_block_persist_seam.py+++ b/tests/unit/test_copilot_code_block_persist_seam.py@@ -55,7 +55,7 @@ from skyvern.forge.sdk.copilot.output_contracts import ()from skyvern.forge.sdk.copilot.output_utils import sanitize_tool_result_for_llmfrom skyvern.forge.sdk.copilot.reached_download_target import ReachedDownloadTarget-from skyvern.forge.sdk.copilot.request_policy import RequestPolicy+from skyvern.forge.sdk.copilot.request_policy import CompletionCriterion, JudgmentTruthCondition, RequestPolicyfrom skyvern.forge.sdk.copilot.run_outcome import TERMINAL_CHALLENGE_BLOCKER_REASON_CODE, RecordedRunOutcomefrom skyvern.forge.sdk.copilot.runtime import AgentContextfrom skyvern.forge.sdk.copilot.tools import (@@ -4110,6 +4110,223 @@ class TestCodeRepairProgressClassification:schema = json.loads(scaffolded[0]["claimed_outcomes"][0]["extraction_schema"])assert schema["properties"]["output"]["properties"]["recorded_value"] == {}+ def test_independent_judgment_output_is_not_required_as_code_return_path(self) -> None:+ judgment_criterion = CompletionCriterion(+ id="login_gate",+ outcome="the target path is blocked by a login gate",+ output_path="output.login_gate_blocks_target",+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target",+ polarity_when_holds=True,+ ),+ )+ judgment_only_ctx = _code_only_ctx()+ judgment_only_ctx.request_policy = RequestPolicy(completion_criteria=[judgment_criterion])++ judgment_only_paths, _, _ = workflow_update_module._output_contract_required_paths_source(judgment_only_ctx)++ assert judgment_only_paths == set()++ ctx = _code_only_ctx()+ ctx.request_policy = RequestPolicy(+ completion_criteria=[+ judgment_criterion,+ CompletionCriterion(+ id="record_id",+ outcome="the record id is returned",+ output_path="output.record_id",+ ),+ ]+ )++ required_paths, source, reason_code = workflow_update_module._output_contract_required_paths_source(ctx)++ assert required_paths == {"output.record_id"}+ assert source == "requested_output_contract"+ assert reason_code == "requested_output_contract_missing_output_coverage"++ def test_independent_judgment_shape_output_is_not_required_as_code_return_path(self) -> None:+ ctx = _code_only_ctx()+ ctx.request_policy = RequestPolicy(+ completion_criteria=[+ CompletionCriterion(+ id="login_gate",+ outcome="the target path is blocked by a login gate",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ )+ ]+ )++ required_paths, _, _ = workflow_update_module._output_contract_required_paths_source(ctx)++ assert required_paths == set()++ def test_independent_judgment_repair_context_is_not_rehydrated_as_code_return_path(self) -> None:+ ctx = _code_only_ctx()+ ctx.request_policy = RequestPolicy(+ completion_criteria=[+ CompletionCriterion(+ id="login_gate",+ outcome="the target path is blocked by a login gate",+ output_path="output.login_gate_blocks_target",+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target",+ polarity_when_holds=True,+ ),+ )+ ]+ )+ ctx.last_code_authoring_repair_context = CodeAuthoringRepairContext(+ block_label="judge_login_gate_blocks_target",+ reason_code="metadata_reject",+ required_goal_value_paths=["output.login_gate_blocks_target"],+ required_extraction_schema_paths=["output.login_gate_blocks_target"],+ required_code_return_paths=["output.login_gate_blocks_target"],+ metadata_contract_source="requested_output_contract",+ metadata_contract_reason_code="requested_output_contract_missing_output_coverage",+ )++ required_paths, _, _ = workflow_update_module._output_contract_required_paths_source(ctx)+ result = workflow_update_module._metadata_contract_run_preflight_reject(ctx, _SAFE_CODE_YAML, [])++ assert required_paths == set()+ assert result is None++ def test_mixed_repair_context_keeps_non_judgment_code_return_path(self) -> None:+ ctx = _code_only_ctx()+ ctx.request_policy = RequestPolicy(+ completion_criteria=[+ CompletionCriterion(+ id="login_gate",+ outcome="the target path is blocked by a login gate",+ output_path="output.login_gate_blocks_target",+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target",+ polarity_when_holds=True,+ ),+ )+ ]+ )+ ctx.last_code_authoring_repair_context = CodeAuthoringRepairContext(+ block_label="extract_entry_output",+ reason_code="metadata_reject",+ required_goal_value_paths=["output.login_gate_blocks_target", "output.record_id"],+ required_extraction_schema_paths=["output.login_gate_blocks_target", "output.record_id"],+ required_code_return_paths=["output.login_gate_blocks_target", "output.record_id"],+ metadata_contract_source="requested_output_contract",+ metadata_contract_reason_code="requested_output_contract_missing_output_coverage",+ )++ required_paths, source, reason_code = workflow_update_module._output_contract_required_paths_source(ctx)++ assert required_paths == {"output.record_id"}+ assert source == "requested_output_contract"+ assert reason_code == "requested_output_contract_missing_output_coverage"++ def test_independent_judgment_runtime_repair_fact_is_not_rehydrated_as_code_return_path(self) -> None:+ ctx = _code_only_ctx()+ ctx.request_policy = RequestPolicy(+ completion_criteria=[+ CompletionCriterion(+ id="login_gate",+ outcome="the target path is blocked by a login gate",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ )+ ]+ )+ ctx.latest_recorded_build_test_outcome = RecordedBuildTestOutcome(+ phase="persisted_block_run",+ attempted_tool="update_and_run_blocks",+ verdict="repairable_failure",+ reason_code="outcome_not_demonstrated",+ workflow_run_id="wr_current",+ structural_failure_identity="completion:runtime-output",+ runtime_output_repair_facts=[+ {+ "workflow_run_id": "wr_current",+ "block_label": "judge_login_gate_blocks_target",+ "output_path": "output.login_gate_blocks_target",+ "output_root": "output",+ "criterion_id": "__copilot_requested_output__output_login_gate_blocks_target",+ "reason_code": "structurally_abstained",+ "grounding_mode": "judgment_boolean",+ "value_status": "structural_abstained",+ }+ ],+ )++ required_paths, _, _ = workflow_update_module._output_contract_required_paths_source(ctx)++ assert required_paths == set()++ def test_mixed_runtime_repair_facts_keep_non_judgment_code_return_path(self) -> None:+ ctx = _code_only_ctx()+ ctx.request_policy = RequestPolicy(+ completion_criteria=[+ CompletionCriterion(+ id="login_gate",+ outcome="the target path is blocked by a login gate",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ ),+ CompletionCriterion(+ id="record_id",+ outcome="the record id is returned",+ output_path="output.record_id",+ ),+ ]+ )+ ctx.latest_recorded_build_test_outcome = RecordedBuildTestOutcome(+ phase="persisted_block_run",+ attempted_tool="update_and_run_blocks",+ verdict="repairable_failure",+ reason_code="outcome_not_demonstrated",+ workflow_run_id="wr_current",+ structural_failure_identity="completion:runtime-output",+ runtime_output_repair_facts=[+ {+ "workflow_run_id": "wr_current",+ "block_label": "extract_entry_output",+ "output_path": "output.login_gate_blocks_target",+ "output_root": "output",+ "criterion_id": "__copilot_requested_output__output_login_gate_blocks_target",+ "reason_code": "structurally_abstained",+ "grounding_mode": "judgment_boolean",+ "value_status": "structural_abstained",+ },+ {+ "workflow_run_id": "wr_current",+ "block_label": "extract_entry_output",+ "output_path": "output.record_id",+ "output_root": "output",+ "criterion_id": "record_id",+ "reason_code": "structurally_abstained",+ "grounding_mode": "missing",+ "value_status": "structural_abstained",+ },+ ],+ )++ required_paths, source, reason_code = workflow_update_module._output_contract_required_paths_source(ctx)++ assert required_paths == {"output.record_id"}+ assert source == "runtime_output_repair"+ assert reason_code == "runtime_output_repair_required"+def test_runtime_output_facts_record_same_run_null_without_evidence_text_backfill(self) -> None:result = {"ok": True,diff --git a/tests/unit/test_copilot_completion_verification.py b/tests/unit/test_copilot_completion_verification.pyindex 4bff2c844..b2e9f2176 100644--- a/tests/unit/test_copilot_completion_verification.py+++ b/tests/unit/test_copilot_completion_verification.py@@ -9,6 +9,7 @@ from types import SimpleNamespacefrom typing import Anyimport pytest+from structlog.testing import capture_logsfrom skyvern.config import settingsfrom skyvern.forge.sdk.copilot.agent import (@@ -75,6 +76,7 @@ from skyvern.forge.sdk.copilot.hooks import _tool_completion_satisfies_turnfrom skyvern.forge.sdk.copilot.reached_download_target import ReachedDownloadTargetfrom skyvern.forge.sdk.copilot.request_policy import (CompletionCriterion,+ JudgmentTruthCondition,RequestPolicy,_parse_completion_criteria,build_classifier_fallback_floor,@@ -103,6 +105,7 @@ from skyvern.forge.sdk.copilot.tools import (from skyvern.forge.sdk.copilot.tools.completion import (_POST_RUN_PAGE_OBSERVATION_LABEL,_artifact_health_blocker_from_result,+ _completion_verification_from_run_result,_reconcile_download_completion_criterion,)from skyvern.forge.sdk.copilot.tools.composition_capture import _active_run_terminal_monitor_enabled@@ -9428,6 +9431,771 @@ def test_judgment_boolean_refuted_when_independent_evidence_resolves_false_leaf(assert verdicts[0].evidence_source == "independent_page_evidence"+def _login_gate_packet() -> dict[str, object]:+ return {+ "forms": [+ {+ "fields": [{"type": "password", "selector": "#password", "disabled": False}],+ "submit_controls": [{"type": "submit", "selector": "button[type=submit]", "disabled": False}],+ }+ ],+ "navigation_targets": [{"href": "https://example.test/account", "selector": "a.account"}],+ "result_containers": [],+ }+++def _gated_login_packet() -> dict[str, object]:+ return {+ "forms": [+ {+ "fields": [{"type": "password", "selector": "#password", "disabled": False}],+ "submit_controls": [{"type": "submit", "selector": "button[type=submit]", "disabled": True}],+ }+ ],+ "navigation_targets": [{"href": "https://example.test/account", "selector": "a.account"}],+ "result_containers": [],+ "challenge_state": {+ "gates_submit_controls": True,+ "gated_submit_controls": [{"selector": "button[type=submit]", "disabled": True}],+ },+ }+++def _target_reached_packet() -> dict[str, object]:+ return {+ "forms": [],+ "navigation_targets": [{"href": "https://example.test/account", "selector": "a.account"}],+ "result_containers": [{"selector": "#account-summary", "row_count": 1}],+ }+++def test_judgment_truth_condition_confirms_login_gate_from_independent_packet() -> None:+ ctx = _run_ctx()+ ctx.code_artifact_metadata = _metadata_for_requested_paths("login_gate_blocks_target")+ criteria = [+ replace(+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ requested_output_evidence_source="independent_run_evidence",+ ),+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=True+ ),+ )+ ]++ with capture_logs() as logs:+ verdicts = grade_requested_output_criteria(+ ctx,+ criteria,+ RunEvidenceSnapshot(+ block_outputs={_POST_RUN_PAGE_OBSERVATION_LABEL: _login_gate_packet()},+ block_output_sources={_POST_RUN_PAGE_OBSERVATION_LABEL: "independent_page_evidence"},+ ),+ )++ assert verdicts[0].state == "satisfied"+ assert verdicts[0].reason_code == "evidence_confirms"+ assert verdicts[0].evidence_ref == f"block_outputs:{_POST_RUN_PAGE_OBSERVATION_LABEL}"+ assert verdicts[0].evidence_source == "independent_page_evidence"+ assert logs[-1]["event"] == "copilot_judgment_evidence_verdict"+ assert logs[-1]["predicate"] == "login_gate_blocks_target"+ assert logs[-1]["packet_label"] == _POST_RUN_PAGE_OBSERVATION_LABEL+ assert logs[-1]["verdict"] == "evidence_confirms"+ assert logs[-1]["origin"] == "criterion"+++def test_bound_gated_submit_login_packet_wins_over_self_emitted_judgment() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "extract_profile")+ ctx.code_artifact_metadata = {+ "extract_profile": {+ "claimed_outcomes": [{"goal_value_paths": ["login_gate_blocks_target"]}],+ "completion_criteria": [+ {+ "output_path": "output.login_gate_blocks_target",+ "requested_output_evidence_source": "independent_run_evidence",+ "judgment_predicate": "login_gate_blocks_target",+ "judgment_polarity_when_holds": True,+ }+ ],+ }+ }+ ctx.composition_page_evidence = _post_run_page_evidence(+ run_id="wr_requested_output",+ visible_text_excerpt="",+ **_gated_login_packet(),+ )+ criteria = [+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ )+ ]++ snapshot = _build_run_evidence_snapshot(+ ctx,+ _requested_output_result({"login_gate_blocks_target": True}),+ )+ with capture_logs() as logs:+ verdicts = grade_requested_output_criteria(ctx, criteria, snapshot)++ assert snapshot.block_output_sources[_POST_RUN_PAGE_OBSERVATION_LABEL] == "independent_page_evidence"+ assert verdicts[0].state == "satisfied"+ assert verdicts[0].reason_code == "evidence_confirms"+ assert verdicts[0].evidence_source == "independent_page_evidence"+ assert verdicts[0].self_emitted_judgment_not_independent is False+ assert any(+ log["event"] == "copilot_judgment_evidence_verdict" and log["verdict"] == "evidence_confirms" for log in logs+ )+++def test_bound_independent_page_output_wins_over_self_emitted_judgment_value() -> None:+ ctx = _run_ctx()+ ctx.code_artifact_metadata = _metadata_for_requested_paths("login_gate_blocks_target")+ criteria = [+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ )+ ]++ verdicts = grade_requested_output_criteria(+ ctx,+ criteria,+ RunEvidenceSnapshot(+ block_outputs={+ "extract_profile": {"login_gate_blocks_target": True},+ _POST_RUN_PAGE_OBSERVATION_LABEL: {"login_gate_blocks_target": True},+ },+ block_output_sources={+ "extract_profile": "runtime_output",+ _POST_RUN_PAGE_OBSERVATION_LABEL: "independent_page_evidence",+ },+ ),+ )++ assert verdicts[0].state == "satisfied"+ assert verdicts[0].reason_code == "evidence_confirms"+ assert verdicts[0].evidence_ref == f"block_outputs:{_POST_RUN_PAGE_OBSERVATION_LABEL}.login_gate_blocks_target"+ assert verdicts[0].evidence_source == "independent_page_evidence"+ assert verdicts[0].self_emitted_judgment_not_independent is False+++def test_value_less_emitted_false_judgment_is_refuted_by_login_gate_packet() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "extract_profile")+ ctx.code_artifact_metadata = {+ "extract_profile": {+ "claimed_outcomes": [{"goal_value_paths": ["login_gate_blocks_target"]}],+ "completion_criteria": [+ {+ "output_path": "output.login_gate_blocks_target",+ "requested_output_evidence_source": "independent_run_evidence",+ "judgment_predicate": "login_gate_blocks_target",+ "judgment_polarity_when_holds": True,+ }+ ],+ }+ }+ ctx.composition_page_evidence = _post_run_page_evidence(+ run_id="wr_requested_output",+ visible_text_excerpt="",+ **_gated_login_packet(),+ )+ criteria = [+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ )+ ]++ snapshot = _build_run_evidence_snapshot(+ ctx,+ _requested_output_result({"login_gate_blocks_target": False}),+ )+ with capture_logs() as logs:+ verdicts = grade_requested_output_criteria(ctx, criteria, snapshot)++ assert verdicts[0].state == "unsatisfied"+ assert verdicts[0].reason_code == "evidence_contradicts"+ assert verdicts[0].evidence_ref == f"block_outputs:{_POST_RUN_PAGE_OBSERVATION_LABEL}"+ assert verdicts[0].evidence_source == "independent_page_evidence"+ assert verdicts[0].has_exact_value is False+ assert any(+ log["event"] == "copilot_judgment_evidence_verdict"+ and log["predicate"] == "login_gate_blocks_target"+ and log["packet_label"] == _POST_RUN_PAGE_OBSERVATION_LABEL+ and log["verdict"] == "evidence_contradicts"+ for log in logs+ )+++def test_producer_declared_value_less_judgment_reaches_packet_verdict() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "extract_profile")+ ctx.code_artifact_metadata = {+ "extract_profile": {+ "claimed_outcomes": [{"goal_value_paths": ["login_gate_blocks_target"]}],+ "completion_criteria": [+ {+ "output_path": "output.login_gate_blocks_target",+ "requested_output_evidence_source": "independent_run_evidence",+ "judgment_predicate": "login_gate_blocks_target",+ "judgment_polarity_when_holds": True,+ }+ ],+ }+ }+ ctx.composition_page_evidence = _post_run_page_evidence(+ run_id="wr_requested_output",+ visible_text_excerpt="",+ **_gated_login_packet(),+ )+ criteria = [+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ requested_output_evidence_source="independent_run_evidence",+ )+ ]++ snapshot = _build_run_evidence_snapshot(+ ctx,+ _requested_output_result({"login_gate_blocks_target": True}),+ )+ with capture_logs() as logs:+ verdicts = grade_requested_output_criteria(ctx, criteria, snapshot)++ assert verdicts[0].state == "satisfied"+ assert verdicts[0].reason_code == "evidence_confirms"+ assert verdicts[0].grounding_mode == "judgment_boolean"+ assert verdicts[0].has_exact_value is False+ assert verdicts[0].evidence_ref == f"block_outputs:{_POST_RUN_PAGE_OBSERVATION_LABEL}"+ assert verdicts[0].evidence_source == "independent_page_evidence"+ assert any(+ log["event"] == "copilot_judgment_evidence_verdict"+ and log["predicate"] == "login_gate_blocks_target"+ and log["packet_label"] == _POST_RUN_PAGE_OBSERVATION_LABEL+ and log["verdict"] == "evidence_confirms"+ and log["origin"] == "producer_metadata"+ for log in logs+ )+++def test_producer_declared_truth_condition_without_metadata_source_reaches_packet_verdict() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "extract_profile")+ ctx.code_artifact_metadata = {+ "extract_profile": {+ "claimed_outcomes": [{"goal_value_paths": ["login_gate_blocks_target"]}],+ "completion_criteria": [+ {+ "output_path": "output.login_gate_blocks_target",+ "judgment_predicate": "login_gate_blocks_target",+ "judgment_polarity_when_holds": True,+ }+ ],+ }+ }+ ctx.composition_page_evidence = _post_run_page_evidence(+ run_id="wr_requested_output",+ visible_text_excerpt="",+ **_gated_login_packet(),+ )+ criteria = [+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ requested_output_evidence_source="independent_run_evidence",+ )+ ]++ snapshot = _build_run_evidence_snapshot(+ ctx,+ _requested_output_result({"login_gate_blocks_target": True}),+ )+ with capture_logs() as logs:+ verdicts = grade_requested_output_criteria(ctx, criteria, snapshot)++ assert verdicts[0].state == "satisfied"+ assert verdicts[0].reason_code == "evidence_confirms"+ assert verdicts[0].grounding_mode == "judgment_boolean"+ assert verdicts[0].has_exact_value is False+ assert verdicts[0].evidence_source == "independent_page_evidence"+ assert any(+ log["event"] == "copilot_judgment_evidence_verdict"+ and log["predicate"] == "login_gate_blocks_target"+ and log["origin"] == "producer_metadata"+ for log in logs+ )+++async def test_definition_level_judgment_output_reaches_independent_packet_grader() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "extract_profile")+ ctx.code_artifact_metadata = {+ "extract_profile": {+ "claimed_outcomes": [{"goal_value_paths": ["login_gate_blocks_target"]}],+ "completion_criteria": [+ {+ "output_path": "output.login_gate_blocks_target",+ "requested_output_evidence_source": "independent_run_evidence",+ "judgment_predicate": "login_gate_blocks_target",+ "judgment_polarity_when_holds": True,+ }+ ],+ }+ }+ ctx.composition_page_evidence = _post_run_page_evidence(+ run_id="wr_requested_output",+ visible_text_excerpt="",+ **_gated_login_packet(),+ )+ criteria = [+ replace(+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ level="definition",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ ),+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=True+ ),+ )+ ]++ with capture_logs() as logs:+ verification = await _completion_verification_from_run_result(+ ctx,+ _requested_output_result({"login_gate_blocks_target": True}),+ time.monotonic(),+ criteria,+ )++ assert verification is not None+ assert verification.verdicts[0].state == "satisfied"+ assert verification.verdicts[0].reason_code == "evidence_confirms"+ assert verification.verdicts[0].evidence_source == "independent_page_evidence"+ assert any(+ log["event"] == "copilot_judgment_evidence_verdict" and log["verdict"] == "evidence_confirms" for log in logs+ )+++async def test_authored_output_fallback_preserves_staged_judgment_truth_condition() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "extract_profile")+ metadata = {+ "extract_profile": {+ "claimed_outcomes": [{"goal_value_paths": ["login_gate_blocks_target"]}],+ "completion_criteria": [+ {+ "output_path": "output.login_gate_blocks_target",+ "requested_output_evidence_source": "independent_run_evidence",+ "judgment_predicate": "login_gate_blocks_target",+ "judgment_polarity_when_holds": True,+ }+ ],+ }+ }+ ctx.workflow_verification_evidence = SimpleNamespace(code_artifact_metadata=metadata)+ ctx.code_artifact_metadata = {}+ ctx.request_policy = RequestPolicy(completion_criteria=build_classifier_fallback_floor([]))+ ctx.composition_page_evidence = _post_run_page_evidence(+ run_id="wr_requested_output",+ visible_text_excerpt="",+ **_gated_login_packet(),+ )++ with capture_logs() as logs:+ verification = await _maybe_run_completion_verification(+ ctx,+ _requested_output_result({"login_gate_blocks_target": True}),+ time.monotonic(),+ )++ assert verification is not None+ verdict = verification.verdicts[0]+ assert verdict.criterion_id == "__copilot_authored_output__output_login_gate_blocks_target"+ assert verdict.state == "satisfied"+ assert verdict.reason_code == "evidence_confirms"+ assert verdict.grounding_mode == "judgment_boolean"+ assert verdict.evidence_source == "independent_page_evidence"+ assert any(+ log["event"] == "copilot_judgment_evidence_verdict"+ and log["predicate"] == "login_gate_blocks_target"+ and log["origin"] == "producer_metadata"+ for log in logs+ )+++async def test_authored_output_fallback_recovers_path_declared_judgment_truth_condition() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "extract_profile")+ metadata = {+ "extract_profile": {+ "claimed_outcomes": [{"goal_value_paths": ["login_gate_blocks_target"]}],+ "completion_criteria": [+ {+ "output_path": "output.login_gate_blocks_target",+ "requested_output_evidence_source": "independent_run_evidence",+ }+ ],+ }+ }+ ctx.workflow_verification_evidence = SimpleNamespace(code_artifact_metadata=metadata)+ ctx.code_artifact_metadata = {}+ ctx.request_policy = RequestPolicy(completion_criteria=build_classifier_fallback_floor([]))+ ctx.composition_page_evidence = _post_run_page_evidence(+ run_id="wr_requested_output",+ visible_text_excerpt="",+ **_gated_login_packet(),+ )++ with capture_logs() as logs:+ verification = await _maybe_run_completion_verification(+ ctx,+ _requested_output_result({"login_gate_blocks_target": True}),+ time.monotonic(),+ )++ assert verification is not None+ verdict = verification.verdicts[0]+ assert verdict.criterion_id == "__copilot_authored_output__output_login_gate_blocks_target"+ assert verdict.state == "satisfied"+ assert verdict.reason_code == "evidence_confirms"+ assert verdict.grounding_mode == "judgment_boolean"+ assert verdict.evidence_source == "independent_page_evidence"+ assert any(+ log["event"] == "copilot_judgment_evidence_verdict"+ and log["predicate"] == "login_gate_blocks_target"+ and log["origin"] == "producer_metadata"+ for log in logs+ )+++async def test_authored_output_duplicate_path_judgment_row_reaches_packet_grader() -> None:+ ctx = _run_ctx()+ _set_workflow_labels(ctx, "validate_login_gate_blocks_target")+ metadata = {+ "validate_login_gate_blocks_target": {+ "claimed_outcomes": [{"goal_value_paths": ["login_gate_blocks_target"]}],+ "completion_criteria": [+ {+ "output_path": "login_gate_blocks_target",+ "requested_output_evidence_source": "runtime_output",+ },+ {+ "output_path": "login_gate_blocks_target",+ "requested_output_evidence_source": "independent_run_evidence",+ "judgment_predicate": "login_gate_blocks_target",+ "judgment_polarity_when_holds": True,+ },+ ],+ }+ }+ ctx.workflow_verification_evidence = SimpleNamespace(code_artifact_metadata=metadata)+ ctx.code_artifact_metadata = {}+ ctx.request_policy = RequestPolicy(completion_criteria=build_classifier_fallback_floor([]))+ ctx.composition_page_evidence = _post_run_page_evidence(+ run_id="wr_requested_output",+ visible_text_excerpt="",+ **_gated_login_packet(),+ )++ with capture_logs() as logs:+ verification = await _maybe_run_completion_verification(+ ctx,+ {+ "ok": True,+ "data": {+ "workflow_run_id": "wr_requested_output",+ "overall_status": "completed",+ "executed_block_labels": ["validate_login_gate_blocks_target"],+ "current_url": "https://example.test/profile",+ "blocks": [+ {+ "label": "validate_login_gate_blocks_target",+ "block_type": "CODE",+ "status": "completed",+ "extracted_data": {"login_gate_blocks_target": False},+ }+ ],+ },+ },+ time.monotonic(),+ )++ assert verification is not None+ verdict = verification.verdicts[0]+ assert verdict.criterion_id == "__copilot_authored_output__output_login_gate_blocks_target"+ assert verdict.state == "unsatisfied"+ assert verdict.reason_code == "evidence_contradicts"+ assert verdict.grounding_mode == "judgment_boolean"+ assert verdict.evidence_source == "independent_page_evidence"+ assert any(+ log["event"] == "copilot_judgment_evidence_verdict"+ and log["predicate"] == "login_gate_blocks_target"+ and log["verdict"] == "evidence_contradicts"+ for log in logs+ )+++def test_judgment_truth_condition_refutes_inverted_login_gate_packet() -> None:+ ctx = _run_ctx()+ ctx.code_artifact_metadata = _metadata_for_requested_paths("login_gate_blocks_target")+ criteria = [+ replace(+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ requested_output_evidence_source="independent_run_evidence",+ ),+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=True+ ),+ )+ ]++ verdicts = grade_requested_output_criteria(+ ctx,+ criteria,+ RunEvidenceSnapshot(+ block_outputs={_POST_RUN_PAGE_OBSERVATION_LABEL: _target_reached_packet()},+ block_output_sources={_POST_RUN_PAGE_OBSERVATION_LABEL: "independent_page_evidence"},+ ),+ )++ assert verdicts[0].state == "unsatisfied"+ assert verdicts[0].reason_code == "evidence_contradicts"+ assert verdicts[0].evidence_ref == f"block_outputs:{_POST_RUN_PAGE_OBSERVATION_LABEL}"+ assert verdicts[0].evidence_source == "independent_page_evidence"+++def test_judgment_truth_condition_hollow_packet_abstains_before_self_emitted_output() -> None:+ ctx = _run_ctx()+ ctx.code_artifact_metadata = _metadata_for_requested_paths("login_gate_blocks_target")+ criteria = [+ replace(+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ requested_output_evidence_source="independent_run_evidence",+ ),+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=True+ ),+ )+ ]++ verdicts = grade_requested_output_criteria(+ ctx,+ criteria,+ RunEvidenceSnapshot(+ block_outputs={+ "extract_profile": {"login_gate_blocks_target": True},+ _POST_RUN_PAGE_OBSERVATION_LABEL: {+ "forms": [],+ "navigation_targets": [],+ "result_containers": [],+ },+ },+ block_output_sources={+ "extract_profile": "runtime_output",+ _POST_RUN_PAGE_OBSERVATION_LABEL: "independent_page_evidence",+ },+ ),+ )++ assert verdicts[0].state == "unsatisfied"+ assert verdicts[0].reason_code == "structurally_abstained"+ assert verdicts[0].evidence_ref == f"block_outputs:{_POST_RUN_PAGE_OBSERVATION_LABEL}"+ assert verdicts[0].evidence_source == "independent_page_evidence"+ assert verdicts[0].self_emitted_judgment_not_independent is False+++def test_judgment_truth_condition_refuses_fabricated_packet_from_registered_output() -> None:+ ctx = _run_ctx()+ ctx.code_artifact_metadata = _metadata_for_requested_paths("login_gate_blocks_target")+ criteria = [+ replace(+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ requested_output_evidence_source="independent_run_evidence",+ ),+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=True+ ),+ )+ ]++ verdicts = grade_requested_output_criteria(+ ctx,+ criteria,+ RunEvidenceSnapshot(+ block_outputs={+ "extract_profile": {"login_gate_blocks_target": True},+ "fake_packet_output": _gated_login_packet(),+ },+ block_output_sources={+ "extract_profile": "runtime_output",+ "fake_packet_output": "registered_output_parameter",+ },+ ),+ )++ assert verdicts[0].state == "unsatisfied"+ assert verdicts[0].reason_code != "evidence_confirms"+ assert verdicts[0].evidence_ref != "block_outputs:fake_packet_output"+++def test_judgment_truth_condition_ignores_packet_label_and_visible_text() -> None:+ ctx = _run_ctx()+ ctx.code_artifact_metadata = _metadata_for_requested_paths("login_gate_blocks_target")+ criteria = [+ replace(+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ requested_output_evidence_source="independent_run_evidence",+ ),+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=True+ ),+ )+ ]++ verdicts = grade_requested_output_criteria(+ ctx,+ criteria,+ RunEvidenceSnapshot(+ block_outputs={+ "login_gate_blocks_target": {+ "visible_text_excerpt": "Login required. Enter password to continue.",+ "evidence_text": "Login required.",+ }+ },+ block_output_sources={"login_gate_blocks_target": "independent_page_evidence"},+ ),+ )++ assert verdicts[0].reason_code != "evidence_confirms"+++def test_producer_declared_judgment_truth_condition_fallback_confirms_packet() -> None:+ ctx = _run_ctx()+ ctx.code_artifact_metadata = {+ "extract_profile": {+ "claimed_outcomes": [{"goal_value_paths": ["login_gate_blocks_target"]}],+ "completion_criteria": [+ {+ "output_path": "output.login_gate_blocks_target",+ "requested_output_evidence_source": "independent_run_evidence",+ "judgment_predicate": "login_gate_blocks_target",+ "judgment_polarity_when_holds": True,+ }+ ],+ }+ }+ criteria = [+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ requested_output_evidence_source="independent_run_evidence",+ )+ ]++ verdicts = grade_requested_output_criteria(+ ctx,+ criteria,+ RunEvidenceSnapshot(+ block_outputs={_POST_RUN_PAGE_OBSERVATION_LABEL: _login_gate_packet()},+ block_output_sources={_POST_RUN_PAGE_OBSERVATION_LABEL: "independent_page_evidence"},+ ),+ )++ assert verdicts[0].state == "satisfied"+ assert verdicts[0].reason_code == "evidence_confirms"+++def test_criterion_judgment_truth_condition_takes_precedence_over_metadata() -> None:+ ctx = _run_ctx()+ ctx.code_artifact_metadata = {+ "extract_profile": {+ "claimed_outcomes": [{"goal_value_paths": ["login_gate_blocks_target"]}],+ "completion_criteria": [+ {+ "output_path": "output.login_gate_blocks_target",+ "requested_output_evidence_source": "independent_run_evidence",+ "judgment_predicate": "login_gate_blocks_target",+ "judgment_polarity_when_holds": True,+ }+ ],+ }+ }+ criteria = [+ replace(+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ expected_output_value=False,+ requested_output_evidence_source="independent_run_evidence",+ ),+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=False+ ),+ )+ ]++ verdicts = grade_requested_output_criteria(+ ctx,+ criteria,+ RunEvidenceSnapshot(+ block_outputs={_POST_RUN_PAGE_OBSERVATION_LABEL: _login_gate_packet()},+ block_output_sources={_POST_RUN_PAGE_OBSERVATION_LABEL: "independent_page_evidence"},+ ),+ )++ assert verdicts[0].state == "satisfied"+ assert verdicts[0].reason_code == "evidence_confirms"++def _judgment_shape_abstention() -> CriterionVerdict:return CriterionVerdict(criterion_id="c_judgment",diff --git a/tests/unit/test_copilot_positive_evidence_carriers.py b/tests/unit/test_copilot_positive_evidence_carriers.pyindex 5f7419ee9..0883812fc 100644--- a/tests/unit/test_copilot_positive_evidence_carriers.py+++ b/tests/unit/test_copilot_positive_evidence_carriers.py@@ -35,6 +35,7 @@ _FLOOR_ID = "__copilot_fallback_floor__run"class _GroundingCtx:def __init__(self) -> None:self.code_artifact_metadata: dict[str, object] = {}+ self.workflow_verification_evidence: SimpleNamespace | None = Noneself.last_workflow_yaml: str | None = Noneself.workflow_yaml: str | None = Nonediff --git a/tests/unit/test_copilot_request_policy_completion_criteria.py b/tests/unit/test_copilot_request_policy_completion_criteria.pyindex 624bbe8bb..dbe316144 100644--- a/tests/unit/test_copilot_request_policy_completion_criteria.py+++ b/tests/unit/test_copilot_request_policy_completion_criteria.py@@ -2,6 +2,7 @@ from __future__ import annotationsimport hashlibimport json+from dataclasses import replacefrom typing import Anyimport pytest@@ -17,6 +18,7 @@ from skyvern.forge.sdk.copilot.completion_criteria_store import (from skyvern.forge.sdk.copilot.config import CopilotConfigfrom skyvern.forge.sdk.copilot.request_policy import (CompletionCriterion,+ JudgmentTruthCondition,RequestPolicy,_apply_requested_output_completion_criteria,_apply_validation_classification_completion_criteria,@@ -1718,6 +1720,56 @@ def test_parser_promotes_goal_judgment_boolean_shape_without_explicit_value() ->assert _criterion_grounding_mode(criteria[0]) == "judgment_boolean"+def test_parser_mints_login_gate_judgment_truth_condition_from_classifier_fields() -> None:+ criteria = _parse_completion_criteria(+ [+ {+ "outcome": "The returned record reports whether the login gate blocks the target.",+ "output_path": "output.login_gate_blocks_target",+ "expected_output_shape": "goal_judgment_boolean",+ "requested_output_evidence_source": "independent_run_evidence",+ "judgment_predicate": "login_gate_blocks_target",+ "judgment_polarity_when_holds": True,+ }+ ]+ )++ assert criteria[0].judgment_truth_condition == JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=True+ )+++def test_backticked_named_output_mints_login_gate_requested_output_truth_condition() -> None:+ policy = RequestPolicy(+ completion_criteria=[+ replace(+ _criterion(+ "c_login_gate",+ "The returned record reports login gate blocks target.",+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ ),+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=True+ ),+ )+ ]+ )++ _apply_requested_output_completion_criteria(+ policy,+ "Create a validation workflow and return the named output `login_gate_blocks_target`.",+ )++ criteria = _criteria_for_path(policy, "output.login_gate_blocks_target")+ assert len(criteria) == 1+ assert criteria[0].expected_output_shape == "goal_judgment_boolean"+ assert criteria[0].requested_output_evidence_source == "independent_run_evidence"+ assert criteria[0].judgment_truth_condition == JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=True+ )++def test_parser_dedup_keeps_runtime_string_and_judgment_boolean_on_same_path() -> None:criteria = _parse_completion_criteria([@@ -1788,6 +1840,26 @@ def test_store_round_trip_preserves_boolean_expected_value_and_source() -> None:assert _criterion_grounding_mode(round_tripped[0]) == "judgment_boolean"+def test_store_round_trip_preserves_judgment_truth_condition() -> None:+ criterion = replace(+ _criterion(+ "c_login_gate",+ "The returned record reports whether the login gate blocks the target.",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ requested_output_evidence_source="independent_run_evidence",+ ),+ judgment_truth_condition=JudgmentTruthCondition(predicate="login_gate_blocks_target", polarity_when_holds=True),+ )++ round_tripped = criteria_from_json(criteria_to_json([criterion]))++ assert round_tripped[0].judgment_truth_condition == JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=True+ )+ assert _criterion_reconcile_key(round_tripped[0]) == _criterion_reconcile_key(criterion)++def test_expected_false_survives_active_criteria_rendering() -> None:criterion = _criterion("c_bool",diff --git a/tests/unit/test_copilot_request_policy_prompt.py b/tests/unit/test_copilot_request_policy_prompt.pyindex 05f283ec1..d60622354 100644--- a/tests/unit/test_copilot_request_policy_prompt.py+++ b/tests/unit/test_copilot_request_policy_prompt.py@@ -11,6 +11,7 @@ from skyvern.forge.sdk.copilot.request_policy import (PROMPT_NAME,RAW_SECRET_REFUSAL_SENTINEL,CompletionCriterion,+ JudgmentTruthCondition,_classifier_fallback_policy,_classify_request,_credential_ids,@@ -91,7 +92,7 @@ class TestRequestPolicyPromptStructure:"{outcome, contingent_on, contingent_antecedent_output_path, ""deliverable_kind, implicit, method_mandated, level, output_path, expected_output_value, ""expected_output_shape, requested_output_evidence_source, kind, terminal_action_family, "- "classification_output_key, expected_classification}"+ "classification_output_key, expected_classification, judgment_predicate, judgment_polarity_when_holds}") in renderedassert "never hide it in outcome prose" in renderedassert (@@ -108,6 +109,17 @@ class TestRequestPolicyPromptStructure:assert "classification_output_key=login_only and expected_classification=true" in renderedassert 'The only supported non-null value is "registered_download"' in rendered+ def test_completion_criteria_schema_exposes_judgment_truth_condition_fields(self) -> None:+ rendered = _render()+ assert "judgment_predicate: null unless expected_output_shape=goal_judgment_boolean" in rendered+ assert "Use only the closed-vocabulary value login_gate_blocks_target" in rendered+ assert "judgment_polarity_when_holds: null unless judgment_predicate is non-null" in rendered+ assert "kind=outcome" in rendered+ assert "output_path=output.login_gate_blocks_target" in rendered+ assert "judgment_predicate=login_gate_blocks_target" in rendered+ assert "judgment_polarity_when_holds=true" in rendered+ assert "Do not use kind=validation_classification for this returned field" in rendered+def test_priority_requested_output_evidence_source_guidance_is_present(self) -> None:rendered = _render()assert (@@ -140,6 +152,28 @@ class TestRequestPolicyPromptStructure:assert '"expected_output_value": "WTR-1842-DEMO"' in renderedassert '"expected_output_shape": "reference_code"' in rendered+ def test_active_completion_criteria_render_judgment_truth_condition(self) -> None:+ active = _render_active_criteria_for_prompt(+ [+ CompletionCriterion(+ id="c0",+ outcome="the returned record reports whether the login gate blocks the target",+ output_path="output.login_gate_blocks_target",+ expected_output_value=True,+ expected_output_shape="goal_judgment_boolean",+ requested_output_evidence_source="independent_run_evidence",+ judgment_truth_condition=JudgmentTruthCondition(+ predicate="login_gate_blocks_target", polarity_when_holds=True+ ),+ )+ ]+ )++ rendered = _render(active_completion_criteria=active)++ assert '"judgment_predicate": "login_gate_blocks_target"' in rendered+ assert '"judgment_polarity_when_holds": true' in rendered+class TestRawSecretBackstop:@pytest.mark.parametrize(