Skip to content

armgit / fara

Public Imported from GitHub
0

Fara1.5 – A family of frontier computer use agent models

Add CUAVerifierBench dataset (build script + README + repo references)

corby8cd48183e0ff
diff --git a/README.md b/README.md
index df413eb..2fef204 100644
--- a/README.md
+++ b/README.md
@@ -8,6 +8,7 @@
[![Hugging Face Model](https://img.shields.io/badge/🤗-Model-yellow)](https://huggingface.co/microsoft/Fara-7b)
[![Foundry](https://img.shields.io/badge/Azure-Foundry-0089D6)](https://aka.ms/foundry-fara-7b)
[![Dataset](https://img.shields.io/badge/🤗-WebTailBench%20Dataset-orange)](https://huggingface.co/datasets/microsoft/WebTailBench)
+[![Dataset](https://img.shields.io/badge/🤗-CUAVerifierBench-orange)](https://huggingface.co/datasets/microsoft/CUAVerifierBench)
[![Paper](https://img.shields.io/badge/Paper-2511.19663-red)](https://arxiv.org/abs/2511.19663)
</div>
@@ -16,6 +17,15 @@
## Updates
+* **2026-04-19** — Released **[CUAVerifierBench](https://huggingface.co/datasets/microsoft/CUAVerifierBench)**,
+ a human-annotated benchmark for evaluating CUA verifiers (i.e. judges that
+ score agent trajectories). Two splits — `fara7b_om2w_browserbase` (106
+ Fara-7B Online-Mind2Web/Browserbase trajectories, ~2 reviewers each) and
+ `internal` (154 trajectories from a heldout aurora-v2 task suite) —
+ with per-judge UV-blind / UV-informed labels, Universal Verifier
+ outputs, and legacy verifier outputs side-by-side. See the new
+ [`cuaverifierbench/`](cuaverifierbench/) directory in this repo for
+ the build script.
* **2026-04-18** — Removed the `autogen-core` / `autogen-ext` dependency
from `webeval`; chat completion clients are now self-contained under
`webeval/src/webeval/oai_clients/`. No more autogen submodule install
@@ -167,6 +177,35 @@ We are releasing **[WebTailBench](https://huggingface.co/datasets/microsoft/WebT
- Task Verification pipeline for LLM-as-a-judge evaluation
- Official human annotations of WebTailBench (in partnership with BrowserBase)
+### CUAVerifierBench: Evaluating the Verifiers Themselves
+
+While WebTailBench measures *agents*, **[CUAVerifierBench](https://huggingface.co/datasets/microsoft/CUAVerifierBench)** measures the *judges that score those agents*. Each row pairs a Fara-7B agent trajectory (instruction, screenshots, web_surfer log, final answer) with one human reviewer's verdict, plus the verdicts produced by the **Universal Verifier (`MMRubricAgent`)** and several legacy verifiers — so researchers can compute verifier–human agreement (Cohen's κ, accuracy, F1) on a fixed corpus and iterate on new judge prompts / architectures against a frozen ground-truth set.
+
+The dataset is exposed as two HuggingFace configs joinable on `task_id`:
+
+| Config | Granularity | Contents |
+|---|---|---|
+| `trajectories` | one row per task | instruction, screenshots, web_surfer log, verifier outputs, task-level human aggregates |
+| `annotations` | one row per (task, judge) | per-reviewer outcome / process labels and free-text justifications |
+
+Two splits ship today:
+
+| Split | Source | Trajectories | Annotation rows |
+|---|---|---|---|
+| `fara7b_om2w_browserbase` | Fara-7B trajectories on Online-Mind2Web tasks executed via Browserbase | 106 | 215 (≈2 reviewers/task; UV-blind **and** UV-informed stages) |
+| `internal` | Heldout aurora-v2 task suite scored with the same WebSurfer + verifier stack | 154 | 154 (1 reviewer/task; UV-blind only) |
+
+Reviewer identities are anonymized as `Judge1` … `JudgeN` using a single shared map across both splits. The build script that produced the dataset (with full schema + provenance) lives at [`cuaverifierbench/build_dataset.py`](cuaverifierbench/build_dataset.py); see the [dataset README](https://huggingface.co/datasets/microsoft/CUAVerifierBench/blob/main/README.md) for the full column list.
+
+```python
+from datasets import load_dataset
+
+trajs = load_dataset("microsoft/CUAVerifierBench", "trajectories",
+ split="fara7b_om2w_browserbase")
+anns = load_dataset("microsoft/CUAVerifierBench", "annotations",
+ split="fara7b_om2w_browserbase")
+```
+
### Evaluation Infrastructure
Our evaluation setup leverages:
diff --git a/cuaverifierbench/README.md b/cuaverifierbench/README.md
new file mode 100644
index 0000000..0199657
--- /dev/null
+++ b/cuaverifierbench/README.md
@@ -0,0 +1,187 @@
+---
+language:
+- en
+license: mit
+size_categories:
+- n<1K
+pretty_name: CUAVerifierBench
+tags:
+- cua
+- agent-evaluation
+- verifier
+- arxiv:2604.06240
+task_categories:
+- image-text-to-text
+configs:
+- config_name: trajectories
+ data_files:
+ - split: fara7b_om2w_browserbase
+ path: trajectories/fara7b_om2w_browserbase-*
+ - split: internal
+ path: trajectories/internal-*
+- config_name: annotations
+ data_files:
+ - split: fara7b_om2w_browserbase
+ path: annotations/fara7b_om2w_browserbase-*
+ - split: internal
+ path: annotations/internal-*
+---
+
+# CUAVerifierBench: A Human-Annotated Benchmark for Computer-Using-Agent Verifiers
+
+[![Microsoft](https://img.shields.io/badge/Microsoft-Project-0078D4?logo=microsoft)](https://aka.ms/msaif/fara)
+[![Hugging Face Model](https://img.shields.io/badge/🤗-Model-yellow)](https://huggingface.co/microsoft/fara-7b)
+[![Github](https://img.shields.io/badge/Github-181717?logo=github&logoColor=white)](https://github.com/microsoft/fara)
+
+Universal Verifier paper: *The Art of Building Verifiers for Computer Use Agents*
+
+## Dataset Summary
+
+**CUAVerifierBench** is an evaluation benchmark for **verifiers** of computer-using agents (CUAs) — i.e. judges that read an agent's trajectory (screenshots + actions + final answer) and decide whether the task was completed correctly. Where benchmarks like WebTailBench measure *agents*, CUAVerifierBench measures the *judges that score those agents*.
+
+Each row pairs a Fara-7B agent trajectory with one human reviewer's verdict, plus the verdicts produced by the **Universal Verifier (MMRubricAgent)** and several legacy verifiers. Researchers can use the dataset to:
+
+- Compute verifier–human agreement (Cohen's κ, accuracy, F1) on a fixed corpus of trajectories
+- Study disagreement between judges and how it changes when reviewers see the verifier's output (the "UV-informed" stage)
+- Iterate on new verifier prompts/architectures against a frozen ground-truth set
+
+## Splits
+
+Both configs (`trajectories`, `annotations`) carry the same two splits:
+
+| Split | Source | Trajectories | Annotation rows | Annotation stages |
+|---|---|---|---|---|
+| `fara7b_om2w_browserbase` | Fara-7B trajectories on the [Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web) tasks executed via the [Browserbase](https://www.browserbase.com/) remote browser | 106 | 215 (≈2 reviewers/task) | UV-blind **and** UV-informed |
+| `internal` | Microsoft-internal task suite — heldout aurora-v2 task definitions evaluated with the same WebSurfer + verifier stack | 154 | 154 (1 reviewer/task) | UV-blind only |
+
+The two splits share the same column schema. The `internal` split was annotated in a single UV-blind stage, so its `informed_*` fields and `human_process_score` are empty / null.
+
+## Dataset Structure
+
+The dataset is exposed as **two HuggingFace configs** that are joinable on `task_id`:
+
+| Config | Granularity | Contents |
+|---|---|---|
+| `trajectories` | one row per task | The agent run — instruction, screenshots, web_surfer log, final answer, plus all verifier outputs and task-level human aggregates |
+| `annotations` | one row per (task, judge) | Free-text and structured human judgments from one reviewer |
+
+Reviewer identities are anonymized as `Judge1` … `JudgeN` using a single shared map across both splits — the same human always gets the same `Judge` ID.
+
+Storing screenshots only in `trajectories` (rather than duplicating across judges) cuts the on-disk size roughly in half.
+
+### Config: `trajectories`
+
+| Field | Type | Description |
+|---|---|---|
+| `task_id` | string | **PK.** Online-Mind2Web task identifier (e.g. `Adidas--11857213`) |
+| `instruction` | string | Natural-language task given to the agent |
+| `init_url` | string | Starting URL |
+| `start_timestamp`, `end_timestamp` | string | Wall-clock bounds of the run |
+| `final_answer` | string | The agent's submitted answer (or `<no_answer>` if it never called `terminate`) |
+| `is_aborted` | bool | Whether the run was aborted before completion |
+| `web_surfer_log` | string | Full JSONL action/observation log from `web_surfer.log` |
+| `screenshots` | sequence of `Image` | Inline PNG screenshots in chronological order, decoded to PIL automatically |
+| `n_screenshots` | int32 | Length of the `screenshots` list |
+| `gpt_eval_json` | string | Raw JSON of the original Online-Mind2Web GPT judge verdict |
+| `uv_rubric_score` | float32 | **Universal Verifier (current)** rubric score in [0, 1] |
+| `uv_outcome_success` | int32 | **Universal Verifier (current)** binary outcome verdict |
+| `mm_is_success` | int32 | **Legacy (deprecated)** — verdict from the original WebTailBench multimodal grounded verifier (see note below) |
+| `verifier_is_success` | int32 | **Legacy (deprecated)** — verdict from the original WebTailBench text-only task verifier (see note below) |
+| `final_human_outcome_label` | int32 | Final adjudicated outcome label across all reviewers of this task |
+| `final_human_process_label` | int32 | Final adjudicated process label across all reviewers of this task |
+| `median_human_rubric_score_agnostic` | float32 | Median of UV-blind process scores across reviewers |
+| `majority_human_outcome_vote` | int32 | Majority vote of UV-blind outcome labels |
+
+> **About the legacy verifiers.** `mm_is_success` and `verifier_is_success` come from the **original WebTailBench verifier suite** used in the Fara-7B tech report (a 3-judge ensemble: text-only task verifier, multimodal grounded verifier, early rubric agent). The entire suite has since been **deprecated and replaced by the [Universal Verifier (MMRubricAgent)](https://github.com/microsoft/fara/blob/main/webeval/src/webeval/rubric_agent/mm_rubric_agent.py)** in [`microsoft/fara`](https://github.com/microsoft/fara). They are included only for backwards-compatible analysis against numbers from the original Fara-7B paper. **New work should use `uv_rubric_score` / `uv_outcome_success`.**
+
+### Config: `annotations`
+
+| Field | Type | Description |
+|---|---|---|
+| `task_id` | string | **FK** → `trajectories.task_id` |
+| `annotator` | string | Anonymized reviewer (`Judge1` … `Judge6`) |
+| `human_judgement_outcome` | string | UV-blind outcome label (`Correct` / `Incorrect` / etc.) |
+| `human_judgement_process` | string | UV-blind process label |
+| `human_process_score` | float32 | UV-blind continuous process score in [0, 1] |
+| `outcome_comment` | string | UV-blind free-text justification for the outcome label |
+| `process_comment` | string | UV-blind free-text justification for the process label |
+| `informed_outcome_agreement` | string | UV-informed: agreement with the Universal Verifier's outcome verdict |
+| `informed_process_agreement` | string | UV-informed: agreement with the Universal Verifier's process verdict |
+| `informed_outcome_comment` | string | UV-informed free-text justification |
+| `informed_process_comment` | string | UV-informed free-text justification |
+
+**UV-blind vs. UV-informed.** Reviewers labeled each trajectory in two stages: first *without* seeing any verifier output (`human_*` and `*_comment` fields), then *after* being shown the Universal Verifier's verdict (`informed_*` fields).
+
+## Loading
+
+Each config is loaded separately and joined on `task_id`. Pass either `fara7b_om2w_browserbase` or `internal` as the split:
+
+```python
+from datasets import load_dataset
+
+split = "fara7b_om2w_browserbase" # or "internal"
+trajs = load_dataset("microsoft/CUAVerifierBench", "trajectories", split=split)
+anns = load_dataset("microsoft/CUAVerifierBench", "annotations", split=split)
+
+# Per-judge analysis: join in pandas
+import pandas as pd
+df = anns.to_pandas().merge(trajs.to_pandas(), on="task_id")
+
+# Or look up trajectories on demand:
+by_id = {r["task_id"]: r for r in trajs}
+print(by_id[anns[0]["task_id"]]["screenshots"][0]) # PIL.Image
+```
+
+## Dataset Creation
+
+### Source trajectories
+
+Trajectories were generated by running [Fara-7B](https://huggingface.co/microsoft/fara-7b) on the public [Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web) task set, executed inside a [Browserbase](https://www.browserbase.com/)-hosted Chromium instance. Each trajectory contains the screenshots the model saw, the structured actions it issued, and the final answer it submitted.
+
+### Annotation protocol
+
+Each task was independently reviewed by ~2 human annotators in two stages:
+
+1. **UV-blind (agnostic)** — Reviewers read the instruction and trajectory and assign outcome / process labels and a continuous process score, *without* seeing any verifier output.
+2. **UV-informed** — Reviewers are then shown the Universal Verifier's verdict and asked whether they agree, with free-text justifications.
+
+Reviewer identities are anonymized as `Judge1`…`Judge6`.
+
+### Universal Verifier outputs
+
+For each trajectory we also include the verdicts of the **MMRubricAgent** (the Universal Verifier shipped with Fara) and two legacy verifiers, so users can directly compute verifier–human agreement.
+
+## Considerations for Using the Data
+
+### Intended Use
+
+- Evaluating CUA verifiers against human judgment
+- Studying inter-annotator agreement and the effect of showing model verdicts to humans
+- Developing new judge prompts / architectures for trajectory evaluation
+
+### Limitations
+
+- 106 tasks is a relatively small corpus; results should be reported with confidence intervals
+- All trajectories come from a single agent (Fara-7B); verifier behavior on trajectories from other agents may differ
+- Tasks inherit the temporal validity and domain biases of Online-Mind2Web
+
+### Licensing
+
+MIT License
+
+### Citation
+
+If you use CUAVerifierBench in your research, please cite the Universal Verifier paper:
+
+```bibtex
+@article{UniversalVerifier2026,
+ title={The Art of Building Verifiers for Computer Use Agents},
+ journal={arXiv preprint arXiv:2604.06240},
+ year={2026},
+ url={https://arxiv.org/abs/2604.06240v1}
+}
+```
+
+### Contributions
+
+Created by Microsoft Research AI Frontiers.
diff --git a/cuaverifierbench/build_dataset.py b/cuaverifierbench/build_dataset.py
new file mode 100644
index 0000000..9990a51
--- /dev/null
+++ b/cuaverifierbench/build_dataset.py
@@ -0,0 +1,446 @@
+"""Build the CUAVerifierBench HuggingFace dataset locally.
+
+Emits two HF *configs* — ``trajectories`` and ``annotations`` — each with
+two splits: ``fara7b_om2w_browserbase`` and ``internal``. Tables are
+joinable on ``task_id``:
+
+* ``trajectories`` — one row per task (instruction, screenshots,
+ web_surfer log, verifier outputs, task-level aggregates).
+* ``annotations`` — one row per (task_id, annotator) human review.
+
+Annotator names are anonymized to ``Judge1`` … ``JudgeN`` using a single
+shared map across both splits, so the same human gets the same ID.
+
+Run:
+ python build_dataset.py [--out OUT_DIR] [--push REPO_ID]
+"""
+
+from __future__ import annotations
+
+import argparse
+import collections
+import csv
+import json
+import logging
+import re
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+from datasets import Dataset, DatasetDict, Features, Image, Sequence, Value
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
+logger = logging.getLogger("build_cuaverifierbench")
+
+# --- om2w_browserbase split -------------------------------------------------
+OM2W_ANN_TSV = Path(
+ "/mnt/users/wangzhe/code/agento/webeval_next/verify_verifiers/"
+ "trajectory_human_annotations/browserbase_om2w_v2/annotations_fixed.tsv"
+)
+OM2W_TRAJ_ROOT = Path("/data/data/Agento/eval/browserbase-om2w/traj")
+OM2W_SCORE_FILE = "gpt_eval.json"
+
+# --- internal split ---------------------------------------------------------
+INTERNAL_ANN_TSV = Path(
+ "/mnt/users/wangzhe/code/agento/webeval_next/verify_verifiers/"
+ "trajectory_human_annotations/"
+ "internal_combined_annotations.outcome_process.fixed.merged.tsv"
+)
+INTERNAL_TRAJ_ROOT = Path(
+ "/data/data/Agento/eval/verify_verifiers/human_annotated_traj/"
+ "internal_combined_traj/runs/"
+ "WebSurfer-orca_qwen25vl_aurorav2_solver_history-100-max_n_images-3/"
+ "models_dummy/corbyrosset/"
+ "HoldOut__data_data_Agento_eval_verify_verifiers_human_annotated_traj_"
+ "internal_combined_traj/"
+ "gpt5.2v4_MMv20_gpt5.2_scored_soft_outcome_verifier/traj"
+)
+INTERNAL_SCORE_FILE = "0.8-5-3.json"
+
+SCREENSHOT_RE = re.compile(r"^screenshot(\d+)\.png$")
+
+
+# ---------------------------------------------------------------------------
+# Trajectory loader
+# ---------------------------------------------------------------------------
+def _load_trajectory(traj_dir: Path, score_filename: str) -> Dict[str, Any]:
+ out: Dict[str, Any] = {}
+
+ td_path = traj_dir / "task_data.json"
+ task_data = json.loads(td_path.read_text()) if td_path.exists() else {}
+ out["instruction"] = task_data.get("task_proposal") or task_data.get("task_summary") or ""
+ out["init_url"] = task_data.get("init_url") or ""
+ out["start_timestamp"] = task_data.get("start_timestamp") or ""
+ out["end_timestamp"] = task_data.get("end_timestamp") or ""
+
+ fa_files = list(traj_dir.glob("*_final_answer.json"))
+ if fa_files:
+ fa = json.loads(fa_files[0].read_text())
+ out["final_answer"] = fa.get("final_answer", "<no_answer>")
+ out["is_aborted"] = bool(fa.get("is_aborted", False))
+ else:
+ out["final_answer"] = "<no_answer>"
+ out["is_aborted"] = True
+
+ wsl_path = traj_dir / "web_surfer.log"
+ out["web_surfer_log"] = wsl_path.read_text(encoding="utf-8") if wsl_path.exists() else ""
+
+ score_path = traj_dir / "scores" / score_filename
+ out["gpt_eval_json"] = score_path.read_text(encoding="utf-8") if score_path.exists() else ""
+
+ screenshots: List[tuple] = []
+ for fname in traj_dir.iterdir():
+ m = SCREENSHOT_RE.match(fname.name)
+ if m:
+ screenshots.append((int(m.group(1)), str(fname)))
+ screenshots.sort()
+ out["screenshots"] = [p for _, p in screenshots]
+ out["n_screenshots"] = len(screenshots)
+ return out
+
+
+def _parse_uv_outcome_from_score_json(raw: str) -> Optional[int]:
+ """Extract rubric_outcome_verification.output_success (bool) → int."""
+ if not raw:
+ return None
+ try:
+ outer = json.loads(raw)
+ inner = json.loads(outer.get("gpt_response_text", "{}"))
+ ov = inner.get("rubric_outcome_verification") or {}
+ v = ov.get("output_success")
+ if v is None:
+ return None
+ return int(bool(v))
+ except (ValueError, TypeError, json.JSONDecodeError):
+ return None
+
+
+# ---------------------------------------------------------------------------
+# Per-row helpers
+# ---------------------------------------------------------------------------
+def _f(d: Dict[str, Any], name: str) -> str:
+ return (d.get(name) or "").strip()
+
+
+def _floatish(d: Dict[str, Any], name: str) -> Optional[float]:
+ v = _f(d, name)
+ if v == "":
+ return None
+ try:
+ return float(v)
+ except ValueError:
+ return None
+
+
+def _binint(d: Dict[str, Any], name: str) -> Optional[int]:
+ v = _f(d, name)
+ if v == "":
+ return None
+ s = v.lower()
+ if s in ("true", "1", "1.0", "yes", "correct"):
+ return 1
+ if s in ("false", "0", "0.0", "no", "incorrect"):
+ return 0
+ try:
+ return int(float(v))
+ except ValueError:
+ return None
+
+
+def _normalize_outcome_label(raw: str) -> str:
+ """Map numeric ('1.0'/'0.0') and string variants to 'Correct'/'Incorrect'."""
+ s = raw.strip().lower()
+ if s in ("1", "1.0", "correct", "true", "yes"):
+ return "Correct"
+ if s in ("0", "0.0", "incorrect", "false", "no"):
+ return "Incorrect"
+ return raw # leave anything else (e.g. 'partial') as-is
+
+
+# ---------------------------------------------------------------------------
+# Annotation TSV adapters: normalize each TSV's columns into a canonical dict
+# ---------------------------------------------------------------------------
+def _read_om2w_annotations(path: Path) -> List[Dict[str, Any]]:
+ with open(path, encoding="utf-8") as f:
+ rows = list(csv.DictReader(f, delimiter="\t"))
+ logger.info(f"loaded {len(rows)} om2w_browserbase annotation rows from {path.name}")
+ return rows # already in canonical om2w schema
+
+
+def _read_internal_annotations(path: Path) -> List[Dict[str, Any]]:
+ """Map internal TSV cols → same canonical keys used by the om2w loader.
+
+ Internal TSV has no UV-informed stage and no continuous process score;
+ those fields are emitted as empty strings so downstream coercion → None.
+ """
+ with open(path, encoding="utf-8") as f:
+ # QUOTE_NONE: NOTES sometimes contains stray double-quotes
+ rows = list(csv.DictReader(f, delimiter="\t", quoting=csv.QUOTE_NONE))
+ # Strip surrounding quotes from column names (QUOTE_NONE keeps them literal).
+ rows = [{k.strip('"'): v for k, v in r.items()} for r in rows]
+ out: List[Dict[str, Any]] = []
+ for r in rows:
+ # Some internal files include a header artifact row; skip if Annotator literally is "Annotator".
+ if (r.get("Annotator") or "").strip() == "Annotator":
+ continue
+ if not (r.get("TASK") or "").strip():
+ continue
+ out.append({
+ "TASK": r.get("TASK", ""),
+ "VERIFIER": r.get("Annotator", ""),
+ "HUMAN JUDGEMENT OUTCOME": _normalize_outcome_label(r.get("HUMAN JUDGEMENT OUTCOME", "")),
+ "HUMAN JUDGEMENT PROCESS": _normalize_outcome_label(r.get("HUMAN JUDGEMENT PROCESS", "")),
+ "HUMAN PROCESS SCORE": "", # not in internal
+ "OUTCOME_COMMENT": r.get("NOTES", ""),
+ "PROCESS_COMMENT": "",
+ "INFORMED_OUTCOME_AGREEMENT": "",
+ "INFORMED_PROCESS_AGREEMENT": "",
+ "INFORMED_OUTCOME_COMMENT": "",
+ "INFORMED_PROCESS_COMMENT": "",
+ "rubric_score": r.get("rubric_score", ""),
+ "outcome_success": "", # parsed from score JSON instead
+ "mm_is_success": r.get('MM_verifier_response[""is_success""]', "")
+ or r.get('MM_verifier_response["is_success"]', ""),
+ "verifier_is_success": r.get("Verifier_bool", ""),
+ })
+ logger.info(f"loaded {len(out)} internal annotation rows from {path.name}")
+ return out
+
+
+# ---------------------------------------------------------------------------
+# Schemas
+# ---------------------------------------------------------------------------
+_TRAJ_FEATURES = Features({
+ "task_id": Value("string"),
+ "instruction": Value("string"),
+ "init_url": Value("string"),
+ "start_timestamp": Value("string"),
+ "end_timestamp": Value("string"),
+ "final_answer": Value("string"),
+ "is_aborted": Value("bool"),
+ "web_surfer_log": Value("string"),
+ "screenshots": Sequence(Image()),
+ "n_screenshots": Value("int32"),
+ "gpt_eval_json": Value("string"),
+ "uv_rubric_score": Value("float32"),
+ "uv_outcome_success": Value("int32"),
+ "mm_is_success": Value("int32"),
+ "verifier_is_success": Value("int32"),
+ "final_human_outcome_label": Value("int32"),
+ "final_human_process_label": Value("int32"),
+ "median_human_rubric_score_agnostic": Value("float32"),
+ "majority_human_outcome_vote": Value("int32"),
+})
+
+_ANN_FEATURES = Features({
+ "task_id": Value("string"),
+ "annotator": Value("string"),
+ "human_judgement_outcome": Value("string"),
+ "human_judgement_process": Value("string"),
+ "human_process_score": Value("float32"),
+ "outcome_comment": Value("string"),
+ "process_comment": Value("string"),
+ "informed_outcome_agreement": Value("string"),
+ "informed_process_agreement": Value("string"),
+ "informed_outcome_comment": Value("string"),
+ "informed_process_comment": Value("string"),
+})
+
+
+# ---------------------------------------------------------------------------
+# Row builders
+# ---------------------------------------------------------------------------
+def _majority(values: List[Optional[int]]) -> Optional[int]:
+ vals = [v for v in values if v is not None]
+ if not vals:
+ return None
+ return collections.Counter(vals).most_common(1)[0][0]
+
+
+def _trajectory_row(
+ task_id: str,
+ traj: Dict[str, Any],
+ anns: List[Dict[str, Any]],
+ *,
+ derive_aggregates: bool,
+ uv_outcome_from_score_json: bool,
+) -> Dict[str, Any]:
+ head = anns[0]
+ if uv_outcome_from_score_json:
+ uv_out = _parse_uv_outcome_from_score_json(traj["gpt_eval_json"])
+ else:
+ uv_out = _binint(head, "outcome_success")
+
+ if derive_aggregates:
+ outcome_votes = [_binint(a, "HUMAN JUDGEMENT OUTCOME") for a in anns]
+ process_votes = [_binint(a, "HUMAN JUDGEMENT PROCESS") for a in anns]
+ final_outcome = _majority(outcome_votes)
+ final_process = _majority(process_votes)
+ median_score = None
+ majority_outcome = final_outcome
+ else:
+ final_outcome = _binint(head, "final_human_outcome_label")
+ final_process = _binint(head, "final_human_process_label")
+ median_score = _floatish(head, "median_human_rubric_score_agnostic")
+ majority_outcome = _binint(head, "majority_human_outcome_vote")
+
+ return {
+ "task_id": task_id,
+ "instruction": traj["instruction"],
+ "init_url": traj["init_url"],
+ "start_timestamp": traj["start_timestamp"],
+ "end_timestamp": traj["end_timestamp"],
+ "final_answer": traj["final_answer"],
+ "is_aborted": traj["is_aborted"],
+ "web_surfer_log": traj["web_surfer_log"],
+ "screenshots": traj["screenshots"],
+ "n_screenshots": traj["n_screenshots"],
+ "gpt_eval_json": traj["gpt_eval_json"],
+ "uv_rubric_score": _floatish(head, "rubric_score"),
+ "uv_outcome_success": uv_out,
+ "mm_is_success": _binint(head, "mm_is_success"),
+ "verifier_is_success": _binint(head, "verifier_is_success"),
+ "final_human_outcome_label": final_outcome,
+ "final_human_process_label": final_process,
+ "median_human_rubric_score_agnostic": median_score,
+ "majority_human_outcome_vote": majority_outcome,
+ }
+
+
+def _annotation_row(task_id: str, annotator: str, ann: Dict[str, Any]) -> Dict[str, Any]:
+ return {
+ "task_id": task_id,
+ "annotator": annotator,
+ "human_judgement_outcome": _f(ann, "HUMAN JUDGEMENT OUTCOME"),
+ "human_judgement_process": _f(ann, "HUMAN JUDGEMENT PROCESS"),
+ "human_process_score": _floatish(ann, "HUMAN PROCESS SCORE"),
+ "outcome_comment": _f(ann, "OUTCOME_COMMENT"),
+ "process_comment": _f(ann, "PROCESS_COMMENT"),
+ "informed_outcome_agreement": _f(ann, "INFORMED_OUTCOME_AGREEMENT"),
+ "informed_process_agreement": _f(ann, "INFORMED_PROCESS_AGREEMENT"),
+ "informed_outcome_comment": _f(ann, "INFORMED_OUTCOME_COMMENT"),
+ "informed_process_comment": _f(ann, "INFORMED_PROCESS_COMMENT"),
+ }
+
+
+# ---------------------------------------------------------------------------
+# Per-split build
+# ---------------------------------------------------------------------------
+def _build_split(
+ *,
+ split_name: str,
+ annotations: List[Dict[str, Any]],
+ traj_root: Path,
+ score_filename: str,
+ alias_map: Dict[str, str],
+ derive_aggregates: bool,
+ uv_outcome_from_score_json: bool,
+) -> Dict[str, Dataset]:
+ annotated_task_ids = sorted({(a.get("TASK") or "").strip() for a in annotations if a.get("TASK")})
+ have_traj = {tid for tid in annotated_task_ids if (traj_root / tid).is_dir()}
+ missing = sorted(set(annotated_task_ids) - have_traj)
+ if missing:
+ logger.warning(f"[{split_name}] {len(missing)} annotated task_ids have no trajectory dir: {missing[:5]}…")
+ logger.info(f"[{split_name}] task_ids w/ both annotation and trajectory: {len(have_traj)} / {len(annotated_task_ids)}")
+
+ anns_by_task: Dict[str, List[Dict[str, Any]]] = {}
+ for ann in annotations:
+ tid = (ann.get("TASK") or "").strip()
+ if tid in have_traj:
+ anns_by_task.setdefault(tid, []).append(ann)
+
+ traj_rows: List[Dict[str, Any]] = []
+ ann_rows: List[Dict[str, Any]] = []
+ for tid in sorted(have_traj):
+ traj = _load_trajectory(traj_root / tid, score_filename)
+ traj_rows.append(_trajectory_row(
+ tid, traj, anns_by_task[tid],
+ derive_aggregates=derive_aggregates,
+ uv_outcome_from_score_json=uv_outcome_from_score_json,
+ ))
+ for ann in anns_by_task[tid]:
+ judge = alias_map.get(_f(ann, "VERIFIER"), _f(ann, "VERIFIER"))
+ ann_rows.append(_annotation_row(tid, judge, ann))
+
+ logger.info(f"[{split_name}] emitted {len(traj_rows)} trajectory rows, {len(ann_rows)} annotation rows")
+ return {
+ "trajectories": Dataset.from_list(traj_rows, features=_TRAJ_FEATURES),
+ "annotations": Dataset.from_list(ann_rows, features=_ANN_FEATURES),
+ }
+
+
+def _build_alias_map(*ann_lists: List[Dict[str, Any]]) -> Dict[str, str]:
+ """Single shared map across all splits: same human → same JudgeN."""
+ raw = set()
+ for anns in ann_lists:
+ for a in anns:
+ name = (a.get("VERIFIER") or "").strip()
+ if name:
+ raw.add(name)
+ return {name: f"Judge{i + 1}" for i, name in enumerate(sorted(raw))}
+
+
+def build_all() -> Dict[str, DatasetDict]:
+ """Returns {config_name: DatasetDict({split_name: Dataset})}."""
+ om2w_anns = _read_om2w_annotations(OM2W_ANN_TSV)
+ internal_anns = _read_internal_annotations(INTERNAL_ANN_TSV)
+
+ alias_map = _build_alias_map(om2w_anns, internal_anns)
+ logger.info(f"anonymized {len(alias_map)} unique annotators → Judge1..Judge{len(alias_map)}")
+ logger.info(f" alias map: {alias_map}")
+
+ om2w_split = _build_split(
+ split_name="fara7b_om2w_browserbase",
+ annotations=om2w_anns,
+ traj_root=OM2W_TRAJ_ROOT,
+ score_filename=OM2W_SCORE_FILE,
+ alias_map=alias_map,
+ derive_aggregates=False, # om2w TSV has pre-computed aggregates
+ uv_outcome_from_score_json=False, # om2w TSV has outcome_success column
+ )
+ internal_split = _build_split(
+ split_name="internal",
+ annotations=internal_anns,
+ traj_root=INTERNAL_TRAJ_ROOT,
+ score_filename=INTERNAL_SCORE_FILE,
+ alias_map=alias_map,
+ derive_aggregates=True, # compute majority votes from per-judge labels
+ uv_outcome_from_score_json=True, # parse from rubric_outcome_verification.output_success
+ )
+
+ return {
+ "trajectories": DatasetDict({
+ "fara7b_om2w_browserbase": om2w_split["trajectories"],
+ "internal": internal_split["trajectories"],
+ }),
+ "annotations": DatasetDict({
+ "fara7b_om2w_browserbase": om2w_split["annotations"],
+ "internal": internal_split["annotations"],
+ }),
+ }
+
+
+def main():
+ p = argparse.ArgumentParser()
+ p.add_argument("--out", default="/tmp/cuaverifierbench_local", help="Local save_to_disk root")
+ p.add_argument("--push", default=None, help="HF repo (e.g. microsoft/CUAVerifierBench)")
+ args = p.parse_args()
+
+ configs = build_all()
+
+ out_dir = Path(args.out)
+ out_dir.mkdir(parents=True, exist_ok=True)
+ for cfg_name, dd in configs.items():
+ local = out_dir / cfg_name
+ dd.save_to_disk(str(local))
+ for split_name, ds in dd.items():
+ logger.info(f" config={cfg_name!r} split={split_name!r} rows={len(ds)} → {local}")
+
+ if args.push:
+ for cfg_name, dd in configs.items():
+ logger.info(f"push_to_hub({args.push!r}, config={cfg_name!r}) …")
+ dd.push_to_hub(args.push, config_name=cfg_name, private=True)
+ logger.info(f"pushed to https://huggingface.co/datasets/{args.push}")
+
+
+if __name__ == "__main__":
+ main()