Skip to content

The open-source app everyone uses to manage agents at work

docs: add eval guide, authoring skills, and public history hub (#13535)

Dotta18989a9e7376
diff --git a/.agents/skills/add-product-e2e-eval/SKILL.md b/.agents/skills/add-product-e2e-eval/SKILL.md
new file mode 100644
index 0000000000..686fa44f13
--- /dev/null
+++ b/.agents/skills/add-product-e2e-eval/SKILL.md
@@ -0,0 +1,84 @@
+---
+name: add-product-e2e-eval
+description: Add or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.
+---
+
+# Add a Product E2E Eval
+
+Use this skill for **Product E2E Evals**: real Chromium, Paperclip server,
+database, runner, provider, and optionally Daytona. Everyday Workflows are in
+this family even when their packaged results are imported into Evalbook. Runner
+protocol cases against the mock control plane belong in
+[add-runner-eval](../add-runner-eval/SKILL.md).
+
+Locate the repository using `PAPERCLIP_ROOT` when supplied, or
+`git rev-parse --show-toplevel` from a checkout. From outside Git, inspect
+workspace roots such as `~/paperclipai/paperclip`; verify the selected root
+contains `tests/runner-e2e` and `packages/paperclip-runner`. Run commands from
+that repository root. The copied skill may live outside the checkout.
+Read `doc/evals.md`, then the authoritative
+`tests/runner-e2e/README.md`, `FIXTURES.md`, `SECURITY.md`, and
+`EVERYDAY-WORKFLOWS.md` for the selected area. Inspect the nearest existing
+catalog entry, case, harness flow, matcher, evidence writer, and report test
+before changing anything. Keep the user journey on production browser/API
+surfaces; do not add private runner hooks or direct fixture database writes.
+
+Define one bounded workflow with a clear user outcome, durable state
+assertions, and independent evidence. Declare profile, environment, expected
+provider turns, timeout, cleanup, screenshots, artifact checks, and billing
+scope. Keep credentials and secrets out of catalog data, screenshots, fixture
+metadata, logs, and tracked files. Daytona images must be immutable digest
+references. Use the existing result validator, failure classifier, screenshot
+policy, and Product E2E report pipeline rather than duplicating them.
+
+Register the workflow in `tests/runner-e2e/catalog.ts` and the relevant
+`fixture-registry.ts` paths. Everyday cases and actions live in
+`everyday-cases.ts` and `everyday-flow.ts`; match the existing suite's structure.
+Wire its
+profile/environment/case IDs through existing selector and matcher tables, and
+add report/catalog coverage tests where the surrounding suite does so. Calibrate
+new grading assertions with a valid outcome and a plausible wrong outcome;
+missing evidence must not produce a pass. Confirm
+discovery before running it:
+
+```sh
+pnpm test:e2e:runner -- --list --suite <suite-name>
+pnpm test:e2e:runner -- --list --suite everyday-workflows
+```
+
+`--all` intentionally excludes the manual `everyday-workflows` suite and other
+explicit-only cells. Use the exact suite or execution ID for those. The Product
+E2E generator is `pnpm test:e2e:runner:report`; an Everyday Workflows result may
+also be imported into Evalbook with its canonical importer, but it remains a
+Product E2E run and should use its packaged dashboard/report first.
+
+Run credential-free checks first:
+
+```sh
+pnpm test:e2e:runner:typecheck
+pnpm test:e2e:runner:unit
+pnpm test:e2e:runner -- --list
+```
+
+For an authorized live check, use the smallest explicit local `--id` selector.
+Select Daytona only with the configured immutable image and credentials. A
+full `--all` campaign is paid and is for the governed workflow. Verify results
+from the packaged attempt evidence and dashboard; passing model-authored tests
+cannot override the independent oracle.
+
+Classify a completed wrong workflow as product or model/provider behavior as
+the assertions warrant. Startup, transport, and timeout symptoms require
+evidence-based attribution: they may indicate a Paperclip/Runner product bug,
+provider behavior, or infrastructure. Preserve the observed failure and cause
+separately, keep the existing machine grade/classifier unchanged, and retain
+partial attempts, retries, source SHA, catalog/definition digest, model/profile,
+environment, grader version, timing, tokens, runtime estimates, and cost
+coverage. Do not claim qualification from a partial/manual selection.
+
+Update `README.md`, `FIXTURES.md`, `SECURITY.md`, or
+`EVERYDAY-WORKFLOWS.md` when their authoritative contract changes, and link
+from `doc/evals.md`. Do not recreate Evalbook HTML or publish raw trusted
+traces, videos, archives, databases, workspaces, credentials, SVG, provider
+session IDs, or hidden reasoning. Public sanitized fixture conversation, marked
+screenshots, and allowlisted structured evidence are expected when the existing
+publisher permits them.
diff --git a/.agents/skills/add-runner-eval/SKILL.md b/.agents/skills/add-runner-eval/SKILL.md
new file mode 100644
index 0000000000..111a1b2c7a
--- /dev/null
+++ b/.agents/skills/add-runner-eval/SKILL.md
@@ -0,0 +1,72 @@
+---
+name: add-runner-eval
+description: Add or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.
+---
+
+# Add a Runner Eval
+
+Use this skill for the **Runner Evals** family: a real Runner/provider session
+against a seeded mock control plane. Product browser/server/database/Daytona
+coverage belongs in [add-product-e2e-eval](../add-product-e2e-eval/SKILL.md).
+
+Locate the Paperclip checkout using `PAPERCLIP_ROOT` when supplied, or
+`git rev-parse --show-toplevel` from a checkout. From outside Git, inspect the
+workspace roots (for example `~/paperclipai/paperclip`) and verify that the
+selected root contains `packages/paperclip-runner` and `tests/runner-e2e`.
+Locate `paperclip-evals` using `PAPERCLIP_EVALS_ROOT` or a discovered sibling;
+a worktree's parent directory need not contain that repository. Read
+`doc/evals.md` and `packages/paperclip-runner/docs/runner-protocol-live-evals.md`,
+then inspect the nearest existing case, roster, schema, and report test before
+editing. Definitions and authored cases belong in the sibling
+`paperclip-evals/evals/paperclip-runner`; Runner integration, aggregation,
+viewer, and publication behavior belongs in `packages/paperclip-runner`.
+Keep the control-plane boundary explicit in names and documentation.
+
+The sibling eval README is the concrete map: cases live under `cases/`,
+company fixtures under `fixtures/`, runtime/model settings under `configs/`,
+selections under `rosters/`, and maintained campaign membership under
+`campaigns/live-direct-full.json`. Update inventory/coverage mappings when the
+program requires them; a new file alone does not join the maintained campaign.
+From the Evals repository root, adapt these provider-free checks to the case
+and roster you changed. Run the reliability-plan validator only when that
+separate plan changes:
+
+```sh
+python3 evals/paperclip-runner/tools/eval_program.py validate \
+ --case evals/paperclip-runner/cases/get-task-context.json \
+ --config evals/paperclip-runner/configs/live-codex-pinned.json
+python3 evals/paperclip-runner/tools/run_live_roster.py validate \
+ --roster evals/paperclip-runner/rosters/live-mini.json --run-id validate-new-case
+python3 evals/paperclip-runner/tools/run_live_campaign.py validate \
+ --campaign evals/paperclip-runner/campaigns/live-direct-full.json
+python3 evals/paperclip-runner/tools/reliability_campaign.py validate \
+ --plan evals/paperclip-runner/campaigns/paperclip-runner-reliability.json
+```
+
+Use nearby positive and negative grader cases/fixtures to calibrate the new
+assertion, including malformed or missing evidence where the grader must fail
+closed. Preserve the existing machine disposition and grade; product,
+model/provider, grading, and infrastructure labels are analytical annotations,
+not instructions to rewrite classifiers.
+
+Author one bounded case with a deterministic semantic assertion and an
+inspectable result. Declare its expected operation, state effect, provider
+lane/profile, timeout and retry policy, and any required evidence. Do not grade
+hidden reasoning, infer success from a provider terminal message, or invent
+conversation/tool evidence. Public output follows the reviewed projection:
+sanitized fixture conversation and allowlisted tool outcomes may be published;
+raw trusted artifacts, credentials, secrets, private references, and hidden
+reasoning may not.
+
+Validate without provider calls first using the commands above and the relevant
+report/render validation documented in the Runner docs. When a live run is
+authorized, pin the Paperclip commit and exact 40-character
+`paperclip-evals` commit, select the smallest useful roster, and retain the
+complete provenance and cost record.
+
+Update authoritative detailed docs when the contract or command changes, then
+link from `doc/evals.md` rather than duplicating the Runner runbook. Keep public
+reports immutable and use the reviewed projection; sanitized fixture
+conversation and allowlisted tool outcomes may be public, while credentials,
+secrets, private references, raw trusted payloads, and hidden reasoning must not
+be exposed.
diff --git a/.agents/skills/paperclip-evals/SKILL.md b/.agents/skills/paperclip-evals/SKILL.md
new file mode 100644
index 0000000000..9735805c80
--- /dev/null
+++ b/.agents/skills/paperclip-evals/SKILL.md
@@ -0,0 +1,63 @@
+---
+name: paperclip-evals
+description: Choose, inspect, validate, and report Paperclip Runner or Product E2E evaluations while preserving evidence, provenance, cost, and failure classification.
+---
+
+# Paperclip evals
+
+Use this skill when a request concerns Paperclip evaluation selection,
+interpretation, evidence, history, or a live run. Read `doc/evals.md` in the
+Paperclip repository first. It defines the two families and their boundaries.
+
+## Discover the repository
+
+Do not assume the skill's installed location is inside a checkout. Locate the
+repo explicitly with `git rev-parse --show-toplevel` from the current directory,
+or inspect likely workspace roots and select the checkout containing
+`package.json`, `tests/runner-e2e`, and `packages/paperclip-runner`. Locate the
+private sibling `paperclip-evals` only when a Runner Eval needs its definitions;
+use an explicit `PAPERCLIP_EVALS_ROOT` or a discovered sibling checkout. Never
+invent a relative path from this copied skill into the repository.
+
+## Route the request
+
+Choose **Runner Evals** for real runner/provider protocol behavior against the
+mock control plane. Authoritative details are in
+`packages/paperclip-runner/docs/runner-protocol-live-evals.md` and the sibling
+`paperclip-evals/evals/paperclip-runner` definitions.
+
+Choose **Product E2E Evals** for real browser/server/database/runner/provider
+workflows, including local and Daytona environments. Read
+`tests/runner-e2e/README.md`, then `FIXTURES.md`, `SECURITY.md`, or
+`EVERYDAY-WORKFLOWS.md` as relevant. Everyday Workflows remain Product E2E
+even when imported into Evalbook. “Headless” is a browser mode, not a family.
+
+## Work safely
+
+Start with read-only catalog inspection and credential-free validation. For
+Product E2E use `pnpm test:e2e:runner:typecheck`,
+`pnpm test:e2e:runner:unit`, and `pnpm test:e2e:runner -- --list`; run one
+explicit cell only when the user has authorized a live/paid run and the needed
+credentials and immutable Daytona image are configured. For Runner Evals use
+the pinned eval revision and the documented workflow/CLI. Never use a partial
+selector as evidence of full coverage.
+
+Keep source revisions, definition/catalog fingerprints, model/profile,
+environment, selected cells, retries, timing, usage/cost coverage, and grader
+version attached to every interpretation. Preserve partial attempts and
+classify failures as product, model/provider behavior, grading/evidence, or
+infrastructure from the observed failure and supported cause. A usable
+completed behavior failure is not infrastructure; missing provider/profile,
+transport, startup, or evidence requires examining the evidence before choosing
+the cause.
+
+Use the existing family generator and viewer. Public projections may contain
+sanitized fixture conversation and allowlisted tool outcomes/evidence; follow
+the family's projection and publisher checks. Do not expose raw trusted
+artifacts, credentials, secrets, private data, provider session IDs, or hidden
+reasoning. A refresh from retained evidence has zero provider calls and remains
+the original measurement with a new presentation. Link the public histories
+and hub from `doc/evals.md` when reporting results.
+
+For adding a case or fixture, use the narrower [add-runner-eval](../add-runner-eval/SKILL.md)
+or [add-product-e2e-eval](../add-product-e2e-eval/SKILL.md) skill.
diff --git a/doc/evals.md b/doc/evals.md
new file mode 100644
index 0000000000..88fedec15e
--- /dev/null
+++ b/doc/evals.md
@@ -0,0 +1,214 @@
+# Paperclip evaluation guide
+
+Paperclip has two live eval families with different questions, owners, and
+evidence. Choose the family before selecting a model, profile, or case.
+
+- **Runner Evals:** real Runner/provider behavior against a seeded mock control
+ plane. Definitions live in `paperclip-evals/evals/paperclip-runner`; see the
+ [direct live protocol evals](../packages/paperclip-runner/docs/runner-protocol-live-evals.md).
+- **Product E2E Evals:** real browser, Paperclip server, database, Runner,
+ provider, and (where selected) Daytona, using an isolated instance and
+ grading oracle. See [`tests/runner-e2e`](../tests/runner-e2e/README.md) and
+ [Everyday Workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md).
+
+Runner Evals answer whether a real runner/provider can perform a bounded
+protocol operation against the expected control-plane contract. Product E2E
+Evals answer whether a person can complete a product workflow through the real
+Paperclip surfaces and whether the resulting artifact and state are usable.
+The names describe the system under test; “headless” is an execution option,
+not an eval category.
+
+## Selecting a family
+
+Use **Runner Evals** for a runner protocol, adapter, transport, native session,
+tool grant, or one-turn provider qualification question. The workflow checks
+out an exact `paperclip-evals` revision, builds the Runner and viewer, runs a
+live roster, and renders the canonical Evalbook report. The control plane is a
+seeded test authority, so a passing result does not prove browser UX, production
+server behavior, database persistence, Daytona behavior, or a real third-party
+mutation.
+
+Use **Product E2E Evals** for browser interaction, issue/task lifecycle,
+approval and clarification UI, project/repository selection, persistence over a
+controller restart, artifact delivery, billing/evidence behavior, or runner
+continuity in local or Daytona environments. The harness creates a fresh
+Paperclip instance per cell and uses public APIs and the production browser
+surface. The suite's [Everyday Workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md)
+are Product E2E even when their results are imported into Evalbook.
+
+Do not combine a partial Runner campaign and a partial Product E2E campaign into
+one score. A campaign is comparable when its definition/grader, model/profile,
+environment, and contract match. The evaluated Paperclip revision may
+intentionally differ for a before/after fix comparison; record it as a
+comparison axis.
+
+## Ownership and codepaths
+
+Runner Evals are owned by the Runner/evals maintainers. Definitions, rosters,
+case prompts, and the report program live in the sibling private repository
+`paperclipai/paperclip-evals`; Runner integration, viewer, aggregation, and
+publication code live under `packages/paperclip-runner` and the
+`runner-protocol-live-evals.yml` workflow. The public-facing report uses the
+same Evalbook renderer and Runner Lab viewer as the trusted report after
+sanitization.
+
+Product E2E Evals are owned by the runner E2E maintainers. The catalog and
+harness are under `tests/runner-e2e`; the package scripts are `test:e2e:runner`,
+`test:e2e:runner:unit`, `test:e2e:runner:typecheck`, and
+`test:e2e:runner:report`. `README.md`, `FIXTURES.md`, `SECURITY.md`, and
+`EVERYDAY-WORKFLOWS.md` are the detailed sources of truth. The harness starts
+the server and embedded database, creates the company/agent/task through the
+real APIs, drives Chromium, and invokes the selected local or Daytona runner.
+
+## Validation ladder
+
+Start with credential-free checks and a catalog listing. For Product E2E:
+
+```sh
+pnpm test:e2e:runner:typecheck
+pnpm test:e2e:runner:unit
+pnpm test:e2e:runner -- --list
+```
+
+For one explicitly selected local cell, configure only the credentials named
+by that cell in `.env.runner-e2e.local`, then run a narrow ID:
+
+```sh
+pnpm test:e2e:runner -- --id core-compatibility.runner-codex.local.message-marker
+```
+
+Use the selectors documented in the [runner E2E README](../tests/runner-e2e/README.md)
+for a suite, profile, case, group, or environment. Daytona needs the immutable
+image digest and `DAYTONA_API_KEY`; follow the README and fixture security guide.
+`--all` excludes manual suites such as `everyday-workflows`. Select that suite
+explicitly; use a narrow selector while developing a fixture.
+
+For Runner Evals, the narrowest useful local validation is the report program's
+help/validation path and the deterministic Runner checks documented in
+[`runner-workflow-evals.md`](../packages/paperclip-runner/docs/runner-workflow-evals.md).
+Hosted direct live runs must use the default-branch workflow, an exact 40
+character `evals_sha`, an explicitly selected roster (or the maintained
+enabled `all` campaign), and the protected paid environment. The complete
+hosted command is intentionally kept in the workflow and
+[direct live protocol guide](../packages/paperclip-runner/docs/runner-protocol-live-evals.md).
+Live provider runs can spend money; use the existing workflow authorization and
+the user's stated scope when selecting them.
+
+## Failure taxonomy
+
+Record the primary failure class and preserve the evidence that supports it.
+
+- **Product failure:** evidence shows Paperclip or Runner behavior violates the
+ authored case or a hard invariant, such as wrong task state, missing approval
+ gate, lost persistence, bad artifact, or incorrect protocol operation.
+- **Model/provider behavior failure:** the provider turn completed with usable
+ evidence but the model gave the wrong answer, ignored an interaction, failed
+ to complete the authored operation, or violated a semantic assertion. It is
+ scored as behavior, not silently retried as infrastructure.
+- **Grading/evidence failure:** the case or matcher cannot establish its claim,
+ a required recording/screenshot/result is malformed, or the report contract
+ is invalid. Fix the harness or grader before interpreting the score.
+- **Infrastructure failure:** the evidence points to provider/profile
+ unavailability, transport admission failure, service startup failure, a
+ missing credential/image, or inability to produce usable evidence. Startup,
+ transport, and timeout symptoms can instead be product defects when evidence
+ implicates Paperclip or Runner; classify from the observed failure and
+ supported cause, rather than the symptom name alone. Preserve the artifact.
+
+Missing usage or price data means unknown, not free. Keep provider-reported
+costs separate from estimates, and include retry costs when available.
+Latency, cleanup, billing coverage, and unpriced usage are dimensions of the
+result and should remain visible alongside the primary class. A timeout after
+successful product state reads can be a product behavior failure; a failed
+server-health read may be infrastructure, but inspect its cause. Use the
+family-specific classifier and read the attempt evidence before changing an
+analytical label.
+
+## Evidence, provenance, and history
+
+An Evalbook report is a presentation of immutable attempt records, not the
+source of truth. Keep the campaign ID, Paperclip commit, `paperclip-evals`
+commit, catalog/roster or definition fingerprint, model/profile, environment,
+grader version, selected cells, retries, and provider/runtime usage with the
+report. Public projections follow each family's reviewed allowlist and may
+include sanitized fixture conversation, named tool outcomes, screenshots, and
+structured evidence intended for public history. Credentials, secrets, private
+data, raw unredacted records, and hidden reasoning stay out of public
+projections.
+
+Distinguish a complete campaign from a partial campaign. A narrow selector,
+manual diagnostic, missing cell, or infrastructure retry can be useful evidence
+without being a qualification run. History should retain both, with explicit
+coverage and completeness, while trend and latest-green views compare only
+compatible complete campaigns. Refreshing an existing report from retained
+evidence has zero provider calls and is a new presentation of the old
+measurement, not a new model run.
+
+Existing public histories are available at
+[Runner protocol history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html)
+and [Runner Product E2E history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/).
+The consolidated eval hub is at
+[pages.paperclip.ing/evals](https://pages.paperclip.ing/evals/).
+
+For a repeatable workflow, use the matching skill: [paperclip-evals](../.agents/skills/paperclip-evals/SKILL.md),
+[add-runner-eval](../.agents/skills/add-runner-eval/SKILL.md), or
+[add-product-e2e-eval](../.agents/skills/add-product-e2e-eval/SKILL.md).
+
+## Install the authoring skills
+
+The reviewable sources live in this repository's `.agents/skills`. For a
+multi-repository workspace, install the three skills at
+`~/paperclipai/.agents/skills` (not `~/paperclipai/skills`). From the Paperclip
+checkout, run:
+
+```sh
+for skill in paperclip-evals add-runner-eval add-product-e2e-eval; do
+ install -d "$HOME/paperclipai/.agents/skills/$skill"
+ install -m 644 ".agents/skills/$skill/SKILL.md" \
+ "$HOME/paperclipai/.agents/skills/$skill/SKILL.md"
+done
+```
+
+This replaces only the three named skill entrypoints. Run it again after
+updating their tracked sources. Each skill locates the repository independently
+of its installation directory.
+
+## Maintain the public hub
+
+The hub is a static directory with two links to the existing history systems.
+It displays a dated snapshot, not a live scoreboard. It does not run models,
+create another result archive, or change the existing campaign URLs.
+
+Build from the public history feeds and check its summary logic:
+
+```sh
+python3 -m unittest discover -s scripts/evals-hub -p 'test_*.py'
+python3 scripts/evals-hub/build.py --output .paperclip/evals-hub
+```
+
+The hub checks need Python 3 and do not call model providers.
+
+For offline checks, pass `--history-dir <directory>` containing
+`runner-protocol-evals-history.json` and `runner-e2e-history.json`.
+For a pre-merge preview, pass `--docs-ref <branch-or-sha>` to link the guide
+at that revision. The default guide link uses `master`.
+
+Publish with the [Paperclip page helper](../.agents/skills/paperclip-page/SKILL.md)
+and the configured page-uploader credentials. Use Bash 4 or newer; macOS's
+system Bash 3 cannot run this helper. On macOS with Homebrew Bash installed,
+put `$(brew --prefix bash)/bin` first in `PATH` before these commands:
+
+```sh
+export PAPERCLIP_PAGE_BUCKET=pages.paperclip.ing
+export PAPERCLIP_PAGE_BASE_URL=https://pages.paperclip.ing
+export AWS_REGION=us-east-1
+bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals --dry-run
+bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals
+```
+
+For later refreshes, rebuild in the same output directory and publish with
+`--update`. Keep its ignored `.paperclip-page/state.json` ownership record;
+without that record, the helper will refuse to overwrite an existing prefix.
+Verify the public page and its links after publication. This manual refresh
+does not add a scheduled workflow. Preserve the measurement date when choosing
+a newer rendering of the same campaign.
diff --git a/packages/paperclip-runner/docs/runner-protocol-live-evals.md b/packages/paperclip-runner/docs/runner-protocol-live-evals.md
index fe9ee06f6e..98270c3201 100644
--- a/packages/paperclip-runner/docs/runner-protocol-live-evals.md
+++ b/packages/paperclip-runner/docs/runner-protocol-live-evals.md
@@ -1,5 +1,10 @@
# Direct live Runner protocol evals
+For choosing between Runner Evals and full-stack Product E2E Evals, and for
+shared guidance on provenance, history, cost, and failure classification, see
+the [Paperclip evaluation guide](../../../doc/evals.md). This document remains
+the authoritative Runner Evals live protocol runbook.
+
## One Evalbook presentation
Every new report uses the canonical Evalbook grid and the existing Runner Lab
diff --git a/scripts/evals-hub/.gitignore b/scripts/evals-hub/.gitignore
new file mode 100644
index 0000000000..c18dd8d83c
--- /dev/null
+++ b/scripts/evals-hub/.gitignore
@@ -0,0 +1 @@
+__pycache__/
diff --git a/scripts/evals-hub/build.py b/scripts/evals-hub/build.py
new file mode 100644
index 0000000000..982e895ce6
--- /dev/null
+++ b/scripts/evals-hub/build.py
@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+"""Build the public eval directory from the two existing public history feeds."""
+import argparse
+from datetime import datetime, timezone
+import html
+import json
+from pathlib import Path
+from string import Template
+from urllib.parse import quote, urlsplit
+from urllib.request import urlopen
+
+ORIGIN = "https://d1p6rlowie26tp.cloudfront.net"
+SYSTEMS = {
+ "runner": ("runner-protocol-evals", "paperclip.runner-protocol-eval.history/v1"),
+ "product": ("runner-e2e", "paperclip.runner-e2e.history/v1"),
+}
+
+
+def latest(history, schema):
+ if history.get("schema") != schema:
+ raise ValueError("Unsupported history schema")
+ campaign_id = history.get("latestCampaignId")
+ if not campaign_id:
+ raise ValueError("History has no latest campaign")
+ candidates = [c for c in history["campaigns"] if
+ c["campaignId"] == campaign_id or
+ c.get("reportRevision", {}).get("sourceCampaignId") == campaign_id]
+ if not candidates:
+ raise ValueError("Latest campaign is absent from history")
+ # Report refreshes replace presentation, never the measurement date.
+ return max(candidates, key=lambda c: c.get("reportRevision", {}).get("renderedAt", c["generatedAt"]))
+
+
+def public_report(url, prefix):
+ parsed = urlsplit(url)
+ if (parsed.scheme != "https" or parsed.netloc != urlsplit(ORIGIN).netloc
+ or not parsed.path.startswith(f"/{prefix}/campaigns/")
+ or parsed.query or parsed.fragment):
+ raise ValueError("Unexpected public report URL")
+ return html.escape(url, quote=True)
+
+
+def display_date(value):
+ date = datetime.fromisoformat(value.replace("Z", "+00:00"))
+ if date.tzinfo is None:
+ raise ValueError("History date must include a timezone")
+ return date.astimezone(timezone.utc).strftime("%d %b %Y · %H:%M UTC")
+
+
+def summarize(history, kind):
+ prefix, schema = SYSTEMS[kind]
+ campaign = latest(history, schema)
+ totals = campaign["totals"] if kind == "runner" else campaign
+ passed, selected = totals["passed"], totals["selected"]
+ if any(type(n) is not int for n in (passed, selected)) or not 0 <= passed <= selected:
+ raise ValueError("Invalid campaign counts")
+ if type(campaign.get("complete")) is not bool:
+ raise ValueError("Missing campaign coverage flag")
+ revision = campaign.get("reportRevision", {})
+ if revision and not isinstance(revision.get("sourceGeneratedAt"), str):
+ raise ValueError("Report refresh has no reliable measurement date")
+ measured = revision.get("sourceGeneratedAt", campaign["generatedAt"])
+ return {
+ "url": public_report(campaign["publicUrl"], prefix),
+ "date": html.escape(display_date(measured)),
+ "counts": f"{passed} / {selected} selected cases passed",
+ "coverage": "Complete campaign" if campaign["complete"] else "Partial campaign",
+ "id": html.escape(revision.get("sourceCampaignId", campaign["campaignId"])),
+ }
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("--history-dir", type=Path, help="Use saved public history JSON files for offline verification")
+ parser.add_argument("--docs-ref", default="master", help="Git ref containing doc/evals.md")
+ args = parser.parse_args()
+ values = {}
+ for kind, (prefix, _) in SYSTEMS.items():
+ if args.history_dir:
+ history = json.loads((args.history_dir / f"{prefix}-history.json").read_text())
+ else:
+ with urlopen(f"{ORIGIN}/{prefix}/history.json", timeout=30) as response:
+ history = json.load(response)
+ values.update({f"{kind}_{k}": v for k, v in summarize(history, kind).items()})
+ values["refreshed"] = display_date(datetime.now(timezone.utc).isoformat())
+ values["guide_url"] = "https://github.com/paperclipai/paperclip/blob/" + quote(args.docs_ref, safe="/") + "/doc/evals.md"
+ template = Template(Path(__file__).with_name("template.html").read_text())
+ rendered = template.substitute(values)
+ args.output.mkdir(parents=True, exist_ok=True)
+ (args.output / "index.html").write_text(rendered)
+ print(args.output / "index.html")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/evals-hub/template.html b/scripts/evals-hub/template.html
new file mode 100644
index 0000000000..45c94132f4
--- /dev/null
+++ b/scripts/evals-hub/template.html
@@ -0,0 +1,75 @@
+<!doctype html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <meta name="description" content="Paperclip evaluation results, run histories, and guides for Runner Evals and Product E2E Evals.">
+ <title>Paperclip Evals</title>
+ <style>
+ :root { color-scheme: light dark; --bg:#f8f8f5; --card:#fff; --text:#20221e; --muted:#60645c; --line:#d9ddd3; --accent:#306340; --tint:#edf3ec; }
+ * { box-sizing:border-box; }
+ body { margin:0; background:var(--bg); color:var(--text); font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif; }
+ a { color:inherit; text-underline-offset:4px; }
+ a:focus-visible { outline:3px solid var(--accent); outline-offset:5px; }
+ .bar { border-bottom:1px solid var(--line); }
+ .bar div { max-width:1160px; margin:auto; padding:20px 28px; display:flex; justify-content:space-between; align-items:center; gap:20px; }
+ .brand { font-weight:750; letter-spacing:-.04em; font-size:21px; text-decoration:none; }
+ .bar span, .eyebrow { color:var(--muted); font-size:12px; text-transform:uppercase; letter-spacing:.13em; }
+ main { max-width:1160px; margin:64px auto; padding:0 28px; }
+ h1 { font-size:clamp(38px,6vw,62px); line-height:1.05; letter-spacing:-.055em; margin:14px 0 20px; font-weight:650; }
+ .intro { max-width:690px; font-size:19px; color:var(--muted); margin-bottom:36px; }
+ .grid { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:24px; }
+ article { background:var(--card); border:1px solid var(--line); border-radius:12px; padding:30px; display:flex; flex-direction:column; }
+ .number { font:13px/1.4 ui-monospace,monospace; color:var(--accent); }
+ h2 { margin:12px 0 12px; letter-spacing:-.035em; font-size:29px; line-height:1.2; }
+ article p { color:var(--muted); margin:0 0 22px; }
+ .boundary { font-size:14px; padding:14px 16px; background:var(--tint); border-radius:6px; margin-bottom:24px; }
+ .latest { border-top:1px solid var(--line); padding-top:20px; margin-top:auto; }
+ .latest .label { color:var(--muted); font-size:12px; text-transform:uppercase; letter-spacing:.08em; }
+ .counts { font-size:19px; font-weight:600; margin:7px 0; }
+ .meta { font-size:13px; color:var(--muted); }
+ .id { overflow-wrap:anywhere; margin-top:4px; font:11px/1.6 ui-monospace,monospace; color:var(--muted); }
+ .links { display:flex; flex-wrap:wrap; align-items:center; gap:18px; margin:24px 0 18px; }
+ .button { display:inline-block; background:var(--text); color:var(--card); padding:10px 15px; border-radius:6px; text-decoration:none; font-weight:600; font-size:14px; }
+ .guide { font-size:13px; }
+ .note { font-size:13px; color:var(--muted); margin:19px 0 38px; }
+ .start { display:flex; gap:28px; justify-content:space-between; align-items:center; padding:25px 0; border-top:1px solid var(--line); border-bottom:1px solid var(--line); }
+ .start h2 { font-size:22px; margin:0 0 6px; }
+ .start p { margin:0; color:var(--muted); font-size:14px; max-width:650px; }
+ footer { margin-top:25px; color:var(--muted); font-size:12px; }
+ @media (prefers-color-scheme:dark) { :root { --bg:#151714; --card:#1d201b; --text:#f2f4ed; --muted:#aeb6a6; --line:#363d31; --accent:#a0cda5; --tint:#273024; } }
+ @media (max-width:700px) { main { margin:36px auto; padding:0 20px; } .grid { grid-template-columns:1fr; } article { padding:24px; } .start { align-items:flex-start; flex-direction:column; gap:18px; } .bar div { padding:16px 20px; } .bar span { font-size:10px; } }
+ </style>
+</head>
+<body>
+ <header class="bar"><div><a class="brand" href="https://paperclip.ing">Paperclip</a><span>Evaluation directory</span></div></header>
+ <main>
+ <div class="eyebrow">Evidence for better agents</div>
+ <h1>Paperclip Evals</h1>
+ <p class="intro">Two ways to test Paperclip. Follow each program from a test case to its evidence, results, and run history.</p>
+ <div class="grid">
+ <article>
+ <div class="number">01 / RUNNER</div>
+ <h2>Runner Evals</h2>
+ <p>Does the runner use tools correctly, respect permissions, and preserve its execution contract?</p>
+ <div class="boundary">Real runner + model provider<br>Controlled mock Paperclip control plane</div>
+ <div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$runner_counts</div><div class="meta">$runner_date · $runner_coverage</div><div class="id">$runner_id</div></div>
+ <div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html">Browse run history →</a><a href="$runner_url">Open report</a></div>
+ <a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/packages/paperclip-runner/docs/runner-protocol-live-evals.md">Runner operations guide</a>
+ </article>
+ <article>
+ <div class="number">02 / PRODUCT</div>
+ <h2>Product E2E Evals</h2>
+ <p>Can a user complete work through Paperclip, from the first request to a usable result?</p>
+ <div class="boundary">Real browser + server + database + runner + model provider<br>Local and Daytona environments, as selected</div>
+ <div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$product_counts</div><div class="meta">$product_date · $product_coverage</div><div class="id">$product_id</div></div>
+ <div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/">Browse run history →</a><a href="$product_url">Open report</a></div>
+ <a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/tests/runner-e2e/README.md">Product E2E operations guide</a>
+ </article>
+ </div>
+ <p class="note">Snapshot refreshed $refreshed. History pages may contain newer runs. A partial campaign covers only its selected cases. These programs measure different boundaries; their pass rates are not combined.</p>
+ <section class="start"><div><h2>Choose a test. Make the result inspectable.</h2><p>The eval guide explains which program to use, how to add a case, and how to distinguish a product defect from model behavior, a grading error, or a setup failure. Evalbook is a report framework used to present evidence; it is not an execution boundary.</p></div><a class="button" href="$guide_url">Read the eval guide →</a></section>
+ <footer>Existing run archives remain the source of truth. Report refreshes do not count as new model runs.</footer>
+ </main>
+</body>
+</html>
diff --git a/scripts/evals-hub/test_build.py b/scripts/evals-hub/test_build.py
new file mode 100644
index 0000000000..2960efd827
--- /dev/null
+++ b/scripts/evals-hub/test_build.py
@@ -0,0 +1,82 @@
+import copy
+import unittest
+
+from build import SYSTEMS, summarize
+
+
+class HistorySummaryTests(unittest.TestCase):
+ def history(self, kind="runner"):
+ prefix, schema = SYSTEMS[kind]
+ campaign = {
+ "campaignId": "run-1", "generatedAt": "2026-09-01T12:00:00Z",
+ "publicUrl": f"https://d1p6rlowie26tp.cloudfront.net/{prefix}/campaigns/run-1/",
+ "complete": False,
+ }
+ counts = {"selected": 3, "passed": 2}
+ campaign.update({"totals": counts} if kind == "runner" else counts)
+ return {"schema": schema, "latestCampaignId": "run-1", "campaigns": [campaign]}
+
+ def test_both_families_keep_partial_coverage(self):
+ for kind in SYSTEMS:
+ with self.subTest(kind=kind):
+ result = summarize(self.history(kind), kind)
+ self.assertEqual(result["coverage"], "Partial campaign")
+ self.assertEqual(result["counts"], "2 / 3 selected cases passed")
+
+ def test_refresh_changes_report_but_not_measurement_date(self):
+ history = self.history()
+ refreshed = copy.deepcopy(history["campaigns"][0])
+ refreshed.update(campaignId="run-1-refresh", generatedAt="2026-09-03T12:00:00Z")
+ refreshed["publicUrl"] += "refresh/"
+ refreshed["reportRevision"] = {
+ "sourceCampaignId": "run-1", "sourceGeneratedAt": "2026-09-01T12:00:00Z",
+ "renderedAt": "2026-09-03T12:00:00Z",
+ }
+ history["campaigns"].append(refreshed)
+ result = summarize(history, "runner")
+ self.assertEqual(result["id"], "run-1")
+ self.assertIn("01 Sep 2026", result["date"])
+ self.assertTrue(result["url"].endswith("/refresh/"))
+
+ def test_missing_latest_does_not_fall_back_to_an_arbitrary_run(self):
+ history = self.history()
+ history["latestCampaignId"] = "absent"
+ with self.assertRaises(ValueError):
+ summarize(history, "runner")
+
+ def test_refresh_without_measurement_date_is_rejected(self):
+ history = self.history()
+ campaign = history["campaigns"][0]
+ campaign["campaignId"] = "run-1-refresh"
+ campaign["reportRevision"] = {
+ "sourceCampaignId": "run-1", "renderedAt": "2026-09-03T12:00:00Z",
+ }
+ with self.assertRaisesRegex(ValueError, "measurement date"):
+ summarize(history, "runner")
+
+ def test_unknown_schema_and_missing_coverage_are_rejected(self):
+ for field in ("schema", "complete"):
+ history = self.history()
+ if field == "schema":
+ history["schema"] = "future-schema"
+ else:
+ del history["campaigns"][0]["complete"]
+ with self.subTest(field=field), self.assertRaises(ValueError):
+ summarize(history, "runner")
+
+ def test_bad_counts_cannot_be_published(self):
+ history = self.history()
+ history["campaigns"][0]["totals"]["passed"] = 4
+ with self.assertRaises(ValueError):
+ summarize(history, "runner")
+
+ def test_external_report_urls_are_rejected(self):
+ history = self.history()
+ for url in ("javascript:alert(1)", "https://example.com/", "https://d1p6rlowie26tp.cloudfront.net/other/"):
+ history["campaigns"][0]["publicUrl"] = url
+ with self.subTest(url=url), self.assertRaises(ValueError):
+ summarize(history, "runner")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md
index 912497d27d..8f20166302 100644
--- a/tests/runner-e2e/README.md
+++ b/tests/runner-e2e/README.md
@@ -1,5 +1,11 @@
# Paid runner full-stack E2E
+For family selection, ownership, provenance, history, and failure taxonomy,
+see the [Paperclip evaluation guide](../../doc/evals.md). This README is the
+authoritative runbook for Product E2E runner cells; the separate Runner Evals
+protocol guide lives at
+`packages/paperclip-runner/docs/runner-protocol-live-evals.md`.
+
This is the billable browser acceptance campaign system for Paperclip runner
profiles. It is deliberately separate from `tests/e2e`: every independently
scheduled execution gets