The open-source app everyone uses to manage agents at work
TypeScript acpxagent-orchestrationagent-skillsagentic-aiai-agentsai-engineeringai-governanceai-toolsautonomous-agentsclaudeclaude-codeclaude-skillscodexhermesllmmcpmulti-agentopenclawproject-managementworkflow-automation
docs: add eval guide, authoring skills, and public history hub (#13535)
Dotta18989a9e7376
diff --git a/.agents/skills/add-product-e2e-eval/SKILL.md b/.agents/skills/add-product-e2e-eval/SKILL.mdnew file mode 100644index 0000000000..686fa44f13--- /dev/null+++ b/.agents/skills/add-product-e2e-eval/SKILL.md@@ -0,0 +1,84 @@+---+name: add-product-e2e-eval+description: Add or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.+---++# Add a Product E2E Eval++Use this skill for **Product E2E Evals**: real Chromium, Paperclip server,+database, runner, provider, and optionally Daytona. Everyday Workflows are in+this family even when their packaged results are imported into Evalbook. Runner+protocol cases against the mock control plane belong in+[add-runner-eval](../add-runner-eval/SKILL.md).++Locate the repository using `PAPERCLIP_ROOT` when supplied, or+`git rev-parse --show-toplevel` from a checkout. From outside Git, inspect+workspace roots such as `~/paperclipai/paperclip`; verify the selected root+contains `tests/runner-e2e` and `packages/paperclip-runner`. Run commands from+that repository root. The copied skill may live outside the checkout.+Read `doc/evals.md`, then the authoritative+`tests/runner-e2e/README.md`, `FIXTURES.md`, `SECURITY.md`, and+`EVERYDAY-WORKFLOWS.md` for the selected area. Inspect the nearest existing+catalog entry, case, harness flow, matcher, evidence writer, and report test+before changing anything. Keep the user journey on production browser/API+surfaces; do not add private runner hooks or direct fixture database writes.++Define one bounded workflow with a clear user outcome, durable state+assertions, and independent evidence. Declare profile, environment, expected+provider turns, timeout, cleanup, screenshots, artifact checks, and billing+scope. Keep credentials and secrets out of catalog data, screenshots, fixture+metadata, logs, and tracked files. Daytona images must be immutable digest+references. Use the existing result validator, failure classifier, screenshot+policy, and Product E2E report pipeline rather than duplicating them.++Register the workflow in `tests/runner-e2e/catalog.ts` and the relevant+`fixture-registry.ts` paths. Everyday cases and actions live in+`everyday-cases.ts` and `everyday-flow.ts`; match the existing suite's structure.+Wire its+profile/environment/case IDs through existing selector and matcher tables, and+add report/catalog coverage tests where the surrounding suite does so. Calibrate+new grading assertions with a valid outcome and a plausible wrong outcome;+missing evidence must not produce a pass. Confirm+discovery before running it:++```sh+pnpm test:e2e:runner -- --list --suite <suite-name>+pnpm test:e2e:runner -- --list --suite everyday-workflows+```++`--all` intentionally excludes the manual `everyday-workflows` suite and other+explicit-only cells. Use the exact suite or execution ID for those. The Product+E2E generator is `pnpm test:e2e:runner:report`; an Everyday Workflows result may+also be imported into Evalbook with its canonical importer, but it remains a+Product E2E run and should use its packaged dashboard/report first.++Run credential-free checks first:++```sh+pnpm test:e2e:runner:typecheck+pnpm test:e2e:runner:unit+pnpm test:e2e:runner -- --list+```++For an authorized live check, use the smallest explicit local `--id` selector.+Select Daytona only with the configured immutable image and credentials. A+full `--all` campaign is paid and is for the governed workflow. Verify results+from the packaged attempt evidence and dashboard; passing model-authored tests+cannot override the independent oracle.++Classify a completed wrong workflow as product or model/provider behavior as+the assertions warrant. Startup, transport, and timeout symptoms require+evidence-based attribution: they may indicate a Paperclip/Runner product bug,+provider behavior, or infrastructure. Preserve the observed failure and cause+separately, keep the existing machine grade/classifier unchanged, and retain+partial attempts, retries, source SHA, catalog/definition digest, model/profile,+environment, grader version, timing, tokens, runtime estimates, and cost+coverage. Do not claim qualification from a partial/manual selection.++Update `README.md`, `FIXTURES.md`, `SECURITY.md`, or+`EVERYDAY-WORKFLOWS.md` when their authoritative contract changes, and link+from `doc/evals.md`. Do not recreate Evalbook HTML or publish raw trusted+traces, videos, archives, databases, workspaces, credentials, SVG, provider+session IDs, or hidden reasoning. Public sanitized fixture conversation, marked+screenshots, and allowlisted structured evidence are expected when the existing+publisher permits them.diff --git a/.agents/skills/add-runner-eval/SKILL.md b/.agents/skills/add-runner-eval/SKILL.mdnew file mode 100644index 0000000000..111a1b2c7a--- /dev/null+++ b/.agents/skills/add-runner-eval/SKILL.md@@ -0,0 +1,72 @@+---+name: add-runner-eval+description: Add or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.+---++# Add a Runner Eval++Use this skill for the **Runner Evals** family: a real Runner/provider session+against a seeded mock control plane. Product browser/server/database/Daytona+coverage belongs in [add-product-e2e-eval](../add-product-e2e-eval/SKILL.md).++Locate the Paperclip checkout using `PAPERCLIP_ROOT` when supplied, or+`git rev-parse --show-toplevel` from a checkout. From outside Git, inspect the+workspace roots (for example `~/paperclipai/paperclip`) and verify that the+selected root contains `packages/paperclip-runner` and `tests/runner-e2e`.+Locate `paperclip-evals` using `PAPERCLIP_EVALS_ROOT` or a discovered sibling;+a worktree's parent directory need not contain that repository. Read+`doc/evals.md` and `packages/paperclip-runner/docs/runner-protocol-live-evals.md`,+then inspect the nearest existing case, roster, schema, and report test before+editing. Definitions and authored cases belong in the sibling+`paperclip-evals/evals/paperclip-runner`; Runner integration, aggregation,+viewer, and publication behavior belongs in `packages/paperclip-runner`.+Keep the control-plane boundary explicit in names and documentation.++The sibling eval README is the concrete map: cases live under `cases/`,+company fixtures under `fixtures/`, runtime/model settings under `configs/`,+selections under `rosters/`, and maintained campaign membership under+`campaigns/live-direct-full.json`. Update inventory/coverage mappings when the+program requires them; a new file alone does not join the maintained campaign.+From the Evals repository root, adapt these provider-free checks to the case+and roster you changed. Run the reliability-plan validator only when that+separate plan changes:++```sh+python3 evals/paperclip-runner/tools/eval_program.py validate \+ --case evals/paperclip-runner/cases/get-task-context.json \+ --config evals/paperclip-runner/configs/live-codex-pinned.json+python3 evals/paperclip-runner/tools/run_live_roster.py validate \+ --roster evals/paperclip-runner/rosters/live-mini.json --run-id validate-new-case+python3 evals/paperclip-runner/tools/run_live_campaign.py validate \+ --campaign evals/paperclip-runner/campaigns/live-direct-full.json+python3 evals/paperclip-runner/tools/reliability_campaign.py validate \+ --plan evals/paperclip-runner/campaigns/paperclip-runner-reliability.json+```++Use nearby positive and negative grader cases/fixtures to calibrate the new+assertion, including malformed or missing evidence where the grader must fail+closed. Preserve the existing machine disposition and grade; product,+model/provider, grading, and infrastructure labels are analytical annotations,+not instructions to rewrite classifiers.++Author one bounded case with a deterministic semantic assertion and an+inspectable result. Declare its expected operation, state effect, provider+lane/profile, timeout and retry policy, and any required evidence. Do not grade+hidden reasoning, infer success from a provider terminal message, or invent+conversation/tool evidence. Public output follows the reviewed projection:+sanitized fixture conversation and allowlisted tool outcomes may be published;+raw trusted artifacts, credentials, secrets, private references, and hidden+reasoning may not.++Validate without provider calls first using the commands above and the relevant+report/render validation documented in the Runner docs. When a live run is+authorized, pin the Paperclip commit and exact 40-character+`paperclip-evals` commit, select the smallest useful roster, and retain the+complete provenance and cost record.++Update authoritative detailed docs when the contract or command changes, then+link from `doc/evals.md` rather than duplicating the Runner runbook. Keep public+reports immutable and use the reviewed projection; sanitized fixture+conversation and allowlisted tool outcomes may be public, while credentials,+secrets, private references, raw trusted payloads, and hidden reasoning must not+be exposed.diff --git a/.agents/skills/paperclip-evals/SKILL.md b/.agents/skills/paperclip-evals/SKILL.mdnew file mode 100644index 0000000000..9735805c80--- /dev/null+++ b/.agents/skills/paperclip-evals/SKILL.md@@ -0,0 +1,63 @@+---+name: paperclip-evals+description: Choose, inspect, validate, and report Paperclip Runner or Product E2E evaluations while preserving evidence, provenance, cost, and failure classification.+---++# Paperclip evals++Use this skill when a request concerns Paperclip evaluation selection,+interpretation, evidence, history, or a live run. Read `doc/evals.md` in the+Paperclip repository first. It defines the two families and their boundaries.++## Discover the repository++Do not assume the skill's installed location is inside a checkout. Locate the+repo explicitly with `git rev-parse --show-toplevel` from the current directory,+or inspect likely workspace roots and select the checkout containing+`package.json`, `tests/runner-e2e`, and `packages/paperclip-runner`. Locate the+private sibling `paperclip-evals` only when a Runner Eval needs its definitions;+use an explicit `PAPERCLIP_EVALS_ROOT` or a discovered sibling checkout. Never+invent a relative path from this copied skill into the repository.++## Route the request++Choose **Runner Evals** for real runner/provider protocol behavior against the+mock control plane. Authoritative details are in+`packages/paperclip-runner/docs/runner-protocol-live-evals.md` and the sibling+`paperclip-evals/evals/paperclip-runner` definitions.++Choose **Product E2E Evals** for real browser/server/database/runner/provider+workflows, including local and Daytona environments. Read+`tests/runner-e2e/README.md`, then `FIXTURES.md`, `SECURITY.md`, or+`EVERYDAY-WORKFLOWS.md` as relevant. Everyday Workflows remain Product E2E+even when imported into Evalbook. “Headless” is a browser mode, not a family.++## Work safely++Start with read-only catalog inspection and credential-free validation. For+Product E2E use `pnpm test:e2e:runner:typecheck`,+`pnpm test:e2e:runner:unit`, and `pnpm test:e2e:runner -- --list`; run one+explicit cell only when the user has authorized a live/paid run and the needed+credentials and immutable Daytona image are configured. For Runner Evals use+the pinned eval revision and the documented workflow/CLI. Never use a partial+selector as evidence of full coverage.++Keep source revisions, definition/catalog fingerprints, model/profile,+environment, selected cells, retries, timing, usage/cost coverage, and grader+version attached to every interpretation. Preserve partial attempts and+classify failures as product, model/provider behavior, grading/evidence, or+infrastructure from the observed failure and supported cause. A usable+completed behavior failure is not infrastructure; missing provider/profile,+transport, startup, or evidence requires examining the evidence before choosing+the cause.++Use the existing family generator and viewer. Public projections may contain+sanitized fixture conversation and allowlisted tool outcomes/evidence; follow+the family's projection and publisher checks. Do not expose raw trusted+artifacts, credentials, secrets, private data, provider session IDs, or hidden+reasoning. A refresh from retained evidence has zero provider calls and remains+the original measurement with a new presentation. Link the public histories+and hub from `doc/evals.md` when reporting results.++For adding a case or fixture, use the narrower [add-runner-eval](../add-runner-eval/SKILL.md)+or [add-product-e2e-eval](../add-product-e2e-eval/SKILL.md) skill.diff --git a/doc/evals.md b/doc/evals.mdnew file mode 100644index 0000000000..88fedec15e--- /dev/null+++ b/doc/evals.md@@ -0,0 +1,214 @@+# Paperclip evaluation guide++Paperclip has two live eval families with different questions, owners, and+evidence. Choose the family before selecting a model, profile, or case.++- **Runner Evals:** real Runner/provider behavior against a seeded mock control+ plane. Definitions live in `paperclip-evals/evals/paperclip-runner`; see the+ [direct live protocol evals](../packages/paperclip-runner/docs/runner-protocol-live-evals.md).+- **Product E2E Evals:** real browser, Paperclip server, database, Runner,+ provider, and (where selected) Daytona, using an isolated instance and+ grading oracle. See [`tests/runner-e2e`](../tests/runner-e2e/README.md) and+ [Everyday Workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md).++Runner Evals answer whether a real runner/provider can perform a bounded+protocol operation against the expected control-plane contract. Product E2E+Evals answer whether a person can complete a product workflow through the real+Paperclip surfaces and whether the resulting artifact and state are usable.+The names describe the system under test; “headless” is an execution option,+not an eval category.++## Selecting a family++Use **Runner Evals** for a runner protocol, adapter, transport, native session,+tool grant, or one-turn provider qualification question. The workflow checks+out an exact `paperclip-evals` revision, builds the Runner and viewer, runs a+live roster, and renders the canonical Evalbook report. The control plane is a+seeded test authority, so a passing result does not prove browser UX, production+server behavior, database persistence, Daytona behavior, or a real third-party+mutation.++Use **Product E2E Evals** for browser interaction, issue/task lifecycle,+approval and clarification UI, project/repository selection, persistence over a+controller restart, artifact delivery, billing/evidence behavior, or runner+continuity in local or Daytona environments. The harness creates a fresh+Paperclip instance per cell and uses public APIs and the production browser+surface. The suite's [Everyday Workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md)+are Product E2E even when their results are imported into Evalbook.++Do not combine a partial Runner campaign and a partial Product E2E campaign into+one score. A campaign is comparable when its definition/grader, model/profile,+environment, and contract match. The evaluated Paperclip revision may+intentionally differ for a before/after fix comparison; record it as a+comparison axis.++## Ownership and codepaths++Runner Evals are owned by the Runner/evals maintainers. Definitions, rosters,+case prompts, and the report program live in the sibling private repository+`paperclipai/paperclip-evals`; Runner integration, viewer, aggregation, and+publication code live under `packages/paperclip-runner` and the+`runner-protocol-live-evals.yml` workflow. The public-facing report uses the+same Evalbook renderer and Runner Lab viewer as the trusted report after+sanitization.++Product E2E Evals are owned by the runner E2E maintainers. The catalog and+harness are under `tests/runner-e2e`; the package scripts are `test:e2e:runner`,+`test:e2e:runner:unit`, `test:e2e:runner:typecheck`, and+`test:e2e:runner:report`. `README.md`, `FIXTURES.md`, `SECURITY.md`, and+`EVERYDAY-WORKFLOWS.md` are the detailed sources of truth. The harness starts+the server and embedded database, creates the company/agent/task through the+real APIs, drives Chromium, and invokes the selected local or Daytona runner.++## Validation ladder++Start with credential-free checks and a catalog listing. For Product E2E:++```sh+pnpm test:e2e:runner:typecheck+pnpm test:e2e:runner:unit+pnpm test:e2e:runner -- --list+```++For one explicitly selected local cell, configure only the credentials named+by that cell in `.env.runner-e2e.local`, then run a narrow ID:++```sh+pnpm test:e2e:runner -- --id core-compatibility.runner-codex.local.message-marker+```++Use the selectors documented in the [runner E2E README](../tests/runner-e2e/README.md)+for a suite, profile, case, group, or environment. Daytona needs the immutable+image digest and `DAYTONA_API_KEY`; follow the README and fixture security guide.+`--all` excludes manual suites such as `everyday-workflows`. Select that suite+explicitly; use a narrow selector while developing a fixture.++For Runner Evals, the narrowest useful local validation is the report program's+help/validation path and the deterministic Runner checks documented in+[`runner-workflow-evals.md`](../packages/paperclip-runner/docs/runner-workflow-evals.md).+Hosted direct live runs must use the default-branch workflow, an exact 40+character `evals_sha`, an explicitly selected roster (or the maintained+enabled `all` campaign), and the protected paid environment. The complete+hosted command is intentionally kept in the workflow and+[direct live protocol guide](../packages/paperclip-runner/docs/runner-protocol-live-evals.md).+Live provider runs can spend money; use the existing workflow authorization and+the user's stated scope when selecting them.++## Failure taxonomy++Record the primary failure class and preserve the evidence that supports it.++- **Product failure:** evidence shows Paperclip or Runner behavior violates the+ authored case or a hard invariant, such as wrong task state, missing approval+ gate, lost persistence, bad artifact, or incorrect protocol operation.+- **Model/provider behavior failure:** the provider turn completed with usable+ evidence but the model gave the wrong answer, ignored an interaction, failed+ to complete the authored operation, or violated a semantic assertion. It is+ scored as behavior, not silently retried as infrastructure.+- **Grading/evidence failure:** the case or matcher cannot establish its claim,+ a required recording/screenshot/result is malformed, or the report contract+ is invalid. Fix the harness or grader before interpreting the score.+- **Infrastructure failure:** the evidence points to provider/profile+ unavailability, transport admission failure, service startup failure, a+ missing credential/image, or inability to produce usable evidence. Startup,+ transport, and timeout symptoms can instead be product defects when evidence+ implicates Paperclip or Runner; classify from the observed failure and+ supported cause, rather than the symptom name alone. Preserve the artifact.++Missing usage or price data means unknown, not free. Keep provider-reported+costs separate from estimates, and include retry costs when available.+Latency, cleanup, billing coverage, and unpriced usage are dimensions of the+result and should remain visible alongside the primary class. A timeout after+successful product state reads can be a product behavior failure; a failed+server-health read may be infrastructure, but inspect its cause. Use the+family-specific classifier and read the attempt evidence before changing an+analytical label.++## Evidence, provenance, and history++An Evalbook report is a presentation of immutable attempt records, not the+source of truth. Keep the campaign ID, Paperclip commit, `paperclip-evals`+commit, catalog/roster or definition fingerprint, model/profile, environment,+grader version, selected cells, retries, and provider/runtime usage with the+report. Public projections follow each family's reviewed allowlist and may+include sanitized fixture conversation, named tool outcomes, screenshots, and+structured evidence intended for public history. Credentials, secrets, private+data, raw unredacted records, and hidden reasoning stay out of public+projections.++Distinguish a complete campaign from a partial campaign. A narrow selector,+manual diagnostic, missing cell, or infrastructure retry can be useful evidence+without being a qualification run. History should retain both, with explicit+coverage and completeness, while trend and latest-green views compare only+compatible complete campaigns. Refreshing an existing report from retained+evidence has zero provider calls and is a new presentation of the old+measurement, not a new model run.++Existing public histories are available at+[Runner protocol history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html)+and [Runner Product E2E history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/).+The consolidated eval hub is at+[pages.paperclip.ing/evals](https://pages.paperclip.ing/evals/).++For a repeatable workflow, use the matching skill: [paperclip-evals](../.agents/skills/paperclip-evals/SKILL.md),+[add-runner-eval](../.agents/skills/add-runner-eval/SKILL.md), or+[add-product-e2e-eval](../.agents/skills/add-product-e2e-eval/SKILL.md).++## Install the authoring skills++The reviewable sources live in this repository's `.agents/skills`. For a+multi-repository workspace, install the three skills at+`~/paperclipai/.agents/skills` (not `~/paperclipai/skills`). From the Paperclip+checkout, run:++```sh+for skill in paperclip-evals add-runner-eval add-product-e2e-eval; do+ install -d "$HOME/paperclipai/.agents/skills/$skill"+ install -m 644 ".agents/skills/$skill/SKILL.md" \+ "$HOME/paperclipai/.agents/skills/$skill/SKILL.md"+done+```++This replaces only the three named skill entrypoints. Run it again after+updating their tracked sources. Each skill locates the repository independently+of its installation directory.++## Maintain the public hub++The hub is a static directory with two links to the existing history systems.+It displays a dated snapshot, not a live scoreboard. It does not run models,+create another result archive, or change the existing campaign URLs.++Build from the public history feeds and check its summary logic:++```sh+python3 -m unittest discover -s scripts/evals-hub -p 'test_*.py'+python3 scripts/evals-hub/build.py --output .paperclip/evals-hub+```++The hub checks need Python 3 and do not call model providers.++For offline checks, pass `--history-dir <directory>` containing+`runner-protocol-evals-history.json` and `runner-e2e-history.json`.+For a pre-merge preview, pass `--docs-ref <branch-or-sha>` to link the guide+at that revision. The default guide link uses `master`.++Publish with the [Paperclip page helper](../.agents/skills/paperclip-page/SKILL.md)+and the configured page-uploader credentials. Use Bash 4 or newer; macOS's+system Bash 3 cannot run this helper. On macOS with Homebrew Bash installed,+put `$(brew --prefix bash)/bin` first in `PATH` before these commands:++```sh+export PAPERCLIP_PAGE_BUCKET=pages.paperclip.ing+export PAPERCLIP_PAGE_BASE_URL=https://pages.paperclip.ing+export AWS_REGION=us-east-1+bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals --dry-run+bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals+```++For later refreshes, rebuild in the same output directory and publish with+`--update`. Keep its ignored `.paperclip-page/state.json` ownership record;+without that record, the helper will refuse to overwrite an existing prefix.+Verify the public page and its links after publication. This manual refresh+does not add a scheduled workflow. Preserve the measurement date when choosing+a newer rendering of the same campaign.diff --git a/packages/paperclip-runner/docs/runner-protocol-live-evals.md b/packages/paperclip-runner/docs/runner-protocol-live-evals.mdindex fe9ee06f6e..98270c3201 100644--- a/packages/paperclip-runner/docs/runner-protocol-live-evals.md+++ b/packages/paperclip-runner/docs/runner-protocol-live-evals.md@@ -1,5 +1,10 @@# Direct live Runner protocol evals+For choosing between Runner Evals and full-stack Product E2E Evals, and for+shared guidance on provenance, history, cost, and failure classification, see+the [Paperclip evaluation guide](../../../doc/evals.md). This document remains+the authoritative Runner Evals live protocol runbook.+## One Evalbook presentationEvery new report uses the canonical Evalbook grid and the existing Runner Labdiff --git a/scripts/evals-hub/.gitignore b/scripts/evals-hub/.gitignorenew file mode 100644index 0000000000..c18dd8d83c--- /dev/null+++ b/scripts/evals-hub/.gitignore@@ -0,0 +1 @@+__pycache__/diff --git a/scripts/evals-hub/build.py b/scripts/evals-hub/build.pynew file mode 100644index 0000000000..982e895ce6--- /dev/null+++ b/scripts/evals-hub/build.py@@ -0,0 +1,96 @@+#!/usr/bin/env python3+"""Build the public eval directory from the two existing public history feeds."""+import argparse+from datetime import datetime, timezone+import html+import json+from pathlib import Path+from string import Template+from urllib.parse import quote, urlsplit+from urllib.request import urlopen++ORIGIN = "https://d1p6rlowie26tp.cloudfront.net"+SYSTEMS = {+ "runner": ("runner-protocol-evals", "paperclip.runner-protocol-eval.history/v1"),+ "product": ("runner-e2e", "paperclip.runner-e2e.history/v1"),+}+++def latest(history, schema):+ if history.get("schema") != schema:+ raise ValueError("Unsupported history schema")+ campaign_id = history.get("latestCampaignId")+ if not campaign_id:+ raise ValueError("History has no latest campaign")+ candidates = [c for c in history["campaigns"] if+ c["campaignId"] == campaign_id or+ c.get("reportRevision", {}).get("sourceCampaignId") == campaign_id]+ if not candidates:+ raise ValueError("Latest campaign is absent from history")+ # Report refreshes replace presentation, never the measurement date.+ return max(candidates, key=lambda c: c.get("reportRevision", {}).get("renderedAt", c["generatedAt"]))+++def public_report(url, prefix):+ parsed = urlsplit(url)+ if (parsed.scheme != "https" or parsed.netloc != urlsplit(ORIGIN).netloc+ or not parsed.path.startswith(f"/{prefix}/campaigns/")+ or parsed.query or parsed.fragment):+ raise ValueError("Unexpected public report URL")+ return html.escape(url, quote=True)+++def display_date(value):+ date = datetime.fromisoformat(value.replace("Z", "+00:00"))+ if date.tzinfo is None:+ raise ValueError("History date must include a timezone")+ return date.astimezone(timezone.utc).strftime("%d %b %Y · %H:%M UTC")+++def summarize(history, kind):+ prefix, schema = SYSTEMS[kind]+ campaign = latest(history, schema)+ totals = campaign["totals"] if kind == "runner" else campaign+ passed, selected = totals["passed"], totals["selected"]+ if any(type(n) is not int for n in (passed, selected)) or not 0 <= passed <= selected:+ raise ValueError("Invalid campaign counts")+ if type(campaign.get("complete")) is not bool:+ raise ValueError("Missing campaign coverage flag")+ revision = campaign.get("reportRevision", {})+ if revision and not isinstance(revision.get("sourceGeneratedAt"), str):+ raise ValueError("Report refresh has no reliable measurement date")+ measured = revision.get("sourceGeneratedAt", campaign["generatedAt"])+ return {+ "url": public_report(campaign["publicUrl"], prefix),+ "date": html.escape(display_date(measured)),+ "counts": f"{passed} / {selected} selected cases passed",+ "coverage": "Complete campaign" if campaign["complete"] else "Partial campaign",+ "id": html.escape(revision.get("sourceCampaignId", campaign["campaignId"])),+ }+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--output", type=Path, required=True)+ parser.add_argument("--history-dir", type=Path, help="Use saved public history JSON files for offline verification")+ parser.add_argument("--docs-ref", default="master", help="Git ref containing doc/evals.md")+ args = parser.parse_args()+ values = {}+ for kind, (prefix, _) in SYSTEMS.items():+ if args.history_dir:+ history = json.loads((args.history_dir / f"{prefix}-history.json").read_text())+ else:+ with urlopen(f"{ORIGIN}/{prefix}/history.json", timeout=30) as response:+ history = json.load(response)+ values.update({f"{kind}_{k}": v for k, v in summarize(history, kind).items()})+ values["refreshed"] = display_date(datetime.now(timezone.utc).isoformat())+ values["guide_url"] = "https://github.com/paperclipai/paperclip/blob/" + quote(args.docs_ref, safe="/") + "/doc/evals.md"+ template = Template(Path(__file__).with_name("template.html").read_text())+ rendered = template.substitute(values)+ args.output.mkdir(parents=True, exist_ok=True)+ (args.output / "index.html").write_text(rendered)+ print(args.output / "index.html")+++if __name__ == "__main__":+ main()diff --git a/scripts/evals-hub/template.html b/scripts/evals-hub/template.htmlnew file mode 100644index 0000000000..45c94132f4--- /dev/null+++ b/scripts/evals-hub/template.html@@ -0,0 +1,75 @@+<!doctype html>+<html lang="en">+<head>+ <meta charset="utf-8">+ <meta name="viewport" content="width=device-width, initial-scale=1">+ <meta name="description" content="Paperclip evaluation results, run histories, and guides for Runner Evals and Product E2E Evals.">+ <title>Paperclip Evals</title>+ <style>+ :root { color-scheme: light dark; --bg:#f8f8f5; --card:#fff; --text:#20221e; --muted:#60645c; --line:#d9ddd3; --accent:#306340; --tint:#edf3ec; }+ * { box-sizing:border-box; }+ body { margin:0; background:var(--bg); color:var(--text); font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif; }+ a { color:inherit; text-underline-offset:4px; }+ a:focus-visible { outline:3px solid var(--accent); outline-offset:5px; }+ .bar { border-bottom:1px solid var(--line); }+ .bar div { max-width:1160px; margin:auto; padding:20px 28px; display:flex; justify-content:space-between; align-items:center; gap:20px; }+ .brand { font-weight:750; letter-spacing:-.04em; font-size:21px; text-decoration:none; }+ .bar span, .eyebrow { color:var(--muted); font-size:12px; text-transform:uppercase; letter-spacing:.13em; }+ main { max-width:1160px; margin:64px auto; padding:0 28px; }+ h1 { font-size:clamp(38px,6vw,62px); line-height:1.05; letter-spacing:-.055em; margin:14px 0 20px; font-weight:650; }+ .intro { max-width:690px; font-size:19px; color:var(--muted); margin-bottom:36px; }+ .grid { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:24px; }+ article { background:var(--card); border:1px solid var(--line); border-radius:12px; padding:30px; display:flex; flex-direction:column; }+ .number { font:13px/1.4 ui-monospace,monospace; color:var(--accent); }+ h2 { margin:12px 0 12px; letter-spacing:-.035em; font-size:29px; line-height:1.2; }+ article p { color:var(--muted); margin:0 0 22px; }+ .boundary { font-size:14px; padding:14px 16px; background:var(--tint); border-radius:6px; margin-bottom:24px; }+ .latest { border-top:1px solid var(--line); padding-top:20px; margin-top:auto; }+ .latest .label { color:var(--muted); font-size:12px; text-transform:uppercase; letter-spacing:.08em; }+ .counts { font-size:19px; font-weight:600; margin:7px 0; }+ .meta { font-size:13px; color:var(--muted); }+ .id { overflow-wrap:anywhere; margin-top:4px; font:11px/1.6 ui-monospace,monospace; color:var(--muted); }+ .links { display:flex; flex-wrap:wrap; align-items:center; gap:18px; margin:24px 0 18px; }+ .button { display:inline-block; background:var(--text); color:var(--card); padding:10px 15px; border-radius:6px; text-decoration:none; font-weight:600; font-size:14px; }+ .guide { font-size:13px; }+ .note { font-size:13px; color:var(--muted); margin:19px 0 38px; }+ .start { display:flex; gap:28px; justify-content:space-between; align-items:center; padding:25px 0; border-top:1px solid var(--line); border-bottom:1px solid var(--line); }+ .start h2 { font-size:22px; margin:0 0 6px; }+ .start p { margin:0; color:var(--muted); font-size:14px; max-width:650px; }+ footer { margin-top:25px; color:var(--muted); font-size:12px; }+ @media (prefers-color-scheme:dark) { :root { --bg:#151714; --card:#1d201b; --text:#f2f4ed; --muted:#aeb6a6; --line:#363d31; --accent:#a0cda5; --tint:#273024; } }+ @media (max-width:700px) { main { margin:36px auto; padding:0 20px; } .grid { grid-template-columns:1fr; } article { padding:24px; } .start { align-items:flex-start; flex-direction:column; gap:18px; } .bar div { padding:16px 20px; } .bar span { font-size:10px; } }+ </style>+</head>+<body>+ <header class="bar"><div><a class="brand" href="https://paperclip.ing">Paperclip</a><span>Evaluation directory</span></div></header>+ <main>+ <div class="eyebrow">Evidence for better agents</div>+ <h1>Paperclip Evals</h1>+ <p class="intro">Two ways to test Paperclip. Follow each program from a test case to its evidence, results, and run history.</p>+ <div class="grid">+ <article>+ <div class="number">01 / RUNNER</div>+ <h2>Runner Evals</h2>+ <p>Does the runner use tools correctly, respect permissions, and preserve its execution contract?</p>+ <div class="boundary">Real runner + model provider<br>Controlled mock Paperclip control plane</div>+ <div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$runner_counts</div><div class="meta">$runner_date · $runner_coverage</div><div class="id">$runner_id</div></div>+ <div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html">Browse run history →</a><a href="$runner_url">Open report</a></div>+ <a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/packages/paperclip-runner/docs/runner-protocol-live-evals.md">Runner operations guide</a>+ </article>+ <article>+ <div class="number">02 / PRODUCT</div>+ <h2>Product E2E Evals</h2>+ <p>Can a user complete work through Paperclip, from the first request to a usable result?</p>+ <div class="boundary">Real browser + server + database + runner + model provider<br>Local and Daytona environments, as selected</div>+ <div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$product_counts</div><div class="meta">$product_date · $product_coverage</div><div class="id">$product_id</div></div>+ <div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/">Browse run history →</a><a href="$product_url">Open report</a></div>+ <a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/tests/runner-e2e/README.md">Product E2E operations guide</a>+ </article>+ </div>+ <p class="note">Snapshot refreshed $refreshed. History pages may contain newer runs. A partial campaign covers only its selected cases. These programs measure different boundaries; their pass rates are not combined.</p>+ <section class="start"><div><h2>Choose a test. Make the result inspectable.</h2><p>The eval guide explains which program to use, how to add a case, and how to distinguish a product defect from model behavior, a grading error, or a setup failure. Evalbook is a report framework used to present evidence; it is not an execution boundary.</p></div><a class="button" href="$guide_url">Read the eval guide →</a></section>+ <footer>Existing run archives remain the source of truth. Report refreshes do not count as new model runs.</footer>+ </main>+</body>+</html>diff --git a/scripts/evals-hub/test_build.py b/scripts/evals-hub/test_build.pynew file mode 100644index 0000000000..2960efd827--- /dev/null+++ b/scripts/evals-hub/test_build.py@@ -0,0 +1,82 @@+import copy+import unittest++from build import SYSTEMS, summarize+++class HistorySummaryTests(unittest.TestCase):+ def history(self, kind="runner"):+ prefix, schema = SYSTEMS[kind]+ campaign = {+ "campaignId": "run-1", "generatedAt": "2026-09-01T12:00:00Z",+ "publicUrl": f"https://d1p6rlowie26tp.cloudfront.net/{prefix}/campaigns/run-1/",+ "complete": False,+ }+ counts = {"selected": 3, "passed": 2}+ campaign.update({"totals": counts} if kind == "runner" else counts)+ return {"schema": schema, "latestCampaignId": "run-1", "campaigns": [campaign]}++ def test_both_families_keep_partial_coverage(self):+ for kind in SYSTEMS:+ with self.subTest(kind=kind):+ result = summarize(self.history(kind), kind)+ self.assertEqual(result["coverage"], "Partial campaign")+ self.assertEqual(result["counts"], "2 / 3 selected cases passed")++ def test_refresh_changes_report_but_not_measurement_date(self):+ history = self.history()+ refreshed = copy.deepcopy(history["campaigns"][0])+ refreshed.update(campaignId="run-1-refresh", generatedAt="2026-09-03T12:00:00Z")+ refreshed["publicUrl"] += "refresh/"+ refreshed["reportRevision"] = {+ "sourceCampaignId": "run-1", "sourceGeneratedAt": "2026-09-01T12:00:00Z",+ "renderedAt": "2026-09-03T12:00:00Z",+ }+ history["campaigns"].append(refreshed)+ result = summarize(history, "runner")+ self.assertEqual(result["id"], "run-1")+ self.assertIn("01 Sep 2026", result["date"])+ self.assertTrue(result["url"].endswith("/refresh/"))++ def test_missing_latest_does_not_fall_back_to_an_arbitrary_run(self):+ history = self.history()+ history["latestCampaignId"] = "absent"+ with self.assertRaises(ValueError):+ summarize(history, "runner")++ def test_refresh_without_measurement_date_is_rejected(self):+ history = self.history()+ campaign = history["campaigns"][0]+ campaign["campaignId"] = "run-1-refresh"+ campaign["reportRevision"] = {+ "sourceCampaignId": "run-1", "renderedAt": "2026-09-03T12:00:00Z",+ }+ with self.assertRaisesRegex(ValueError, "measurement date"):+ summarize(history, "runner")++ def test_unknown_schema_and_missing_coverage_are_rejected(self):+ for field in ("schema", "complete"):+ history = self.history()+ if field == "schema":+ history["schema"] = "future-schema"+ else:+ del history["campaigns"][0]["complete"]+ with self.subTest(field=field), self.assertRaises(ValueError):+ summarize(history, "runner")++ def test_bad_counts_cannot_be_published(self):+ history = self.history()+ history["campaigns"][0]["totals"]["passed"] = 4+ with self.assertRaises(ValueError):+ summarize(history, "runner")++ def test_external_report_urls_are_rejected(self):+ history = self.history()+ for url in ("javascript:alert(1)", "https://example.com/", "https://d1p6rlowie26tp.cloudfront.net/other/"):+ history["campaigns"][0]["publicUrl"] = url+ with self.subTest(url=url), self.assertRaises(ValueError):+ summarize(history, "runner")+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.mdindex 912497d27d..8f20166302 100644--- a/tests/runner-e2e/README.md+++ b/tests/runner-e2e/README.md@@ -1,5 +1,11 @@# Paid runner full-stack E2E+For family selection, ownership, provenance, history, and failure taxonomy,+see the [Paperclip evaluation guide](../../doc/evals.md). This README is the+authoritative runbook for Product E2E runner cells; the separate Runner Evals+protocol guide lives at+`packages/paperclip-runner/docs/runner-protocol-live-evals.md`.+This is the billable browser acceptance campaign system for Paperclip runnerprofiles. It is deliberately separate from `tests/e2e`: every independentlyscheduled execution gets