Skip to content

Turn any idea, plan, or codebase into a beautiful interactive diagram. An agent skill for Claude Code, Codex, and more.

docs(benchmark): define real repository generation comparison pilot

tt-a1ie1f35ac9ce4d
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 9248571c..99037060 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -17,6 +17,8 @@ Current-base trigger or rationale, intended outcome, why the approach is worth m
Comparison base and candidate head; applicable commands/results or evidence links. Identify reused evidence by its original revision and explain material coverage gaps. Follow CONTRIBUTING.md#choose-evidence-by-impact; required remote CI still applies.
+<!-- Generation-chain pilot PRs: link the real-repository comparison report (benchmarks/real-repo-generation/README.md), or explain the agreed narrower scope/exemption. Include original session records and both versions' artifacts; identify synthetic checks separately. -->
+
## Visual evidence
Omit this section for non-visual changes.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 40a29c5e..2438a530 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -88,6 +88,10 @@ source assertions when a behavior test owns that protection. Keep explicit
geometry, compatibility, accessibility, failure cleanup, and protection against
lost or corrupted output as observable contracts.
+### Real generation comparisons
+
+For PRs that change diagram generation, select and link before/after evidence using the [real repository benchmark](benchmarks/real-repo-generation/README.md). Its initial rollout is an agreed three-PR pilot; record the selected scope or a justified exemption before running. Contributors supply the comparison and session records; maintainers review outcomes and sample evidence. Synthetic error injection alone does not establish real Agent benefit. Reuse unaffected evidence at its recorded revision.
+
### CI by affected responsibility
CI uses the ownership groups in `scripts/ci-scope.mjs`. Product changes run core
diff --git a/REVIEWING.md b/REVIEWING.md
index a1f899eb..7f262007 100644
--- a/REVIEWING.md
+++ b/REVIEWING.md
@@ -25,6 +25,8 @@ Use existing tests, receipts, and browser tools. For behavior comparisons, use f
Evidence should be sufficient for the affected contract. Avoid rebuilding an unchanged artifact or replaying unaffected checks merely to restate existing results.
+For generation-comparison pilot PRs, use the [real repository benchmark review procedure](benchmarks/real-repo-generation/README.md#6-submit-and-review-one-report): inspect the paired outcomes and sample the source/session evidence before requesting reruns. A missing real-world benefit is an unverified claim; synthetic repair success does not establish it.
+
## Give actionable feedback
Lead with whether the approach is worthwhile, then explain findings and a clear disposition:
diff --git a/benchmarks/ordinary-model-floor/README.md b/benchmarks/ordinary-model-floor/README.md
index 44cea4cc..4338231e 100644
--- a/benchmarks/ordinary-model-floor/README.md
+++ b/benchmarks/ordinary-model-floor/README.md
@@ -1,5 +1,7 @@
# Ordinary-Model Floor v1
+**Metric scope:** historical `firstPassUsable` means usability after one complete Agent invocation, which may include repairs. It is not first-draft success. For PR comparisons with draft history and full sessions, use [real repository generation comparisons](../real-repo-generation/README.md).
+
This benchmark measures a narrow product question: can an ordinary coding agent produce an Archify diagram that is usable on attempt 1, without a human repairing the JSON?
It is a delivery gate, not a model leaderboard. A run is `firstPassUsable` only when all three gates pass:
diff --git a/benchmarks/real-repo-generation/README.md b/benchmarks/real-repo-generation/README.md
new file mode 100644
index 00000000..e6fc4a4f
--- /dev/null
+++ b/benchmarks/real-repo-generation/README.md
@@ -0,0 +1,90 @@
+# Real repository generation benchmark v1
+
+**Status: pilot protocol and task corpus; no generation runs or quality results yet.** Use this for agreed pilot PRs first. Evaluate the first three submissions before making it the default for all generation-chain PRs. Existing CI and merge authority remain unchanged. This directory supplies instructions and evidence templates; it does not launch Agents, capture sessions, or automatically score diagrams.
+
+The question is: **does the change help a real Agent deliver a factually correct, readable diagram with less rework?** Contributors provide comparable evidence; maintainers inspect changes and sample the underlying record. Quality comes before speed.
+
+## 1. Declare scope before running
+
+Use the five public, pinned projects in [projects.json](projects.json). Their tasks cover OpenPI delegation, Hive workers, MCO provider orchestration, Squad task coordination, and SkillRoster filesystem changes. These are five specific developer-tool projects, not a representative sample of every software domain. Read project sources in place and link to them; the corpus does not redistribute them or change their licenses.
+
+| Change | Default evidence |
+| --- | --- |
+| One diagram's layout, routes, typography, schema, or generation instructions | That diagram type on all five projects, once on base and once on candidate: 10 independent generation sessions. Add same-JSON replay when rendering or validation changes. |
+| Shared authoring, repair, delivery, or layout behavior | Architecture and workflow on all five projects: 20 sessions. Add representative saved real inputs for other affected types. Expand live generation only where the changed path is not exercised by that coverage. |
+| Shared Viewer presentation/export only | Replay saved real artifacts under the affected interaction, viewport and export conditions. Fresh source-to-diagram sessions are needed only if generation or repair behavior also changes. |
+| Repository prose, metadata or unrelated CI changes | Explain why generation evidence is not applicable; use the existing focused checks. |
+
+Run the agreed matrix once the approach is settled and the PR is ready for final review, not on every push. Classify by actual callers, not PR title or file extension. A narrower run is acceptable when the author explains the unaffected paths and equivalent existing evidence; agree on the scope in the PR before spending on runs. A later expansion needs a concrete uncovered risk. No new per-PR approval form is required.
+
+Architecture and workflow have ready prompt variants in [prompts.md](prompts.md). For another type, agree on a natural source-backed task using the same projects before running. If a project has no meaningful instance (for example, no relational model for an ER diagram), record it as not applicable and select an appropriate public replacement task in advance. Keep five applicable projects where practical; explicitly report a smaller agreed matrix. Never invent a database, timing measurement or relationship to fill a slot. A new diagram type unsupported by base has candidate-only evidence, clearly labeled; it is not a before/after improvement.
+
+## 2. Freeze the comparison
+
+Copy [report-template.md](report-template.md) and record the run plan before generating:
+
+- The suite revision, selected cases/types, full source-project commits, exact prompt text and prompt hash. Keep these identical across arms. Update the corpus through a separate reviewed revision; do not silently track project HEAD.
+- The current PR target commit as **base**, and exact PR head as **candidate**. Materialize each tracked Archify revision in a separate clean local package/checkout and record its source SHA and package/tree hash. A release number alone is insufficient. Use the same preparation method for both; keep setup logs. Do not install or replace live Skills for this benchmark.
+- The Agent/client version, model ID, reasoning setting, available tools, browser/runtime/OS, resource limits and concurrency. Use a contributor-accessible capable coding model; keep the configuration identical within each comparison. Results across different configurations must not be pooled as version effects.
+- A common time budget, declared before launch (pilot default: 20 minutes per session). Crossing it records a timeout, not a quality score. If the budget proves unsuitable, preserve the failed run and rerun both arms under the revised limit. Use the same token/cost cap when the client offers one.
+
+Use one active generation session at a time by default and alternate base/candidate order between projects. Parallel runs are allowed with the same concurrency policy and environment on both sides; record queueing and contention and avoid speed claims when they are uncontrolled. Download/setup happens before the generation clock and is recorded separately.
+
+## 3. Run fresh natural-language tasks
+
+Use a new session or a fresh subagent with only the selected project checkout, that arm's Archify entry point, the composed prompt and ordinary project instructions. A subagent must not inherit this maintenance conversation, a prior answer, or another run's context. Disable cross-run memory where possible and disclose any unavoidable shared context. Keep benchmark reports, reference diagrams and the other arm outside the generation workspace. Isolate browser/local-storage state and output directories as well as chat history.
+
+The Agent reads source and follows that revision's Archify Skill to produce a diagram. It can validate, inspect the result, and repair naturally. Do not give it predetermined coordinates, a reference JSON, or hints about which PR behavior should improve. Preserve the requested business scope and permit layout choices. Run the project checkout read-only except for the isolated diagram output; the task does not require installing the project into a user's environment or invoking live services.
+
+Capture the earliest complete diagram JSON **before its first validation/render attempt**, every later candidate submitted to the CLI, command arguments/results, receipts and final artifacts. Use client tool logging/file snapshots or an external recorder, not a retrospective reconstruction of the first draft. Record the capture method. If the client cannot preserve a first draft or a file revision, mark that metric unavailable and disclose the gap; never substitute the final candidate. That run can still support a final-artifact claim, but not a first-draft or complete-repair-history claim.
+
+Retain all scheduled runs, failed commands, timeouts and repeats. Extra attempts receive new run IDs; a successful retry does not replace an earlier failure. If a claimed diagnostic fix never occurs naturally, report that the mechanism was not exercised. Add an original failure captured from real use when available; artificially corrupted inputs do not demonstrate Agent benefit.
+
+## 4. Retain inspectable evidence
+
+Each run includes:
+
+- Original composed prompt; run metadata from [run-template.json](run-template.json).
+- Exported session record with ordered messages, tool calls/results, timestamps and nested-session links/exports where used. Preserve commands, errors, edits and restarts, not just the final answer. Native exports are preferred; an event export is acceptable when its origin and completeness are stated. Private model reasoning is not required.
+- First JSON, ordered attempted JSON versions, final JSON, HTML/SVG and CLI receipts when produced. Missing output is an explicit result.
+- Browser screenshots of the final artifact: light/Classic/Still, browser zoom 100%, 1440 × 900 CSS pixels, device scale 1 where supported. Retain both the initial viewport and a full-page image when needed; label any different conditions. Inspect dark mode or exports when affected. If defaults change, additionally show each version's actual default instead of hiding it behind forced settings.
+- A small semantic checklist with links into the pinned source for the key nodes, directions, conditions, ownership boundaries and failure paths. Have the reviewer verify this independently of the generated JSON; do not score agreement with a reference layout.
+
+Publish a redacted evidence bundle or durable artifact link with a digest and a retention/location note. Keep it accessible through review and the resulting release cycle. Remove tokens, credentials, private source, unrelated history and personal data; retain meaningful failures and annotate redactions. Session share links alone are insufficient when they omit tools or expire. Only publish material the contributor may disclose; a sanitization gap is reported, not filled with invented logs. A transcript aids audit but does not cryptographically prove that a run was authentic. Evidence marked complete must include reconstructible candidate revisions; if those are missing, limit the claim and list the missing evidence explicitly.
+
+Large traces and images belong in PR artifacts or an evidence archive, not the source tree. Do not fabricate an empty JSON or screenshot for an absent result.
+
+## 5. Compare output and authoring effort separately
+
+For layout, renderer or validator changes, also replay the same frozen real JSON on base and candidate. Use the **final JSON from both arms** when their content differs, plus the original real failure for a bug fix. This cross-replay adds no model invocation and helps distinguish authoring choices from renderer changes. Preserve the input bytes; report unsupported new fields or a base validation failure honestly. Retain failed outputs and diagnostics; a relaxed profile may be used for an explicitly labeled diagnostic view, not a passing result.
+
+Report these observations for each run, without collapsing them into a total score:
+
+| Observation | Definition |
+| --- | --- |
+| Semantics | Reviewer outcome: correct / incorrect / unverified, with concrete source-backed findings. |
+| Readability | Reviewer outcome: readable / issues / unreviewed; note clipping, route ambiguity, needless detours, unreadable text or wasted space. Automated checks are reported separately. |
+| Delivery | Delivered / failed / timeout / provider error / unsupported. CLI success alone does not establish semantic or visual acceptance. |
+| First draft | First captured JSON's check outcome, plus semantic/visual outcome only if inspected; unknown when not observed. A whole session with repairs is not a first draft. |
+| Repair rounds | Number of revised candidates after the first check/inspection cycle, grouped with their feedback and next validation. Multiple commands on unchanged JSON do not create another repair round. Record voluntary visual revisions as well as validation-driven ones. |
+| Tool work | Generation/validation tool calls, failures and repeated work, with the counting method. |
+| Time | Dispatch-to-final-response wall time, time to first JSON, CLI time and queue/wait time when observable. Use null with a reason when unavailable. Overlapping child/tool durations are work totals, not additive end-to-end latency. |
+
+The initial 10-session matrix is a diagnostic comparison, not a stable speed benchmark. Repeat the affected base/candidate pairs for a disputed regression or a speed claim; retain every repetition and report spread and completion failures alongside timing. Never advertise a percentage speedup from one pair or faster failed deliveries.
+
+## 6. Submit and review one report
+
+Fill [report-template.md](report-template.md). Present each project side by side with before/after screenshots and links to originals, JSON, replay and sessions. A Markdown report or static HTML page is sufficient; reviewers should not have to reconstruct the comparison from terminal logs. Put changed outcomes, regressions and coverage gaps first. Label author inspection, independent review, CI results and reused evidence separately.
+
+Maintainers first check revision/scope/evidence completeness, inspect the comparison images and source-backed findings, then sample one session and investigate every material regression. Re-run only disputed, unavailable or invalidated evidence. When base/head changes, explain which tested paths changed; reuse unaffected results under their original SHA instead of rerunning the matrix for unrelated prose or packaging updates.
+
+Unsupported facts, lost relationships, unreadable output or a new delivery failure need correction or an explicit maintainer decision. Intentional layout changes, pixel differences and minor timing noise are not automatic blockers. A validator disagreement can justify revisiting the validator, but retain the original failure and distinguish visual acceptance from the automated result. No aggregate score or screenshot similarity threshold authorizes a merge.
+
+For each of the first three pilot PRs, record execution cost when available, time spent assembling evidence, maintainer review time and useful defects found. Adjust the required scope or tooling if the process costs more than it saves. Until those runs exist, this is a protocol, not an established quality baseline or a new mandatory CI gate. A contributor without suitable model access may supply an original real case and request a maintainer-assisted run; mark the evidence pending rather than inventing it.
+
+## Existing suites
+
+- [repair-rounds](../repair-rounds/README.md) remains synthetic diagnostic regression coverage. It cannot demonstrate real Agent mistakes, rework savings or end-to-end generation improvement.
+- [ordinary-model-floor](../ordinary-model-floor/README.md) provides earlier bounded Agent-task evidence. Its historical `firstPassUsable` is task-level usability after one invocation, which can contain repairs; it must not be relabeled first-draft success.
+
+Reuse applicable existing CLI, semantic and browser checks with their actual coverage. This protocol adds real task evidence; it does not duplicate a provider SDK, model leaderboard or a second product validator.
diff --git a/benchmarks/real-repo-generation/projects.json b/benchmarks/real-repo-generation/projects.json
new file mode 100644
index 00000000..0fa59cca
--- /dev/null
+++ b/benchmarks/real-repo-generation/projects.json
@@ -0,0 +1,73 @@
+{
+ "suite_id": "real-repo-generation-v1",
+ "status": "pilot-unrun",
+ "frozen_on": "2026-10-16",
+ "primary_diagram_types": [
+ "architecture",
+ "workflow"
+ ],
+ "projects": [
+ {
+ "id": "openpi",
+ "name": "OpenPI",
+ "repository": "https://github.com/openpi-dev/openpi",
+ "commit": "677b4959108976b99fb63fb5029198b04aa51cb5",
+ "task": "Use Archify to explain how OpenPI accepts a task, delegates work to an independent subagent, and makes its status and result available to the caller. Read the checked-out source and focus on that path.",
+ "workflow_scenario": "A user delegates one bounded task to a subagent and receives its result. Include the actual failure or cancellation path where supported by the source.",
+ "reviewer_source_entrypoints": [
+ "extensions/",
+ "docs/architecture/"
+ ]
+ },
+ {
+ "id": "hive",
+ "name": "Hive",
+ "repository": "https://github.com/tt-a1i/hive",
+ "commit": "f8f1343da6595a232f0baba7f0ea5b5a58fe0e9c",
+ "task": "Use Archify to explain how a task in the Hive browser workbench reaches a worker running as a CLI process, and how progress and the result return to the team. Read the checked-out source and focus on that path.",
+ "workflow_scenario": "A team dispatches a task to a worker and observes its progress and completion. Include the actual failure or stop path where supported by the source.",
+ "reviewer_source_entrypoints": [
+ "src/cli/",
+ "src/server/"
+ ]
+ },
+ {
+ "id": "mco",
+ "name": "MCO",
+ "repository": "https://github.com/mco-org/mco",
+ "commit": "3cfce130362dbcc2aedda6170d1a31e39cf67d3e",
+ "task": "Use Archify to explain how MCO accepts one review request, runs the selected providers, and returns their individual results. Read the checked-out source and focus on that path.",
+ "workflow_scenario": "A user requests a read-only review with selected providers and inspects their results. Include how an unavailable or failed provider is handled in the implementation.",
+ "reviewer_source_entrypoints": [
+ "bin/",
+ "docs/contracts/"
+ ]
+ },
+ {
+ "id": "squad",
+ "name": "Squad",
+ "repository": "https://github.com/mco-org/squad",
+ "commit": "8146bcc1c38c439aedaf3ff44548c830654c8621",
+ "task": "Use Archify to explain how Squad coordinates a manager and worker through its CLI and persisted task/message state. Read the checked-out source and focus on that path.",
+ "workflow_scenario": "A manager assigns work, a worker finds it and reports completion, and the manager checks the outcome. Include any actual ownership or invalid-state check relevant to that path.",
+ "reviewer_source_entrypoints": [
+ "src/main.rs",
+ "src/store.rs",
+ "src/tasks.rs"
+ ]
+ },
+ {
+ "id": "skillroster",
+ "name": "SkillRoster",
+ "repository": "https://github.com/tt-a1i/skillroster",
+ "commit": "21d14c7cff25c66a016562e238e62d0538eb696f",
+ "task": "Use Archify to explain how SkillRoster goes from discovering installed Skills to planning, applying, and undoing a change. Read the checked-out source and focus on the boundaries between inspection and filesystem changes.",
+ "workflow_scenario": "A user inspects installed Skills, reviews a change plan, applies it, and later requests undo. Include the actual confirmation and conflict checks that protect existing files.",
+ "reviewer_source_entrypoints": [
+ "src/scan.rs",
+ "src/change.rs",
+ "src/roster_plan.rs"
+ ]
+ }
+ ]
+}
diff --git a/benchmarks/real-repo-generation/prompts.md b/benchmarks/real-repo-generation/prompts.md
new file mode 100644
index 00000000..c3e6c656
--- /dev/null
+++ b/benchmarks/real-repo-generation/prompts.md
@@ -0,0 +1,23 @@
+# Composing a generation prompt
+
+For architecture/workflow, use the exact `task` text from the selected entry in [projects.json](projects.json), followed by one variant below. Save the fully composed prompt before launching either arm. Replace `<OUTPUT_DIR>` with the equivalent isolated output path and supply the chosen Archify `SKILL.md` path as environment context. Source paths and output roots may differ; all substantive task text stays identical. The recorder handles evidence collection outside the Agent task.
+
+## Architecture
+
+Create an architecture diagram of that implementation. Show the main components, their responsibilities, communication paths and important boundaries at a useful overview level. Ground the relationships in source code and keep relevant source references. Use readable labels and a clear layout for a laptop screen. Deliver the diagram and its source JSON in <OUTPUT_DIR> using Archify's normal workflow. Keep application source unchanged. If something cannot be established from the code, explain the uncertainty rather than inventing it.
+
+## Workflow
+
+Create a workflow diagram for this scenario:
+
+<workflow_scenario from the same project entry>
+
+Show the actual steps, decisions, branches and failure/recovery paths needed to understand the scenario. Ground them in source code and keep relevant source references. Use readable labels and a clear layout for a laptop screen. Deliver the diagram and its source JSON in <OUTPUT_DIR> using Archify's normal workflow. Keep application source unchanged. If something cannot be established from the code, explain the uncertainty rather than inventing it.
+
+## Other diagram types
+
+The pilot corpus is calibrated first on architecture/workflow. For another affected type, record a source-backed natural-language scenario and any project substitutions in the PR's run plan before either arm runs. Use the same prompt in both arms. Do not turn a process into an invented relational schema or assign invented durations to make the diagram type applicable.
+
+## Reviewer context
+
+The `reviewer_source_entrypoints` in the manifest are starting points verified to exist at the pinned revisions, not a complete semantic oracle. Reviewers trace the task and record the key facts in the report. They are not a node list or a layout supplied to the generating Agent. Source truth, not visual similarity to another generated diagram, determines correctness.
diff --git a/benchmarks/real-repo-generation/report-template.md b/benchmarks/real-repo-generation/report-template.md
new file mode 100644
index 00000000..c29c5838
--- /dev/null
+++ b/benchmarks/real-repo-generation/report-template.md
@@ -0,0 +1,53 @@
+# Generation comparison: PR <number>
+
+Template only: replace placeholders, retain failures and use **unknown/not run** for missing evidence. A completed form is not proof that its claims are true.
+
+## Decision summary
+
+- User problem and intended benefit:
+- Observed improvement / unchanged cases / regressions:
+- Evidence status: author report / independently checked / reused / pending:
+- Important limitations and requested decisions:
+
+## Frozen plan
+
+- Suite revision and source-project pins: <projects.json copy/hash>
+- Base Archify commit + package/tree hash:
+- Candidate commit + package/tree hash:
+- Cases/types and any narrower scope or substitutions, with rationale:
+- Agent/client, model, reasoning, tools, runtime, browser/OS:
+- Time/resource budget, concurrency/order and cache/memory policy:
+- Prompt files + hashes; session and file-history capture method:
+- Conditions changed between arms, if any:
+
+## Results
+
+Add one row per project/type/repetition. Show all planned cases, including failures. Keep different model configurations separate.
+
+| Project / type / repetition | Base semantics / readability / delivery | Candidate semantics / readability / delivery | First-draft outcome, base → candidate | Repair rounds, base → candidate | Total time, base → candidate | What changed / evidence |
+| --- | --- | --- | --- | --- | --- | --- |
+| <project / type / 1> | <outcomes> | <outcomes> | <outcomes or unknown> | <counts or unknown> | <seconds or unknown> | <links> |
+
+## Visual and source comparison
+
+Repeat this section for each project/type. Include a named reviewer and explicit author/independent-review status.
+
+| Base screenshot | Candidate screenshot |
+| --- | --- |
+| <embedded image, link full size> | <embedded image, link full size> |
+
+- Original HTML / first JSON / final JSON / session: <base links> · <candidate links>
+- Same-JSON replay: <both origins on both versions, or why inapplicable>
+- Screenshot conditions and any differing defaults:
+- Semantic facts checked, pinned source links, and failures/uncertainties:
+- Readability observations; reviewer and inspection status:
+- Automated results (separate from visual review):
+
+## Audit and cost
+
+- Evidence bundle URL, digest, retention/location, native session export provenance:
+- Missing outputs, unavailable metrics, redactions and truncated logs:
+- Every retry/timeout/provider failure and disposition:
+- Reused evidence with original SHA and relevance to current head:
+- Execution cost if available; evidence assembly time; maintainer review time:
+- Useful defects discovered and any excessive verification work:
diff --git a/benchmarks/real-repo-generation/run-template.json b/benchmarks/real-repo-generation/run-template.json
new file mode 100644
index 00000000..54db4d9e
--- /dev/null
+++ b/benchmarks/real-repo-generation/run-template.json
@@ -0,0 +1,71 @@
+{
+ "template": true,
+ "suite_id": "real-repo-generation-v1",
+ "suite_commit": null,
+ "run_id": null,
+ "project_id": null,
+ "project_commit": null,
+ "diagram_type": null,
+ "arm": null,
+ "repetition": 1,
+ "archify": {
+ "commit": null,
+ "package_or_tree_sha256": null,
+ "preparation_log": null
+ },
+ "agent": {
+ "client": null,
+ "version": null,
+ "model": null,
+ "reasoning": null,
+ "tools": [],
+ "memory_policy": null
+ },
+ "environment": {
+ "os": null,
+ "runtime": null,
+ "browser": null,
+ "concurrency": 1,
+ "time_limit_seconds": 1200
+ },
+ "prompt": {
+ "path": null,
+ "sha256": null
+ },
+ "session": {
+ "id": null,
+ "export": null,
+ "capture_method": null,
+ "child_exports": [],
+ "redactions": [],
+ "completeness": null
+ },
+ "artifacts": {
+ "first_json": null,
+ "attempted_json_versions": [],
+ "final_json": null,
+ "html": null,
+ "receipts": [],
+ "screenshots": []
+ },
+ "observations": {
+ "delivery": null,
+ "semantics": null,
+ "readability": null,
+ "reviewer": null,
+ "review_origin": null,
+ "findings": [],
+ "first_draft_check": null,
+ "repair_rounds": null,
+ "tool_calls": null
+ },
+ "timing": {
+ "dispatched_at": null,
+ "first_json_at": null,
+ "finished_at": null,
+ "wall_seconds": null,
+ "cli_work_seconds": null,
+ "queue_wait_seconds": null
+ },
+ "unavailable": []
+}
diff --git a/benchmarks/repair-rounds/README.md b/benchmarks/repair-rounds/README.md
index a48c27e5..fec2c132 100644
--- a/benchmarks/repair-rounds/README.md
+++ b/benchmarks/repair-rounds/README.md
@@ -1,5 +1,7 @@
# Repair Rounds v1
+**Evidence scope:** synthetic diagnostic regression checks. The injected inputs and rules repairer are not real Agent behavior. Use [real repository generation comparisons](../real-repo-generation/README.md) for claims about authoring quality, rework or end-to-end speed.
+
This benchmark measures a narrow product question: **how efficiently do Archify receipts let a caller converge to a passing artifact?** It complements `ordinary-model-floor` (first-pass usability) and the `authoring-cost` studies (real agent time and token spend) by isolating the part of authoring cost that lives inside the feedback loop — how many validation rounds a defect costs, how early each defect is surfaced, and how much text each round forces the caller to read.
It is a measurement harness, not a model leaderboard, and it contains no human or model judgment. Everything is driven by deterministic defect injection against the checked-in example fixtures and the real Archify CLI.
@@ -15,7 +17,7 @@ It is a measurement harness, not a model leaderboard, and it contains no human o
| Actionability | repairs a rules-only repairer could derive from diagnostics alone; `unactionable` receipts are counted honestly |
| Observable token cost | a deterministic lexical estimate over the required-reading set, each round's receipt, and each round's candidate |
-A defect reported only after earlier gates passed means at least one extra authoring round in the real loop; `viewer/viewport-overflow` is the canonical example. Diagnostics that name no subject, carry no evidence, and suggest no fix produce `unactionable` repairs rather than silent passes.
+A late defect such as `viewer/viewport-overflow` can add another check in this scripted loop. The benchmark does not establish its cost to a real Agent. Diagnostics that name no subject, carry no evidence, and suggest no fix produce `unactionable` repairs rather than silent passes.
## Defect classes
@@ -50,7 +52,7 @@ Reports require one command and repair mode per input file; mixed configurations
## Repair modes
-- `rules` (default): the repairer sees only the receipt — `subject` paths, `evidence`, `supportedFixes`, and pixel figures inside `layout/constraint` messages. It models the weakest honest agent: repairs derived purely from what the tool reports.
+- `rules` (default): the repairer sees only the receipt — `subject` paths, `evidence`, `supportedFixes`, and pixel figures inside `layout/constraint` messages. It is a mechanical consumer of repairs derivable from what the tool reports; it does not model Agent reasoning.
- `oracle`: each attributed defect is restored exactly. This upper bound isolates staging (how early defects surface) from actionability (whether the receipt suffices to fix them).
## Metrics provenance