Skip to content

Turn any idea, plan, or codebase into a beautiful interactive diagram. An agent skill for Claude Code, Codex, and more.

fix: keep model benchmark tool access fair

tt-a1i13c65f392f4a
diff --git a/benchmarks/ordinary-model-floor/README.md b/benchmarks/ordinary-model-floor/README.md
index 0f386de1..40d4c6cf 100644
--- a/benchmarks/ordinary-model-floor/README.md
+++ b/benchmarks/ordinary-model-floor/README.md
@@ -24,7 +24,7 @@ node benchmarks/ordinary-model-floor/benchmark.mjs check --manifest benchmarks/o
## Fair-run protocol
-Every compared configuration must use the same prompt, the same repository commit, the same Archify skill and schema, the same time limit, and a clean candidate output path. Record the exact agent and model names. One complete agent invocation is attempt 1: the agent may use Archify's own diagnostics inside that invocation, but the resulting `benchmark-candidate.json` is frozen when the invocation ends. Preserve that unedited candidate and permit no post-hoc edits, including human edits, before verification.
+Every compared configuration must use the same prompt, the same repository commit, the same Archify skill and schema, the same time limit, identical tool access, and a clean candidate output path. Record the exact agent and model names. One complete agent invocation is attempt 1; the resulting `benchmark-candidate.json` is frozen when the invocation ends. If the prompt assigns validation to the external harness, the agent must not self-certify the result. Preserve that unedited candidate and permit no post-hoc edits, including human edits, before verification.
Do not let a later correction replace attempt 1. Correction attempts may be retained for diagnosis, but the first-pass report accepts only attempt 1 receipts. Run every case in the manifest for every configuration; an incomplete or duplicated matrix is not evidence-eligible.
diff --git a/benchmarks/ordinary-model-floor/benchmark.mjs b/benchmarks/ordinary-model-floor/benchmark.mjs
index 6bb6c0e7..fcaae082 100644
--- a/benchmarks/ordinary-model-floor/benchmark.mjs
+++ b/benchmarks/ordinary-model-floor/benchmark.mjs
@@ -20,6 +20,7 @@ const PROMPT_CONTRACT_PHRASES = [
'Do not edit any other file.',
'The candidate file, not the prose response, is the attempt 1 artifact.',
'Do not copy a checked-in example.',
+ 'Do not claim that validation passed; the external harness will validate the frozen file.',
];
class BenchmarkError extends Error {
diff --git a/benchmarks/ordinary-model-floor/prompts/agent-run.lifecycle.md b/benchmarks/ordinary-model-floor/prompts/agent-run.lifecycle.md
index 917a3bb4..0281a770 100644
--- a/benchmarks/ordinary-model-floor/prompts/agent-run.lifecycle.md
+++ b/benchmarks/ordinary-model-floor/prompts/agent-run.lifecycle.md
@@ -4,4 +4,4 @@ Use the Archify skill in this repository to create a lifecycle diagram for an ag
Author a fresh typed JSON diagram specification and validate it with the `showcase` quality profile. Choose your own stable internal IDs and layout. Keep active work, interruptions, recovery, and terminal outcomes legible.
-Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. Run the real Archify CLI against that file with the `showcase` quality profile; you may diagnose and correct your own candidate during this single agent invocation. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example.
+Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example. Do not claim that validation passed; the external harness will validate the frozen file.
diff --git a/benchmarks/ordinary-model-floor/prompts/agent-tool-call.workflow.md b/benchmarks/ordinary-model-floor/prompts/agent-tool-call.workflow.md
index c68f073d..b8d6a847 100644
--- a/benchmarks/ordinary-model-floor/prompts/agent-tool-call.workflow.md
+++ b/benchmarks/ordinary-model-floor/prompts/agent-tool-call.workflow.md
@@ -4,4 +4,4 @@ Use the Archify skill in this repository to create a workflow diagram for an AI
Author a fresh typed JSON diagram specification and validate it with the `showcase` quality profile. Choose your own stable internal IDs and layout. Make the approval decision, denied route, execution route, and external call easy to read.
-Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. Run the real Archify CLI against that file with the `showcase` quality profile; you may diagnose and correct your own candidate during this single agent invocation. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example.
+Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example. Do not claim that validation passed; the external harness will validate the frozen file.
diff --git a/benchmarks/ordinary-model-floor/prompts/cache-miss.sequence.md b/benchmarks/ordinary-model-floor/prompts/cache-miss.sequence.md
index 7e6ab4ad..c0eb13b3 100644
--- a/benchmarks/ordinary-model-floor/prompts/cache-miss.sequence.md
+++ b/benchmarks/ordinary-model-floor/prompts/cache-miss.sequence.md
@@ -4,4 +4,4 @@ Use the Archify skill in this repository to create a sequence diagram for a dash
Author a fresh typed JSON diagram specification and validate it with the `showcase` quality profile. Choose your own stable internal IDs and layout. Preserve message direction and distinguish calls, returns, security checks, and asynchronous trace emission.
-Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. Run the real Archify CLI against that file with the `showcase` quality profile; you may diagnose and correct your own candidate during this single agent invocation. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example.
+Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example. Do not claim that validation passed; the external harness will validate the frozen file.
diff --git a/benchmarks/ordinary-model-floor/prompts/product-analytics.dataflow.md b/benchmarks/ordinary-model-floor/prompts/product-analytics.dataflow.md
index 84dbe79e..42d2c671 100644
--- a/benchmarks/ordinary-model-floor/prompts/product-analytics.dataflow.md
+++ b/benchmarks/ordinary-model-floor/prompts/product-analytics.dataflow.md
@@ -4,4 +4,4 @@ Use the Archify skill in this repository to create a data-flow diagram for produ
Author a fresh typed JSON diagram specification and validate it with the `showcase` quality profile. Choose your own stable internal IDs and layout. Make the privacy boundary and restricted identity path visually distinct from the ordinary analytics path.
-Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. Run the real Archify CLI against that file with the `showcase` quality profile; you may diagnose and correct your own candidate during this single agent invocation. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example.
+Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example. Do not claim that validation passed; the external harness will validate the frozen file.
diff --git a/benchmarks/ordinary-model-floor/prompts/web-runtime.architecture.md b/benchmarks/ordinary-model-floor/prompts/web-runtime.architecture.md
index 7cac1129..7eae0019 100644
--- a/benchmarks/ordinary-model-floor/prompts/web-runtime.architecture.md
+++ b/benchmarks/ordinary-model-floor/prompts/web-runtime.architecture.md
@@ -4,4 +4,4 @@ Use the Archify skill in this repository to create an architecture diagram for a
Author a fresh typed JSON diagram specification and validate it with the `showcase` quality profile. Choose your own stable internal IDs and layout. Preserve the system roles and labelled technical relationships.
-Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. Run the real Archify CLI against that file with the `showcase` quality profile; you may diagnose and correct your own candidate during this single agent invocation. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example.
+Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example. Do not claim that validation passed; the external harness will validate the frozen file.