Turn any idea, plan, or codebase into a beautiful interactive diagram. An agent skill for Claude Code, Codex, and more.
HTML agent-skillsai-agentsarchitecture-diagramclaude-codeclaude-skillscodexcoding-agentsdeepseek-harnessdeveloper-toolsdiagramsdiagrams-as-codedsh-pluginflowchartllmmermaidopencodesequence-diagramsoftware-architecturesystem-designvisualization
test: measure agent-repaired diagram quality
tt-a1ib47a8228c98a
diff --git a/archify/test/ordinary-model-floor.test.mjs b/archify/test/ordinary-model-floor.test.mjsindex 7c207e07..84a76822 100644--- a/archify/test/ordinary-model-floor.test.mjs+++ b/archify/test/ordinary-model-floor.test.mjs@@ -382,7 +382,9 @@ test('benchmark records a timeout without a candidate as a complete first-pass ftest('benchmark semantic requirements bind by accepted technical labels instead of forcing model-authored internal IDs', () => {const source = JSON.parse(fs.readFileSync(path.join(skillRoot, 'examples/web-app.architecture.json'), 'utf8'));+ source.components.find((component) => component.id === 'users').label = 'Browser Users';const rename = new Map([+ ['users', 'browser-users-v1'],['api', 'service-api-v1'],['cache', 'redis-cache-v1'],]);@@ -391,6 +393,9 @@ test('benchmark semantic requirements bind by accepted technical labels insteadconnection.from = rename.get(connection.from) || connection.from;connection.to = rename.get(connection.to) || connection.to;}+ source.connections.find(+ (connection) => connection.from === 'service-api-v1' && connection.to === 'redis-cache-v1',+ ).label = 'cache read-through GET / SET';for (const view of source.meta.views || []) {view.focus = view.focus.map((id) => rename.get(id) || id);}@@ -405,6 +410,7 @@ test('benchmark semantic requirements bind by accepted technical labels insteadquality_profile: 'showcase',requirements: {nodes: [+ { key: 'users', labels: ['Users'], type: 'external' },{ key: 'api', labels: ['API', 'API Server'], type: 'backend' },{ key: 'cache', labels: ['Redis', 'Redis Cache'], type: 'database' },],@@ -432,6 +438,7 @@ test('benchmark semantic requirements bind by accepted technical labels insteadconst receipt = JSON.parse(result.stdout);assert.equal(receipt.gates.semantic.ok, true);assert.deepEqual(receipt.gates.semantic.bindings, {+ users: 'browser-users-v1',api: 'service-api-v1',cache: 'redis-cache-v1',});@@ -467,7 +474,7 @@ test('checked-in benchmark suite covers all five diagram types without presentin}});-test('checked-in prompts assign deterministic validation only to the external harness', () => {+test('checked-in prompts permit bundled CLI repair while retaining external validation authority', () => {const suiteRoot = path.join(repoRoot, 'benchmarks/ordinary-model-floor');const manifest = JSON.parse(fs.readFileSync(path.join(suiteRoot, 'manifest.json'), 'utf8'));@@ -475,10 +482,14 @@ test('checked-in prompts assign deterministic validation only to the external haconst prompt = fs.readFileSync(path.join(suiteRoot, entry.prompt), 'utf8');assert.match(prompt,- /Target the `showcase` quality profile; the external harness will validate the frozen candidate\./,+ /Use the bundled Archify CLI to validate and repair the candidate when shell access is available\./,+ entry.prompt,+ );+ assert.match(+ prompt,+ /The external harness will independently validate the frozen candidate\./,entry.prompt,);- assert.doesNotMatch(prompt, /and validate it with the `showcase` quality profile/, entry.prompt);}});@@ -622,6 +633,8 @@ test('benchmark documentation locks the fair-run and truthful-evidence contract''same repository commit','packaged skill root','model-visible working tree',+ 'bundled Archify CLI',+ 'independently revalidates','attempt 1','no post-hoc edits','Reference fixtures are not benchmark evidence',diff --git a/benchmarks/ordinary-model-floor/README.md b/benchmarks/ordinary-model-floor/README.mdindex 1ba17a19..1229cf7b 100644--- a/benchmarks/ordinary-model-floor/README.md+++ b/benchmarks/ordinary-model-floor/README.md@@ -24,7 +24,7 @@ node benchmarks/ordinary-model-floor/benchmark.mjs check --manifest benchmarks/o## Fair-run protocol-Every compared configuration must use the same prompt, the same repository commit, the same Archify skill and schema, the same time limit, identical tool access, and a clean candidate output path. Record the exact agent and model names. One complete agent invocation is attempt 1; the resulting `benchmark-candidate.json` is frozen when the invocation ends. If the prompt assigns validation to the external harness, the agent must not self-certify the result. Preserve that unedited candidate and permit no post-hoc edits, including human edits, before verification.+Every compared configuration must use the same prompt, the same repository commit, the same Archify skill and schema, the same time limit, identical tool access, and a clean candidate output path. Record the exact agent and model names. One complete agent invocation is attempt 1; the resulting `benchmark-candidate.json` is frozen when the invocation ends. The agent may use the bundled Archify CLI to validate and repair its candidate during that invocation; the external harness independently revalidates the frozen file and remains the final authority. Preserve that unedited candidate and permit no post-hoc edits, including human edits, before verification.Run candidate generation from the extracted **packaged skill root** produced by that commit, not from the development repository root. Keep the benchmark harness, cases, prompts, and reference fixtures outside the model-visible working tree; deliver the selected prompt through the external runner. This tests the surface users actually install and prevents benchmark internals from changing exploration cost or leaking evaluation evidence.diff --git a/benchmarks/ordinary-model-floor/benchmark.mjs b/benchmarks/ordinary-model-floor/benchmark.mjsindex 737bc01d..f449e1e2 100644--- a/benchmarks/ordinary-model-floor/benchmark.mjs+++ b/benchmarks/ordinary-model-floor/benchmark.mjs@@ -20,7 +20,8 @@ const PROMPT_CONTRACT_PHRASES = ['Do not edit any other file.','The candidate file, not the prose response, is the attempt 1 artifact.','Do not copy a checked-in example.',- 'Target the `showcase` quality profile; the external harness will validate the frozen candidate.',+ 'Use the bundled Archify CLI to validate and repair the candidate when shell access is available.',+ 'The external harness will independently validate the frozen candidate.','Do not claim that validation passed.',];const OPERATIONAL_FAILURES = new Set(['timeout', 'no_candidate', 'provider_error']);@@ -60,14 +61,22 @@ function normalizeTechnicalLabel(value) {.trim();}+function technicalLabelMatches(actual, expected) {+ const normalizedActual = normalizeTechnicalLabel(actual);+ const normalizedExpected = normalizeTechnicalLabel(expected);+ if (!normalizedActual || !normalizedExpected) return false;+ const paddedActual = ` ${normalizedActual} `;+ const paddedExpected = ` ${normalizedExpected} `;+ return paddedActual.includes(` ${normalizedExpected} `)+ || paddedExpected.includes(` ${normalizedActual} `);+}+function relationshipMatches(candidate, requirement) {return candidate.from === requirement.from&& candidate.to === requirement.to&& (requirement.label === undefined || candidate.label === requirement.label)&& (requirement.labels === undefined- || requirement.labels.some(- (label) => normalizeTechnicalLabel(candidate.label) === normalizeTechnicalLabel(label),- ))+ || requirement.labels.some((label) => technicalLabelMatches(candidate.label, label)))&& (requirement.variant === undefined || candidate.variant === requirement.variant);}@@ -86,7 +95,7 @@ function bindRequiredNode(required, nodesById, nodes) {if (acceptedLabels.length === 0) return { node: null, ambiguous: [] };const matches = nodes.filter(- (node) => acceptedLabels.includes(normalizeTechnicalLabel(node.label)),+ (node) => acceptedLabels.some((label) => technicalLabelMatches(node.label, label)),);if (matches.length === 1) return { node: matches[0], ambiguous: [] };if (matches.length > 1) return { node: null, ambiguous: matches.map((node) => node.id) };diff --git a/benchmarks/ordinary-model-floor/cases/web-runtime.architecture.case.json b/benchmarks/ordinary-model-floor/cases/web-runtime.architecture.case.jsonindex f9a77223..c31fe468 100644--- a/benchmarks/ordinary-model-floor/cases/web-runtime.architecture.case.json+++ b/benchmarks/ordinary-model-floor/cases/web-runtime.architecture.case.json@@ -13,7 +13,7 @@],"relationships": [{ "from": "users", "to": "cdn", "labels": ["HTTPS"] },- { "from": "api", "to": "cache", "labels": ["read-through", "cache read"] },+ { "from": "api", "to": "cache", "labels": ["read-through", "cache read", "GET / SET", "Redis access"] },{ "from": "api", "to": "db", "labels": ["SQL", "database query"] }]}diff --git a/benchmarks/ordinary-model-floor/prompts/agent-run.lifecycle.md b/benchmarks/ordinary-model-floor/prompts/agent-run.lifecycle.mdindex e2430a38..4aa0de4c 100644--- a/benchmarks/ordinary-model-floor/prompts/agent-run.lifecycle.md+++ b/benchmarks/ordinary-model-floor/prompts/agent-run.lifecycle.md@@ -2,6 +2,6 @@Use the Archify skill in this repository to create a lifecycle diagram for an agent run. Show the main ordered phases from queued through planning, execution, review, and completion. Execution may pause for human approval or fail recoverably. Review may block while waiting for input. A blocked run may expire, and a waiting approval may be cancelled by the user. Terminal exits must remain distinct from active and waiting states.-Author a fresh typed JSON diagram specification. Target the `showcase` quality profile; the external harness will validate the frozen candidate. Choose your own stable internal IDs and layout. Keep active work, interruptions, recovery, and terminal outcomes legible.+Author a fresh typed JSON diagram specification targeting the `showcase` quality profile. Choose your own stable internal IDs and layout. Keep active work, interruptions, recovery, and terminal outcomes legible. Use the bundled Archify CLI to validate and repair the candidate when shell access is available. The external harness will independently validate the frozen candidate.Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example. Do not claim that validation passed.diff --git a/benchmarks/ordinary-model-floor/prompts/agent-tool-call.workflow.md b/benchmarks/ordinary-model-floor/prompts/agent-tool-call.workflow.mdindex 2194f541..a43ebdd4 100644--- a/benchmarks/ordinary-model-floor/prompts/agent-tool-call.workflow.md+++ b/benchmarks/ordinary-model-floor/prompts/agent-tool-call.workflow.md@@ -2,6 +2,6 @@Use the Archify skill in this repository to create a workflow diagram for an AI agent tool call. A user request reaches a chat surface and planner. A router decides whether human approval is required. Approval permits tool execution against an external provider; denial enters a blocked path that may retry. The final result returns to the user, while tool results and context are recorded for tracing.-Author a fresh typed JSON diagram specification. Target the `showcase` quality profile; the external harness will validate the frozen candidate. Choose your own stable internal IDs and layout. Make the approval decision, denied route, execution route, and external call easy to read.+Author a fresh typed JSON diagram specification targeting the `showcase` quality profile. Choose your own stable internal IDs and layout. Make the approval decision, denied route, execution route, and external call easy to read. Use the bundled Archify CLI to validate and repair the candidate when shell access is available. The external harness will independently validate the frozen candidate.Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example. Do not claim that validation passed.diff --git a/benchmarks/ordinary-model-floor/prompts/cache-miss.sequence.md b/benchmarks/ordinary-model-floor/prompts/cache-miss.sequence.mdindex ef92926d..814d3b0c 100644--- a/benchmarks/ordinary-model-floor/prompts/cache-miss.sequence.md+++ b/benchmarks/ordinary-model-floor/prompts/cache-miss.sequence.md@@ -2,6 +2,6 @@Use the Archify skill in this repository to create a sequence diagram for a dashboard request. A browser calls an API, the API validates the JWT, then reads Redis. Redis returns a cache miss, so the API queries PostgreSQL for profile and metric data, stores the result back in Redis, emits a trace, and returns JSON for the browser to render.-Author a fresh typed JSON diagram specification. Target the `showcase` quality profile; the external harness will validate the frozen candidate. Choose your own stable internal IDs and layout. Preserve message direction and distinguish calls, returns, security checks, and asynchronous trace emission.+Author a fresh typed JSON diagram specification targeting the `showcase` quality profile. Choose your own stable internal IDs and layout. Preserve message direction and distinguish calls, returns, security checks, and asynchronous trace emission. Use the bundled Archify CLI to validate and repair the candidate when shell access is available. The external harness will independently validate the frozen candidate.Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example. Do not claim that validation passed.diff --git a/benchmarks/ordinary-model-floor/prompts/product-analytics.dataflow.md b/benchmarks/ordinary-model-floor/prompts/product-analytics.dataflow.mdindex ba861942..f8462ea5 100644--- a/benchmarks/ordinary-model-floor/prompts/product-analytics.dataflow.md+++ b/benchmarks/ordinary-model-floor/prompts/product-analytics.dataflow.md@@ -2,6 +2,6 @@Use the Archify skill in this repository to create a data-flow diagram for product analytics. Web and mobile clients send events to an edge ingestion API. Consent is checked before identity data enters a protected PII vault. Accepted events enter a stream, become normalized warehouse facts, and feed dashboards. Aggregated warehouse data also feeds a feature store and model.-Author a fresh typed JSON diagram specification. Target the `showcase` quality profile; the external harness will validate the frozen candidate. Choose your own stable internal IDs and layout. Make the privacy boundary and restricted identity path visually distinct from the ordinary analytics path.+Author a fresh typed JSON diagram specification targeting the `showcase` quality profile. Choose your own stable internal IDs and layout. Make the privacy boundary and restricted identity path visually distinct from the ordinary analytics path. Use the bundled Archify CLI to validate and repair the candidate when shell access is available. The external harness will independently validate the frozen candidate.Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example. Do not claim that validation passed.diff --git a/benchmarks/ordinary-model-floor/prompts/web-runtime.architecture.md b/benchmarks/ordinary-model-floor/prompts/web-runtime.architecture.mdindex 2b715389..b011fbb1 100644--- a/benchmarks/ordinary-model-floor/prompts/web-runtime.architecture.md+++ b/benchmarks/ordinary-model-floor/prompts/web-runtime.architecture.md@@ -2,6 +2,6 @@Use the Archify skill in this repository to create an architecture diagram for a production web application. Browser users enter through a CDN over HTTPS, then traffic reaches a load balancer and an application API. The API verifies identity with an auth provider, reads through a Redis cache, queries a PostgreSQL primary database, and enqueues background work for a worker. Static assets are served from object storage.-Author a fresh typed JSON diagram specification. Target the `showcase` quality profile; the external harness will validate the frozen candidate. Choose your own stable internal IDs and layout. Preserve the system roles and labelled technical relationships.+Author a fresh typed JSON diagram specification targeting the `showcase` quality profile. Choose your own stable internal IDs and layout. Preserve the system roles and labelled technical relationships. Use the bundled Archify CLI to validate and repair the candidate when shell access is available. The external harness will independently validate the frozen candidate.Write the final candidate to exactly `benchmark-candidate.json` in the repository root. Do not edit any other file. The candidate file, not the prose response, is the attempt 1 artifact. Do not copy a checked-in example. Do not claim that validation passed.