Official Codebase of 3DCodeVerse
Python
Condense docs/SKILLS.md to its measured facts (741 → 394 lines)
gaoypeng878309a2aec9
diff --git a/harness/docs/SKILLS.md b/harness/docs/SKILLS.mdindex db05d2f5..af9e7b74 100644--- a/harness/docs/SKILLS.md+++ b/harness/docs/SKILLS.md@@ -1,632 +1,339 @@# Skills-A **skill** is a task-scoped rule sheet the harness attaches to a coding session-automatically — from the track, the language, the round kind, the plan, and **the previous-round's gate findings**. Nobody writes `skills: [...]`.--It is not a recipe library. **No language ships a cookbook (D106 scene track, D107 the rest,-owner 2026-10-15)**: the interface and language facts live in `contract.md` / `system.md` (each-contract keeps ONE executable interface example), and a skill is a sheet of measured requirements-with a few SHORT optional practices (real API names, numbers only where a gate or the judge-measures them, no code block over ~12 lines) — never a chapter to copy, since the 60-iteration-0 argon scenes were the inlined chapters executed verbatim.--`SKILL.md` is an open standard (agentskills.io, Dec 2025), which is why the format below is-not ours to bend: **claude-code, codex and agy discover these bundles-themselves** — proven live, see §7 (gemini-cli did too, until its backend was deleted in D103). (The in-process `api-agent`, the one backend that-needed harness-side injection, was deleted 2026-08-28; bundles now reach a session only-through its vendor's native skill loader.)--Everything is behind `C3D_SKILLS`, and **since 2026-09-22 it is ON by default** —-`C3D_SKILLS=0` turns it off (an A/B's no-skills arm must now say so). It shipped OFF from-2026-08-25 because `api-agent` read 0 of 5 routed bundles in the measured A/B; that backend-was deleted 2026-08-28, and on 2026-09-22 all four vendor CLIs were watched, through their-own transcripts, indexing the bundles and activating the matching ones unprompted. §6-keeps the history; its dated note has the reasons and the evidence.+A **skill** is a task-scoped rule sheet the harness attaches to a coding session automatically —+from the track, the language, the round kind, the plan and **the previous round's gate+findings**. Nobody writes `skills: [...]`.++It is not a recipe library. No language ships a cookbook (D106 for the scene track, D107 for the+rest; owner, 2026-10-15): interface and language facts live in `contract.md` / `system.md` (one+executable example each), and a skill is a sheet of measured requirements with a few SHORT+optional practices — real API names, numbers only where a gate or the judge measures them, no+code block over ~12 lines. The 60 iteration-0 argon scenes were the inlined chapters executed+verbatim; that is what a chapter buys.++`SKILL.md` is an open standard ([agentskills.io](https://agentskills.io/specification), Dec 2025),+so the format is not ours to bend — and it is why the bundles reach a session only through its+vendor's native skill loader: claude-code, codex and agy discover them themselves (§7). The one+backend that needed harness-side injection, the in-process `api-agent`, read 0 of 5 routed+bundles and was deleted 2026-08-28.++`C3D_SKILLS` is **on by default since 2026-09-22** (D81); `C3D_SKILLS=0` is the off switch and an+A/B's no-skills arm. Knobs: `C3D_SKILLS_MAX` (5 bundles a session), `C3D_SKILLS_UNVERIFIED`+(the six `inherited-unverified` bundles, on), `C3D_SKILLS_ONLY` (restrict the library before+routing), `C3D_SKILLS_DIR` (another library).---## 0. How the library is managed-**One library, one policy, thin adapters.** The bundles under `codeverse3d/skills/<name>/`-are plain [agentskills.io](https://agentskills.io/specification) `SKILL.md` directories and-know nothing about any backend — anyone can `cp -r codeverse3d/skills/<name> ~/.claude/skills/`-and use them without this harness at all. `registry.py` picks the set from typed inputs.--Everything that genuinely differs between coding agents is **one bit**, and it lives in-one file, `skills/prompting.py`:+**One library, one policy, thin adapters.** The bundles under `codeverse3d/skills/<name>/` are+plain `SKILL.md` directories that know nothing about any backend — `cp -r` one into+`~/.claude/skills/` and it works without the harness. `registry.py` picks the set from typed+inputs. The one thing that differs between coding agents is the discovery root, in one file,+`skills/prompting.py` (`_BACKEND_ROOTS`, read by `skill_path`; `tests/skills/test_delivery.py`+fails if per-backend knowledge leaks anywhere else):| backend | discovery root ||---|---|| `claude-code` | `.claude/skills` || `codex`, `agy` | `.agents/skills` |-It was read out of the shipped binaries, not assumed — and re-read 2026-09-22 against the-installed versions, with what each CLI needs to see the bundles and what its own record of-a session says it did with them:+Read out of the shipped binaries (2026-09-22) and re-run live on 2026-11-03 (§7):-| CLI (2026-09-22) | workspace roots · user roots | what must hold | what the model is shown | activation, as its own record logs it |+| CLI | roots (workspace · user) | what must hold | what the model is shown | activation, as its own record logs it ||---|---|---|---|---|-| `claude-code` 2.1.280 | `.claude/skills` · `~/.claude/skills` (+ claude.ai-synced skills, plugins) | `Skill` in `--allowedTools`; never `--disable-slash-commands` ("Disable all skills") | name + description list; stream-json `init` event's `skills` | `Skill` tool `{"skill": name}`, then `Read` of `references/` |-| `gemini-cli` 0.53.0 (backend deleted in D103) | `.gemini/skills`, `.agents/skills` — **only in a trusted folder** · `~/.gemini/skills`, `~/.agents/skills` | `skills.enabled` (default true; the harness pins it in the per-session system settings, merged last) and folder trust (`folderTrust.enabled=false` ⇒ trusted) | `<available_skills>` XML (name, description, location) in the system prompt + `activate_skill` with an enum of names | `activate_skill` `{"name": name}` returns the body; `read_file` for `references/` |-| `codex` 0.155.1 | `.agents/skills` of the working root (`-C <ws>`, also the repo root) · `$CODEX_HOME/skills` (+ `.system`) | nothing — listed from an untrusted `/tmp` workspace too (`codex debug prompt-input`) | `## Skills` section: name, description and an aliased path `r1/<name>/SKILL.md` with a `### Skill roots` table | a shell `sed`/`cat` of `SKILL.md` (an `exec --json` `command_execution` item) |-| `agy` 1.2.2 | `.agents/skills` under the repo root · a global customization root under `~/.gemini/` | nothing; `--disable-slash-commands` only stops `/name` expansion of the prompt (1.1.9 changelog) | a skills section of its system prompt (paths included: its first calls were the four absolute `SKILL.md` paths, with no listing before them) | `view_file` of `SKILL.md` (a step in its conversation database) |+| `claude-code` 2.1.317 | `.claude/skills` · `~/.claude/skills` (+ synced skills, plugins) | `Skill` in `--allowedTools`; never `--disable-slash-commands` ("Disable all skills") | a name + description list; the stream-json `init` event's `skills` | `Skill {"skill": name}`, then `Read` — or a shell `cat` — of `references/` |+| `codex` 0.166.0 | `.agents/skills` of the working root · `$CODEX_HOME/skills` (+ `.system`) | nothing: listed from an untrusted `/tmp` workspace too (`codex debug prompt-input`) | a `## Skills` section with an aliased path `r1/<name>/SKILL.md` | a shell `sed`/`cat` of `SKILL.md` (an `exec --json` `command_execution` item) |+| `agy` 1.4.12 | `.agents/skills` under the repo root · a customization root under `~/.gemini/` | nothing; `--disable-slash-commands` only stops `/name` expansion of the prompt | a skills section of its system prompt, paths included | `read_file` of `SKILL.md` (`view_file` before 1.4) — a step in its conversation database |++(gemini-cli discovered them too, until its backend went in D103.)-Bundles reach a session only through its vendor's native skill loader: the harness writes them under the discovery root and-hands the agent no second index (the loader's own would list the same skills twice) and no tool;-the body file gets one MANDATORY sentence.+The harness writes the bundles under the discovery root and hands the agent no second index (the+loader's own would list them twice) and no tool; the body file gets one MANDATORY sentence.-**And a session sees nothing but the routed bundles** (2026-09-22). Every CLI also lists-skills of its own; each is switched off with the CLI's documented knob where it has one-(`agents/backends.py`), checked at zero cost against the real binaries with a local fake API-(the claude `init` event, the codex and gemini request bodies) and `codex debug prompt-input`:+**A session sees nothing but the routed bundles** (2026-09-22). Every CLI lists skills of its+own; each is switched off with the CLI's documented knob (`agents/backends.py`), checked at zero+cost against the real binaries with a local fake API:| CLI | listed besides the routed bundles | switch | left ||---|---|---|---|-| `claude-code` 2.1.280 | 26: bundled (`update-config`, `code-review`, `loop`, …), account-synced (`deep-research`, `anthropic-skills:*`, …) | `--setting-sources project` (synced skills and plugins come with the "user" source) + `CLAUDE_CODE_DISABLE_BUNDLED_SKILLS=1` + `--settings {"skillOverrides": {"design": "off", "doctor": "off"}}` (the two that survive the bundled switch by design) | nothing — and the owner's `~/.claude/settings.json` (effortLevel, hooks, env) no longer reaches a session |-| `codex` 0.155.1 | 5 `.system` (imagegen, openai-docs, plugin-creator, skill-creator, skill-installer) | `-c skills.bundled.enabled=false` | any user root (`~/.codex/skills`, `~/.agents/skills`): no stable switch; none here |-| `gemini-cli` 0.53.0 (deleted in D103) | 2 built-ins (skill-creator, antigravity-support) — `gemini skills list` hides them, the model does not | `skills.disabled` in the per-session system settings (by name) | the user roots (`~/.gemini/skills`, `~/.agents/skills`): no switch; none here |-| `agy` 1.2.8 | 5 built-ins (agy-customizations, antigravity-guide, generative_ui, migrate-workflows, permissioned-github; `agy -p /skills` lists them without a turn) | none per session: built-ins survive even an agent's `inherit_user: false` | all 5 |--**Adding a backend is one row in that table** (`_BACKEND_ROOTS`, read by `skill_path`). A test-(`tests/skills/test_delivery.py`) fails if per-backend knowledge leaks back out into the other-modules, and another checks that a model-qualified kind (`codex:…`) resolves to its backend's row.--### Why only native loaders--Measured 2026-08-25, same library, same workspace, same task: `codex`, `claude-code` and-`agy` each read **5 of 5** routed bundles unprompted through their own loaders.-`api-agent` read **0 of 5** — not from unwillingness (it made **52 `read_file` calls** that-session, and called `read_cookbook` twice), but because the bundles sit in hidden-directories its `list_files` deliberately skips, and the index was prose in a 2.3 kB system-prompt competing with the contract and the rules. Prose is not an affordance. The-`read_skill` tool built to fix that was deleted with `api-agent` on 2026-08-28; every backend-left discovers the bundles through its own loader.+| `claude-code` | 26: bundled (`update-config`, `code-review`, `loop`, …) and account-synced (`deep-research`, `anthropic-skills:*`, …) | `--setting-sources project` + `CLAUDE_CODE_DISABLE_BUNDLED_SKILLS=1` + `--settings {"skillOverrides": {"design": "off", "doctor": "off"}}` | nothing — and the owner's `~/.claude/settings.json` (effortLevel, hooks, env) no longer reaches a session |+| `codex` | 5 `.system` (imagegen, openai-docs, plugin-creator, skill-creator, skill-installer) | `-c skills.bundled.enabled=false` | any user root (`~/.codex/skills`, `~/.agents/skills`): no switch; none here |+| `agy` | 5 built-ins (agy-customizations, antigravity-guide, generative_ui, migrate-workflows, permissioned-github) | none per session | all 5 |+**Why only native loaders.** Measured 2026-08-25, same library, workspace and task: codex,+claude-code and agy each read 5 of 5 routed bundles unprompted; `api-agent` read 0 of 5 while+making 52 `read_file` calls — the bundles sat in hidden directories its `list_files` skipped, and+the index was prose in a 2.3 kB system prompt. Prose is not an affordance.## 1. The model: progressive disclosure, and why it is also the measurement-The spec's three tiers, and what each costs us per turn:-| tier | what the agent sees | when | our cost ||---|---|---|---|| 1 · index | `name` + `description`, listed by the CLI's own loader | every turn | 41 tokens (our one mandate sentence) |-| 2 · body | the whole `SKILL.md` | on activation | 1,081–2,223 tokens, once |+| 2 · body | the whole `SKILL.md` | on activation | 851–2,498 tokens, once || 3 · depth | `references/*.md` | when the agent chooses to go deeper | as read |-**Since 2026-09-22 the probe reads the CLI's own record first** (§4): every backend folds-its session's tool calls into `transcript.jsonl`, and a bundle is *surfaced* when a call-activated it or named its `SKILL.md`, *deep* when a call named a file under its-`references/` (`SkillsUsage.evidence == "transcript"`). The atime probe below is the-fallback for a session that left no trace — and the rest of this section is why it can only-be a fallback.--The same three tiers are the atime read probe. `materialize_skills` writes every file with-`atime == mtime`; the root filesystem is ext4 with `relatime`, so the first read afterwards-bumps `atime`:--* `atime(SKILL.md) > mtime` → **surfaced**: something opened the bundle.-* `atime(references/*.md) > mtime` → **deep**: something opened the body's depth file.--A bundle with no `references/` reports `deep_measurable: false` rather than quietly-scoring 0. `api-agent` owned its own `read_file`, so it logged exact reads with turn numbers.--### The probe carries its own falsification, because it had to--The design read `deep` as "the agent chose to go deeper". **Measured 2026-08-25, it does-not mean that.** Two independent causes, both reproduced:--1. **`Workspace.changed_files` reads every file.** It runs `git add -A -N` then- `git diff --numstat` after *every* agent session to compute `files_changed`, and git- reads each untracked file to diff it. On a real git workspace that alone flips all four- files of every bundle to "read", with no agent involved. Reproduced in- `test_git_diff_alone_trips_the_control`.-2. **Every CLI opens `references/` while activating a skill.** Negative control: one- bundle whose description does not match the task, prompt "do not read any files, do not- activate any skill" — codex 0.149.0, claude-code 2.1.245, agy 1.1.20 and gemini-cli- 0.53.0 *all* opened `SKILL.md` **and** `references/`. (Each touched only the root it- owns, which is a nice independent confirmation of the root mapping in §7.)--So a fourth signal was added: `materialize.write_control` puts one **never-routed,-never-indexed** bundle beside the real ones. Nothing should ever open it. When it comes-back opened, `SkillsUsage.control_read` is set and-the session's deep-read rate is **`None`** — not 100%, not 0. `3dcode skills report` excludes those-sessions from the rate and prints how many it dropped.--This is the difference between a metric and a number that would have read 100% forever.+The read probe (`skills/telemetry.probe_reads`) reads the CLI's own record first (§4): a bundle+is *surfaced* when a call activated it or named its `SKILL.md`, *deep* when a call named a file+under its `references/` — `SkillsUsage.evidence == "transcript"`. The atime probe is the+fallback for a session that left no trace: `materialize_skills` writes every file with+`atime == mtime` on an ext4 `relatime` root, so the first read afterwards bumps `atime`+(`SKILL.md` → surfaced, `references/*.md` → deep; a bundle with no `references/` reports+`deep_measurable: false` rather than scoring 0).++**Why atime can only be a fallback** — measured 2026-08-25, both reproduced:++1. `Workspace.changed_files` runs `git add -A -N` + `git diff --numstat` after every session,+ and git reads each untracked file to diff it — every bundle flips to "read" with no agent+ involved (`test_git_diff_alone_trips_the_control`).+2. Every CLI opened `SKILL.md` **and** `references/` of a bundle whose description did not match+ the task, under "do not read any files, do not activate any skill" (§7).++So `materialize.write_control` puts one **never-routed, never-indexed** bundle beside the real+ones. Nothing should open it; when it comes back opened, `SkillsUsage.control_read` is set, the+session's deep-read rate is **`None`** — not 100 %, not 0 — and `3dcode skills report` says how+many sessions it dropped. That is the difference between a metric and a number that would have+read 100 % forever. One job stays with atime under transcript evidence: a reference opened by a+call the trace cannot parse (claude-code's `cd … && cat` loop, 2026-11-03) counts as deep while+the control's `references/` are untouched — nothing scanned references/ wholesale (D115).---## 2. The library-**Twenty bundles.** `lines`/`tokens` are the body, against caps of 350 and 2,500.+**Twenty bundles** (2026-11-03). `lines` is `SKILL.md` as shipped, `tokens` the body; the caps+are 350 body lines and 2,500 tokens.| skill | track | language | evidence | lines | tokens | routes ||---|---|---|---|---|---|---|-| `c3d-part-contact` | object tracks | any | measured | 125 | 1,707 | R1, R2 |-| `c3d-bbox-contract` | object + scene | any | measured | 75 | 1,081 | R3, R4 |-| `c3d-repeats-and-mirrors` | object tracks | any | measured | 112 | 1,527 | R6, R7 |-| `c3d-blender-forms` | any | blender, urdf_blender | measured | 97 | 1,811 | R8, R9 |-| `c3d-urdf-joints` | articulated_object | urdf_blender | measured | 139 | 1,989 | R12, R13 |-| `c3d-scene-composition` | scene | scene_threejs | mixed | 135 | 2,119 | R14, R15, R34 |-| `c3d-scene-lighting` | scene | scene_threejs | mixed | 115 | 1,748 | R16, R17 |-| `c3d-scene-motion` | scene | scene_threejs | mixed | 185 | 2,472 | R18, R19 |-| `c3d-threejs-shader-traps` | scene (+ static on a preflight finding) | threejs both | mixed | 112 | 1,734 | R20, R21 |-| `c3d-glsl-craft` | graphics | glsl_shader | mixed | 140 | 1,958 | R22, R24-glsl |-| `c3d-opengl-pipeline` | graphics | opengl_python | mixed | 98 | 1,497 | R23, R24-opengl |-| `c3d-cadquery-forms` | any | cadquery | **inherited-unverified** | 82 | 1,482 | R10 |-| `c3d-threejs-forms` | static_object | threejs | **inherited-unverified** | 120 | 1,929 | R11 |-| `c3d-scene-atmosphere` | scene | scene_threejs | **inherited-unverified** | 76 | 1,030 | R25 |-| `c3d-scene-water` | scene | scene_threejs | **inherited-unverified** | 84 | 1,144 | R26 |-| `c3d-scene-night` | scene | scene_threejs | **inherited-unverified** | 60 | 787 | R27 |-| `c3d-scene-materials` | scene | scene_threejs | **inherited-unverified** | 99 | 1,434 | R28 |-| `c3d-bpy-scene-placement` | scene | scene_blender | mixed | 133 | 2,496 | R29, R30 |-| `c3d-bpy-scene-lighting` | scene | scene_blender | mixed | 124 | 2,356 | R31, R32 |-| `c3d-bpy-scene-materials` | scene | scene_blender | mixed | 113 | 2,075 | R33 |--**All twenty route by default since 2026-09-22** (the three `c3d-bpy-scene-*` bundles since 2026-10-14, D105) (`C3D_SKILLS` and-`C3D_SKILLS_UNVERIFIED` both default on; `C3D_SKILLS=0` turns the system off,-`C3D_SKILLS_UNVERIFIED=0` drops the six `inherited-unverified` bundles). The cap of 5 per-session still holds. Zero have a measured effect — they route because the owner made skills-part of the product and every backend now loads them natively (§6, 2026-09-22), not because-an A/B promoted them. Measuring a subset needs no new machinery: `C3D_SKILLS_ONLY` restricts-the library before routing, measured against a `C3D_SKILLS=0` arm.--### Retired--| skill | cut | why |-|---|---|---|-| `c3d-form-manifest` | 2026-08-25 | opened **2 of 19** times (11%, CI [3%, 31%]) — unread before *and* after its description was rewritten to lead with the trigger. The only bundle whose read-rate interval excluded every other's, and it cost ~1,917 tokens in 19 of the 27 sessions that could have read it. Text and full numbers: git history (`docs/skills-attic/` was removed 2026-09-21) |--An attic bundle is not loaded: `all_skills()` reads `codeverse3d/skills/*/SKILL.md` only, so it-costs no index line and no tokens. Its route id (**R5**) is retired and never reused — a run-record refers to a routing decision by id, so recycling it would silently relabel history.-`tests/skills/test_registry.py::RETIRED_RULES` pins that.--### Two words this doc keeps apart--`evidence` is about **provenance** — where the bundle's prose came from. A bundle can be-`measured` for provenance and have **no measured effect**; five are. Every bundle also-declares **one deterministic quantity it claims to move**, in its own frontmatter-(`target_metric`, `target_direction`, `target_baseline`) and in `codeverse3d/addons/skill_targets.py`.-`python eval/bench/skill_targets.py eval/bench/out` prints them all — per battery or paired across an-A/B's two arms — and the optional local skills ledger records historical measurements.--`cadquery-forms` and `threejs-forms` stay `inherited-unverified` — routed by default since-2026-09-22, but unranked: their languages have **4 (cadquery) and 3 (threejs) graded runs**-against the 20 the rule requires (design §5.2 law 3).+| `c3d-part-contact` | object tracks | any | measured | 138 | 1,728 | R1, R2 |+| `c3d-bbox-contract` | object + scene | any | measured | 90 | 1,108 | R3, R4 |+| `c3d-repeats-and-mirrors` | object tracks | any | measured | 130 | 1,545 | R6, R7 |+| `c3d-blender-forms` | any | blender, urdf_blender | measured | 128 | 2,195 | R8, R9 |+| `c3d-urdf-joints` | articulated_object | urdf_blender | measured | 171 | 2,172 | R12, R13 |+| `c3d-scene-composition` | scene | scene_threejs | mixed | 151 | 2,150 | R14, R15, R34 |+| `c3d-scene-lighting` | scene | scene_threejs | mixed | 130 | 1,748 | R16, R17 |+| `c3d-scene-motion` | scene | scene_threejs | mixed | 200 | 2,473 | R18, R19 |+| `c3d-threejs-shader-traps` | scene (+ static on a preflight finding) | threejs both | mixed | 128 | 1,722 | R20, R21 |+| `c3d-glsl-craft` | graphics | glsl_shader | mixed | 124 | 1,613 | R22, R24-glsl |+| `c3d-opengl-pipeline` | graphics | opengl_python | mixed | 121 | 1,696 | R23, R24-opengl |+| `c3d-cadquery-forms` | any | cadquery | inherited-unverified | 107 | 1,731 | R10 |+| `c3d-threejs-forms` | static_object | threejs | inherited-unverified | 143 | 2,151 | R11 |+| `c3d-scene-atmosphere` | scene | scene_threejs | inherited-unverified | 86 | 1,061 | R25 |+| `c3d-scene-water` | scene | scene_threejs | inherited-unverified | 83 | 1,045 | R26 |+| `c3d-scene-night` | scene | scene_threejs | inherited-unverified | 78 | 851 | R27 |+| `c3d-scene-materials` | scene | scene_threejs | inherited-unverified | 117 | 1,484 | R28 |+| `c3d-bpy-scene-placement` | scene | scene_blender | mixed | 148 | 2,498 | R29, R30 |+| `c3d-bpy-scene-lighting` | scene | scene_blender | mixed | 139 | 2,356 | R31, R32 |+| `c3d-bpy-scene-materials` | scene | scene_blender | mixed | 128 | 2,075 | R33 |++All twenty route by default (the `c3d-bpy-scene-*` three since 2026-10-14, D105), capped at 5 a+session. None has a measured effect: they route because the owner made skills part of the+product and every backend loads them natively, not because an A/B promoted them (§6). To measure+a subset, `C3D_SKILLS_ONLY` restricts the library before routing, against a `C3D_SKILLS=0` arm.++**Retired:** `c3d-form-manifest` (2026-08-25) — opened 2 of 19 times (11 %, CI [3 %, 31 %]), the+one bundle whose read-rate interval excluded every other's, at ~1,917 tokens a listing. Its route+id **R5** is never reused: a run record refers to a routing decision by id+(`tests/skills/test_registry.py::RETIRED_RULES`). The text is in git history.++**Two words kept apart.** `evidence` is provenance — where the prose came from; a `measured`+bundle can have no measured effect, and five do. Every bundle also declares the ONE+deterministic quantity it claims to move (`target_metric` / `target_direction` /+`target_baseline` in its frontmatter and `addons/skill_targets.py`;+`python eval/bench/skill_targets.py eval/bench/out` prints them, per battery or paired across an+A/B). `cadquery-forms` and `threejs-forms` stay `inherited-unverified` — routed, unranked —+until their languages reach the 20 graded runs the rule requires (4 and 3 at the last count).## 3. Routing-`codeverse3d/skills/registry.py` holds the typed table (R1–R28) and `finding_kind()`, which names-a finding `<gate>/<data.kind>` — the kind the gate wrote, never its message; its router half (`select()`) turns-`(track, language, kind, plan signals, findings)` into a ranked, capped, reasoned set.--Four laws, all tested:--1. **Gate findings outrank everything.** A row that answers a finding sits at priority- ≥ 90; standing rows sit at ≤ 80. A repair round spends its budget on what broke. This is- the input no CLI's own skill loader can see, and the reason we route in code.-2. **The cap is 5** (`C3D_SKILLS_MAX`). It cuts the low-priority tail, never the- gate-fired head.-3. **Quiet kinds attach nothing** on their own — `asset`, `asset_fix`, `reference` are- short, narrow sessions with no defect class attached to them. A gate-fired row still- reaches them (R13/R15/R17/R19/R21 are kind-agnostic; R2/R4/R7/R9 are not — both halves- are pinned in `test_router`).-4. **Unknown input degrades to `[]`, never an exception.** A planner failure must not also+`codeverse3d/skills/registry.py` holds the typed table (34 rows; ids R1–R34, R5 retired, R24+split by language) and `finding_kind()`, which names a finding `<gate>/<data.kind>` — the kind+the gate wrote, never its message; `select()` turns `(track, language, kind, plan signals,+findings)` into a ranked, capped, reasoned set. Four laws, all tested:++1. **Gate findings outrank everything.** A row answering a finding sits at priority ≥ 90,+ standing rows at ≤ 80: a repair round spends its budget on what broke — the input no CLI's+ own loader can see, and the reason routing is in code.+2. **The cap is 5** (`C3D_SKILLS_MAX`); it cuts the low-priority tail, never the gate-fired head.+3. **Quiet kinds attach nothing on their own** — `asset`, `asset_fix`, `reference` are short,+ narrow sessions. A gate-fired row still reaches them (R13/R15/R17/R19/R21 are+ kind-agnostic, R2/R4/R7/R9 are not; both halves pinned in `test_router`).+4. **Unknown input degrades to `[]`, never an exception** — a planner failure must not alsotake out the round's skills.-`test_routing_property.py` runs every gate finding kind the corpus produces from every-track × language × round kind over the real library (cap, priority ≥ 90, a reason that names-the finding), and checks that every signal a route requires is one `plan_signals()` sets. The-cap and the order themselves come from `select`'s final sort and slice, pinned by row in-`test_router` (the exhaustive plan-signal sweep went in the D96 second pass, 2026-09-23).--`finding_kind()` reads no message text (audit 2026-09-24, N52): the message regex it replaced-left seven `scene_frames` kinds unrouted and R19 unreachable, and any gate reword could unroute-a skill. Every gate writes a `data.kind` — the lints from one vocabulary-(`contracts.artifacts.LintKind`), connectivity `stray_islands` / `no_ground`, motion_direction-`wrong_direction` / `skipped`, render_console `console_error` (2026-09-24), scene_probe `draw_calls` (D99) — so R9-answers `lint/part_not_imported` and R13 `motion_direction/wrong_direction`, not every lint or-motion line. A finding recorded before its gate wrote one is `<gate>/untyped`, which only a-family row answers; the lint gates are one `lint` family. INFO findings are census, not defects, and have no kind — routing off-"all 7 parts are connected" would attach the penetration sheet to a run with none.--Adding a bundle means adding its rows in the same commit: an unroutable bundle pays the-index and never helps, and a test fails on both halves of that.+`test_routing_property.py` runs every finding kind the corpus produces from every track ×+language × round kind over the real library (cap, priority ≥ 90, a reason naming the finding)+and checks that every signal a route requires is one `plan_signals()` sets. `finding_kind()`+reads no message text (N52, 2026-09-24: the regex it replaced left seven `scene_frames` kinds+unrouted and R19 unreachable). Every gate writes a `data.kind` — lints from+`contracts.artifacts.LintKind`, connectivity `stray_islands` / `no_ground`, motion_direction+`wrong_direction` / `skipped`, render_console `console_error`, scene_probe `draw_calls` — so R9+answers `lint/part_not_imported`, not every lint line; a finding from before its gate wrote a+kind is `<gate>/untyped`, answered only by a family row; INFO findings have no kind (census, not+defects). A bundle is added with its rows in the same commit — an unroutable bundle pays the+index and never helps, and a test fails on both halves.---## 4. Telemetry-Per round, `RoundRecord.skills` carries a `SkillsUsage`: what was listed, what was-surfaced, what was read deep, `index_tokens`, `body_tokens_read`, and `evidence` — where the-reads came from. Each attached body's hash goes into `record.prompt_hashes` as-`skill:<name>`, so "what text decided this run" stays answerable. Per session,-`telemetry/skills.jsonl` gets one line.--### Ground truth: the CLI's own record (2026-09-22)+Per round, `RoundRecord.skills` is a `SkillsUsage`: listed, surfaced, read deep, `index_tokens`,+`body_tokens_read`, `evidence` and the control's state. Each attached body's hash goes into+`record.prompt_hashes` as `skill:<name>`; per session, `telemetry/skills.jsonl` gets one line.-Each backend now writes what its CLI says it called into `trajectories/<label>/transcript.jsonl`-— one `tool_call` row per call (arguments compacted to 400 characters a value; the CLI's own-failure verdict as `failed`), then one `tool_trace` row that closes the session-(`agents/cli_common.record_tool_calls`). Where each one reads it:+**Ground truth: the CLI's own record** (2026-09-22). Each backend writes what its CLI says it+called into `trajectories/<label>/transcript.jsonl` — one `tool_call` row per call (arguments cut+to 400 characters a value, the CLI's own failure verdict as `failed`), then one `tool_trace` row+closing the session (`agents/cli_common.record_tool_calls`):-| backend | source | skill activation looks like |+| backend | source | activation looks like ||---|---|---|-| claude-code | `--output-format stream-json --verbose` (was `json`, which carried no tool stream); the `init` event's `skills` list is kept as `skills_index` | `Skill {"skill": name}` |-| gemini-cli (deleted in D103) | its chat record, `~/.gemini/tmp/<project>/chats/session-*.jsonl`, found by the project's `.project_root` + mtime — so a session the watchdog killed before it printed JSON still has one (and, since 2026-09-22, its usage: `read_gemini_chats` reads calls, tokens and times in one pass). `--output-format json` stays: it has per-tool COUNTS only, and `stream-json` drops the `thoughts` tokens pricing needs | `activate_skill {"name": name}` |-| codex | the `exec --json` events it already streamed (`command_execution`, `mcp_tool_call`, `file_change`) | a `sed`/`cat` of `…/<name>/SKILL.md` |-| agy | its conversation database, `<HOME>/.gemini/antigravity-cli/conversations/<conversation_id>.db` (HOME = the API-mode session's own, read before it is deleted; else `~`) — a schema-free walk of each step's protobuf for (call id, tool, JSON args) | `view_file` of `…/<name>/SKILL.md` |+| claude-code | `--output-format stream-json --verbose` (plain `json` carried no tool stream); the `init` event's `skills` list is kept as `skills_index` | `Skill {"skill": name}` |+| codex | the `exec --json` events it streams (`command_execution`, `mcp_tool_call`, `file_change`) | a `sed`/`cat` of `…/<name>/SKILL.md` |+| agy | its conversation database, `<HOME>/.gemini/antigravity-cli/conversations/<id>.db`, read before the session's HOME is deleted — a schema-free walk of each step's protobuf for (call id, tool, JSON args) | `read_file` of `…/<name>/SKILL.md` |-`skills/telemetry.probe_reads` credits only sessions logged after the attach-(`SkillsMaterialized.attached_at`), and uses the transcript only when **every** such session-closed a trace — a partial trace under-counts exactly as the blind probe over-counts. Under-transcript evidence `control_read` means the AGENT opened the control: counted by-`3dcode skills report` (and flagged), never dropped as blind.+`probe_reads` credits only sessions logged after the attach (`SkillsMaterialized.attached_at`)+and uses the transcript only when **every** such session closed a trace — a partial trace+under-counts exactly as the blind probe over-counts. Under transcript evidence `control_read`+means the AGENT opened the control: counted by the report and flagged, never dropped as blind.```-3dcode skills report eval/bench/out/<battery> # deep-read rate per skill per backend-3dcode skills report eval/bench/out/<battery> --json-3dcode doctor --skills # library + discovery wiring+3dcode skills report eval/bench/out/<battery> # deep-read rate per skill per backend (--json)+3dcode doctor --skills # library + discovery wiring```-Targets: ≥ 60% deep-read for CLI backends (the api-agent's 80% target died with it, 2026-08-28), counted only over sessions-the control (§1) says were measurable. **A skill under 20% over 20 such sessions is merged-or deleted.** A library that only ever grows is how this ends as bloat.-The read rate is the FIRST readout, before any score: `read_cookbook` was called by 0 of 16-zone sessions on `scenes_v1_flash` although the prompt named five chapters by title, and-shipping a nicer file format without measuring reads would repeat that at a new price.--### Measured, on an ordinary build task — 2026-09-22, from the transcripts+Target: ≥ 60 % deep-read, counted only over sessions the control says were measurable. **A skill+under 20 % over 20 such sessions is merged or deleted** — a library that only grows ends as bloat.+The read rate is the first readout, before any score: the former `read_cookbook` tool was called+by 0 of 16 zone sessions although the prompt named five chapters by title.-A harness-built static_object/blender workspace (four routed bundles + the control), each-CLI launched by its own backend's `run()`, a chair brief that never mentions skills:+**Measured 2026-09-22, from the transcripts** — a harness-built static_object/blender workspace,+four routed bundles + the control, each CLI launched by its own backend, a chair brief that never+mentions skills:-| CLI · model | index the CLI showed the model | activated (surfaced) | deep | control | atime probe on the same session |+| CLI · model | index the CLI showed | activated | deep | control | atime probe, same session ||---|---|---|---|---|---|-| gemini-cli 0.53.0 · gemini-3.7-flash (×2) | `gemini skills list`: the 4 + control | **4 / 4** in both sessions, all in the first turn (`activate_skill`); both later killed by a provider storm | 0 | untouched | 4/4 deep, control "read" → blind |-| claude-code 2.1.280 · sonnet | `init.skills`: the 4 + control (+ 26 bundled / user-level) | **4 / 4** (`Skill`) | 0 | untouched | 4/4 deep, control "read" → blind |-| codex 0.155.1 · gpt-5.6-luna | `## Skills`: the 4 + control (+ 5 `.system`) | **4 / 4** (`sed` of `SKILL.md`) | 0 | untouched | 4/4 deep, control "read" → blind |-| agy 1.2.2 · gemini-3.7-flash-medium | its first four calls were the four `SKILL.md` paths | **4 / 4** (`view_file`) | 0 | untouched | 4/4 deep, control "read" → blind |--Every CLI read every routed body and no reference file; the atime probe on the same four-sessions reported every reference "read" and the control opened — git, not the agent.-Details, costs and commands: the local skills ledger (not distributed) §0b.--### Measured 2026-08-25, on the atime probe--Not a compliance prompt — "write `src/model.py` and `src/parts/*.py` for a four-legged-dining chair, follow whatever conventions this workspace documents", five routed bundles-sitting in the workspace, nothing in the prompt about skills (2026-08-25):+| claude-code 2.1.280 · sonnet | `init.skills`: the 4 + control (+ 26 of its own) | **4 / 4** (`Skill`) | 0 | untouched | 4/4 deep, control "read" → blind |+| codex 0.155.1 · gpt-5.6-luna | `## Skills`: the 4 + control (+ 5 `.system`) | **4 / 4** (`sed`) | 0 | untouched | same |+| agy 1.2.2 · gemini-3.7-flash-medium | its first four calls were the four `SKILL.md` paths | **4 / 4** (`view_file`) | 0 | untouched | same |-| backend | listed | surfaced | deep | deep rate | body tokens read | exit |-|---|---|---|---|---|---|---|-| codex 0.149.0 | 5 | 5 | 5 | **100%** | 8,041 | completed, 18 turns |-| claude-code 2.1.245 | 5 | 5 | 5 | **100%** | 8,041 | timed out on our 600 s cap after reading |-| agy 1.1.20 | 5 | 5 | 5 | **100%** | 8,041 | completed |--**Read those 100%s as an upper bound.** These runs predate the control bundle, and §1 says-why every one of them would report 100% whether or not the agent chose to read anything.--What is *not* ambiguous is the behavioural evidence underneath: codex's-`src/parts/common.py` came back with `WELD_OVERLAP = 0.001` and the comment-"1 mm weld overlap with the seat" — `c3d-part-contact`'s recommendation, inside-`PENETRATION_WARN_M`. Left to the prompt corpus alone it would have read 2–5 mm (§9). The-content reached the model and changed the output; the atime number is what cannot prove it.--Two things this also says, and neither is comfortable:--* **They read ALL five.** 8,041 tokens per session, every time. The cap of 5 is therefore- the real cost knob, not a safety net, and router *precision* — how often a skill is read- whose defect class never fires — is the number that should size it (§9).-* **`api-agent` is not in this table.** It needs the Gemini pool, which was in an outage- while this was measured, so the backend with the strictest target (80%) and the only- exact-read ground truth is the one still unmeasured.+Every CLI read every routed body and no reference; the atime probe called every reference "read"+and the control opened — git, not the agent. The 2026-08-25 atime figures (5 of 5 deep on every+CLI) are therefore an upper bound. What is not ambiguous is the behavioural trace: codex's+`src/parts/common.py` came back with `WELD_OVERLAP = 0.001` and "1 mm weld overlap with the seat"+— `c3d-part-contact`'s number, where the prompt corpus of the day said 2–5 mm.---## 5. What the tests guarantee-`tests/skills/` keeps the default checks hermetic. The two `slow`-marked cases-build a wheel (still selected by default; use `-m "not slow"` to omit them), and-the `live` cases drive a real CLI.+`tests/skills/` is hermetic by default; two `slow` cases build a wheel, the `live` cases drive a+real CLI.| file | guarantees ||---|---|| `test_loader.py` | the spec's field rules, as the loader enforces them |-| `test_spec_compliance.py` | the same rules re-derived from the raw bytes, **plus the reference validator** (`pip install skills-ref` → the `agentskills` CLI) and a field-by-field agreement check between its `read-properties` and our loader |-| `test_library.py` | body budget, no code fence over 12 lines in SKILL.md or a reference (D107), no restating `conventions.py`, every `_claims` number still matches its live constant, **no two skills point one claim key at different numbers** |-| `test_freshness.py` | every tool, gate kind, rubric criterion, constant, switch and sibling skill a bundle names still exists, every function it tells the agent to call is defined in the harness's python, and no bundle points at a cookbook |+| `test_spec_compliance.py` | the same rules re-derived from the raw bytes, plus the reference validator (`pip install skills-ref` → the `agentskills` CLI) agreeing field by field with our loader |+| `test_library.py` | body budget, no code fence over 12 lines (D107), no restating `conventions.py`, every `_claims` number matches its live constant, no two skills point one claim key at different numbers |+| `test_freshness.py` | every tool, gate kind, rubric criterion, constant, switch and sibling skill a bundle names still exists; every function it tells the agent to call is defined; no bundle points at a cookbook || `test_router.py` / `test_routing_property.py` | the four routing laws by row; every corpus finding kind from every session |-| `test_telemetry.py` | the read probe, **including the control that catches git reading the tree** |-| `test_packaging.py` | **a built wheel contains all 17 `SKILL.md`, all 17 `references/`, every `_claims` file** |-| `test_live_discovery.py` | §7 — a real CLI actually finds and opens a bundle |--Two contradiction checks are worth separating, because they answer different questions:--* **value-level**, library-wide: two skills may not point one claim key at different- numbers. Compared on the *pre-scale value*, not the rendered text — `c3d-bbox-contract`- says "1 cm" and `c3d-repeats-and-mirrors` says "0.01" metres, and both are right.- Failing that pair would teach the next author to delete the claim.-* **text-level**, co-routing only: two skills that can land in one session and chose the- same units must read the same.--### What these tests caught--* **Three `contract.md` files told the agent to interpenetrate.** `blender` said "2–4 mm- overlap is fine", `cadquery` "overlap (≥ 2 mm) … is required", and `threejs`'s worked- example drove a leg 4 mm into a seat — while `connectivity.py` WARNs above- `PENETRATION_WARN_M = 2 mm` and its own `fix_hint` says "overlap by ≤ 2 mm". Fixed, and- the check is now standing.-* **Four bundles' `verified:` dates were YAML dates, not strings** — the spec says metadata- is string→string. Quoted; `validate_bundle` now reports it instead of coercing.-* **`c3d-opengl-pipeline` claimed `measured` on n=5.** Now `mixed`.-* **The api-agent index cost 780 tokens for five skills** against a 300-token budget,- because it quoted whole 1024-char descriptions. It quotes the first clause now (356- tokens for the worst real session, 400 is the ceiling): our router already decided, so- that index is a pointer, not a matcher.-* **The live smoke had never run.** It skipped every CLI because it looked for a binary- named after the agent kind — `gemini-cli` runs `gemini`, `claude-code` runs `claude`.-* **A body could name one of our constants without pinning it.** `c3d-glsl-craft` quoted- `DUPLICATE_DIFF` at `1e-4` with no claim row behind it, so moving the constant would have- left a confident sentence with the old number and nothing to notice.-* **The read probe was measuring git**, not the agent (§1). The most consequential finding- of the pass, and the reason the control bundle exists.-------## 6. Does it help? The A/B--### 2026-09-22: default ON--**The owner's call, and the evidence behind it.** Everything below this note is the-2026-08-25/26 history, kept as written. What changed since it was written:--1. **The reason for OFF is gone.** Reason 1 of the verdict below is `api-agent`'s 0 % read- rate — and `api-agent` was deleted 2026-08-28. Every backend left is a vendor CLI with a- native skill loader, and the ground-truth channel this section wished it had (§9, "CLI- backends' read rate is UNMEASURED") now exists: each backend logs its CLI's own tool calls.-2. **All four CLIs load the bundles, unprompted, and it is proven from their transcripts**- (§4 table; the local skills ledger (not distributed) §0b): gemini-cli, claude-code, codex and agy each- activated **4 of 4** routed bundles on an ordinary chair brief, none touched the control,- none read a reference file. The discovery wiring was re-read from the installed binaries- and two gaps were closed: gemini-cli's `skills.enabled` is now pinned in the per-session- system settings (a user's `false` would have hidden every bundle), and claude-code streams- (`stream-json`) so its `Skill` calls are on record.-3. **An effect is still unmeasured, and that is now the A/B's job, not the default's.** The- shipping bar in reason 2 cannot be cleared by anything at n = 8 (its own A/A fails it);- skills are part of the product. An A/B of skills now names its no-skills arm- explicitly: `eval/bench/ab_plan.py --variant-env C3D_SKILLS=0` — ab_plan's control is the- driver's env minus the variant's keys, so `--variant-env C3D_SKILLS=1` is now two- identical arms, and the OFF arm has to be the variant (its delta reads OFF − ON).--What default ON costs: the mandate sentence (41 tokens) in every session's body file, plus-the bodies the CLI chooses to activate — measured 2026-09-22 at all four of four, i.e. up-to the cap of five bodies (1,081–2,223 tokens each) per session. `C3D_SKILLS_MAX` is the-cost knob; `C3D_SKILLS=0` the off switch.+| `test_telemetry.py` | the read probe, the control that catches git, the atime complement under the control |+| `test_packaging.py` | a built wheel contains all 20 `SKILL.md`, their `references/`, every `_claims` file |+| `test_live_discovery.py` | §7 — a real CLI finds and opens a bundle |-**Verdict (2026-08-25/26, superseded 2026-09-22): ship OFF behind `C3D_SKILLS`.**+Two contradiction checks answer different questions: **value-level**, library-wide — two skills+may not point one claim key at different numbers, compared on the pre-scale value ("1 cm" and+"0.01" metres are both right); **text-level**, co-routing only — two skills that can land in one+session and chose the same units must read the same.-### The rig+----`eval/bench/ab_plan.py`, arms differing in exactly one thing — `C3D_SKILLS=on` on the variant —-on `static_objects_v2`, `--rounds 1`, paired, generator `gemini-cli:gemini-3.6-flash`,-fixed judge `gemini:gemini-3.1-pro-preview` at `n_samples 2`, `--max-in-flight 8` against-32–48 of pool headroom, `--wait-for-provider`. The eight prompt ids are **the same eight-`eval/bench/out/plan_loop/C0` used for its A/A**, so the noise floor below was measured on this-exact battery, on this rig, on this day.+## 6. Does it help? The A/B-The switch is generation-side: the planner call happens in the `plan` stage before any-skill is attached, and `attach_for_round` only touches generation and repair rounds. That-matters for §8.1 of `eval/docs/EVAL.md` and is the reason the next run of this can pin the plan.+**Default ON since 2026-09-22** (owner's call, D81). The 2026-08-25 verdict had been OFF, for+one reason that is gone — `api-agent`, then the bench's generator, read 0 of 5 bundles with+ground truth (52 `read_file` calls, none on a skill path) and was deleted 2026-08-28 — and one+that is now the A/B's job: every CLI backend activates 4 of 4 routed bundles unprompted (§4),+and an *effect* cannot be shown at n = 8 on this rig at all (below). What ON costs: the+41-token mandate sentence plus the bodies the CLI activates — all of them, up to the cap of five+at 851–2,498 tokens each. An A/B names its no-skills arm explicitly:+`eval/bench/ab_plan.py --variant-env C3D_SKILLS=0` (ab_plan's control is the driver's env minus+the variant's keys, so the OFF arm must be the variant; its delta reads OFF − ON).-### The noise floor, measured first+**The rig** (2026-08-25): `ab_plan.py`, arms differing in `C3D_SKILLS` only, `static_objects_v2`,+`--rounds 1`, paired, judge `gemini:gemini-3.1-pro-preview` at `n_samples 2`; the eight prompts+are the ones `eval/bench/out/plan_loop/C0` used for its A/A. The switch is generation-side (the+planner runs in the `plan` stage before anything is attached), so plan-once-write-both is valid+for it (`eval/docs/EVAL.md` §8.1).-`eval/bench/out/plan_loop/C0`, 2026-08-25, **two identical arms** on these eight prompts:+**The noise floor, measured first** — C0, two identical arms, the same eight prompts:| | ||---|---|-| paired prompts scored | 6 of 8 (2 lost to `infra_failed`) |-| mean delta | **+0.043** — from nothing |-| paired sd | 0.202 · SE 0.082 · 2 SE band **±0.165** |-| separated from noise | **NO** |+| paired prompts scored | 6 of 8 (2 `infra_failed`) |+| mean delta | **+0.043** from nothing; paired sd 0.202, SE 0.082, 2 SE band **±0.165** || sign test | 2 up / 3 down, p = 1.000 || worst "regression" | `mech_hard_pitcher_pump` **−0.206** |-| prompts needed to resolve ±0.02 | ~408 |--Read the last two rows against the shipping bar this wave was given — *mean delta ≥ 0 and-no prompt regressed by more than 0.03*. **The A/A fails that bar.** Two of its six paired-prompts regressed past 0.03 with byte-identical arms. So at n = 8 on this rig the bar is-not a test of the change; nothing can pass it. That is a fact about the instrument, and it-is the honest first line of any verdict it produces.--`eval/docs/EVAL.md` §8.1 opened C0's worst pair and found the cause: the two identical arms-planned 1 part and 10 parts for the same pitcher pump. The dominant variance term is the-**planner**, not the judge and not the generator.--### The first attempt measured nothing, and said nothing about it--Recorded here because it is the more useful half of this section. The first run of this-A/B was **void**: `ab_plan.spawn_cell` starts each child as a file path, so `sys.path[0]`-is `eval/bench/`, and `ab_plan` imported `codeverse3d._compat` *above* its own `sys.path`-bootstrap — which let the editable install resolve `codeverse3d` to the **main tree**. Both-arms ran a harness with no `codeverse3d/skills` package at all. No error, no warning; the-variant workspace simply had no `.agents/skills` directory and no `skills.attached` event,-and the run would have reported "no effect" with a straight face.--The plan-loop wave hit the same thing hours earlier-(`eval/bench/out/plan_loop/C0/invalid_attempt1_maintree_import`) and worked around it with-`PYTHONPATH` in a launch script. It is fixed in the code now — bootstrap first, a guard-that refuses to start against a foreign `codeverse3d`, and-`eval/tests/test_worktree_import.py` — because a workaround protects whoever-remembers it, not the run.--### What this A/B actually measured--`eval/bench/out/ab_skills`, **still running when this was written** — re-read `summary.md` for the-current state, and re-run with `--redo-status infra_failed` before quoting it as final.-Weather was hostile: the Gemini pool sat at 0–4 of 6 keys answering for most of the window,-so the rig parked on its preflight for two hours and then lost whole pairs to 503 storms-mid-cell.--| | |-|---|---|-| paired prompts scored | **1** of 8 (`ctrl_med_dining_chair`) |-| mean delta | **+0.002** (control 0.937, variant 0.939) |-| 95% CI | **not computable at n = 1**; the A/A band on this battery is ±0.165 |-| sign test | 1 up / 0 down, p = 1.000 |-| `infra_failed` cells | **3+** (both arms of `ctrl_med_toaster`, control of `mech_hard_coffee_grinder`), excluded, never scored 0 |-| verdict printed by the rig | `inconclusive`, `separated from noise: NO` |--**And the reason that delta is +0.002 is not that the skills did not help. It is that the-variant arm never read them.**--### The read rate, with ground truth: 0 of 5--`api-agent` is the calibration arm — we own its `read_file` tool, so-`telemetry/skill_reads.jsonl` is not a probe, it is a log. On the scored variant cell:--| | |-|---|---|-| skills listed in message 0, marked MANDATORY | **5** |-| index cost | 356 tokens |-| `read_file` calls the agent made, across 4 sessions | **52** |-| of those, on a skill path | **0** |-| `read_cookbook` calls instead | 2 |-| what the atime probe reported | 5 of 5 "deep" |-| what the control reported | `control_read: true` — **the probe was blind**, exactly as §1 predicts |--The next variant cell (`mech_hard_coffee_grinder`) repeated it: five bundles materialised,-`read_file` called, **zero** skill paths.--So on the backend the entire bench runs on, the mechanism is inert: the agent read 52 files-and not one of them was a skill. The two arms differed by 356 tokens of index that nobody-opened, and +0.002 is what that is worth.--This is not the same answer for every backend. The three subscription CLIs read all five-unprompted (§4) and codex's output carried the skill's number. The difference is the-affordance: a CLI has a first-class skill tool its runtime surfaces, and `api-agent` has a-line in a long markdown file plus a generic `read_file`.--### Verdict (2026-08-25/26 — superseded by the 2026-09-22 note above)--**Ship OFF behind `C3D_SKILLS`**, and not because the number was negative:--1. On `api-agent`, the read rate is **0 %** with ground truth. Shipping a default-on feature- that its main generator does not use would pay 356 tokens a turn for nothing.-2. The shipping bar it was given — mean delta ≥ 0 and no prompt regressed by more than 0.03- — **cannot be cleared by anything on this rig at n = 8**: the A/A of two identical arms- fails it, twice over (−0.038, −0.206).-3. The deterministic readouts are not an escape hatch either (below).-4. Two of the original thirteen bundles have no graded runs behind them at all and are already routed off; the four 2026-09-01 scene-graphics ports are `inherited-unverified` by construction until the sceneloop A/B grades them.--Nothing here says the library is wrong. The CLI evidence says the opposite. It says the-delivery mechanism for `api-agent` is a pointer nobody follows, and that is fixable.---### The deterministic readouts are not the escape hatch either--The design planned around this: make the **gate counts** the primary readout, since they-are deterministic and are what a contact skill actually targets, and put the judged score-second under a sign test. `eval/bench/ab_gate_rates.py` computes them, paired per prompt.--Run it over the SAME A/A — two identical arms — and it says:--| readout (variant − control, identical arms) | mean delta | better / worse / tied |-|---|--:|---|-| penetrating pairs | **+5.67** | 0 / 2 / 1 |-| worst penetration depth (mm) | **+2.80** | 0 / 3 / 0 |-| floating parts | +0.00 | 0 / 0 / 3 |-| stray islands | −0.33 | 1 / 0 / 2 |-| contract findings | −1.00 | 2 / 1 / 0 |--`mech_hard_pitcher_pump` alone went **1 penetrating pair → 17** between two identical arms.-Of course it did: the same §8.1 planner variance that produced a 1-part plan and a 10-part-plan produces 1 contact pair and 17. The gate counts inherit the planner's spread *directly*-— a bigger plan has more parts, more parts have more contacts. **They are not the low-noise-readout the design hoped for**, and this A/A calibration of them did not exist before today.--So at n = 8 with a free plan, neither readout can separate this switch from nothing.--### What to try next, in order--0. **Pin the plan** (`eval/docs/EVAL.md` §8.1). The skills switch is generation-side — the- planner runs in the `plan` stage before anything is attached — so plan-once-write-both is- valid here, and it removes the dominant variance term from *both* readouts instead of- averaging it down. It also costs one planner call *less* per pair. This is the single- highest-value change and nothing else is worth running before it.-1. **Reconcile the prompt corpus on weld overlap** (§9). `contract.md` is fixed, but- `tracks/generate_static.j2` still says "overlap neighbours by ≥ 0.002 m (push a leg- 2-5 mm into the seat)". While that stands, `c3d-part-contact` is arguing with the- prompt inside the same session, and the pair-count readout is measuring the argument.-2. Only then re-run this A/B, with the gate counts as the primary readout.+| prompts to resolve ±0.02 | ~408 |++The shipping bar the wave was given — mean delta ≥ 0 and no prompt regressed by more than 0.03 —+is failed by the A/A itself, so at n = 8 it tests the instrument, not the change. C0's worst+pair planned 1 part and 10 parts for the same pitcher pump: the dominant variance term is the+**planner**, and the deterministic readouts inherit it directly — on the same A/A,+`eval/bench/ab_gate_rates.py` saw penetrating pairs go **1 → 17** between identical arms+(mean +5.67 pairs, +2.80 mm worst depth): a bigger plan has more parts and more contacts.++**What the A/B measured** (`eval/bench/out/ab_skills`): one paired prompt scored out of eight in+a 503 storm, delta **+0.002** (0.937 → 0.939), `inconclusive` — and the variant arm had not read+the bundles (the 0 of 5 above). Its first attempt was **void** with no symptom: `spawn_cell`+started each child as a file path, `ab_plan` imported `codeverse3d._compat` above its `sys.path`+bootstrap, and both arms ran the main tree with no `skills` package at all; fixed with a+bootstrap-first import, a guard that refuses a foreign `codeverse3d`, and+`eval/tests/test_worktree_import.py`.++**Next, in order:** pin the plan (removes the dominant term from both readouts and costs one+planner call less per pair); re-run on a post-D107 tree (the corpus no longer argues with+`c3d-part-contact` about weld overlap) with gate counts as the primary readout, as hit rates (§9).---## 7. Native loading — proven, not argued-`pytest -m live tests/skills/test_live_discovery.py`. A probe bundle is materialised into a-scratch workspace, the CLI is given a task matching its description, and the assertion is-the read probe itself plus a magic word that only lives in `references/`.+`pytest -m live tests/skills/test_live_discovery.py`: a probe bundle is materialised into a+scratch workspace, the CLI gets a task matching its description, and the assertion is the read+probe plus a magic word that only lives in `references/`.-| CLI | version | discovery root | surfaced | deep | followed |+| CLI | first proven | re-run 2026-11-03 | discovery root | surfaced | deep ||---|---|---|---|---|---|-| gemini-cli | 0.53.0 | `.agents/skills/` | yes | yes | yes |-| codex | 0.149.0 | `.agents/skills/` | yes | yes | yes |-| claude-code | 2.1.245 | `.claude/skills/` | yes | yes | yes |-| agy (antigravity) | 1.1.20 | `.agents/skills/` | yes | yes | yes |--Re-checked 2026-09-22 on gemini-cli 0.53.0, claude-code 2.1.280, codex 0.155.1 and agy-1.2.2, through the harness's own launch path and read from each CLI's own transcript (§4):-same roots, all four activate the routed bundles. Two more things that could only be-settled against the binaries: gemini-cli skips BOTH workspace roots in an untrusted folder-(`SkillManager.discoverSkills`), which the harness's `folderTrust.enabled=false` already-avoids; and agy's `--disable-slash-commands`, which the harness passes, only stops `/name`-expansion of the prompt — its model still reads the bundles.--Re-checked 2026-11-03 on claude-code 2.1.317, codex 0.166.0 and agy 1.4.12 (`pytest -m live-tests/skills/test_live_discovery.py`): same roots, all three surface and read deep. claude-code-now reads `references/` with a shell loop (`cd …/<name> && for f in $(find references -type f);-do cat "$f"; done`) that names no path the trace can match; the probe credits such a read from-the file's atime while the control's `references/` stay untouched (`telemetry.probe_reads`, D115).--All four discover the library natively, with no injection from us. Two things this settled-that had only been argued statically:--* `Skill` was **absent** from `ALLOWED_TOOLS` in `agents/backends.py`, so claude-code- would have denied its own skill tool. Added; `3dcode doctor --skills` now checks it.-* gemini-cli's `activate_skill` consent prompt does not block us under- `--approval-mode yolo`. This was the one claim that could not be verified statically.--**And the same run refutes the surfaced-vs-deep split.** Negative control, same rig: a-bundle whose description does *not* match the task, prompt "do not read any files, do not-activate any skill". All four CLIs opened `SKILL.md` **and** `references/`:--| CLI | `.agents/` SKILL.md | `.agents/` references | `.claude/` SKILL.md | `.claude/` references |-|---|---|---|---|---|-| codex | opened | **opened** | — | — |-| agy | opened | **opened** | — | — |-| gemini-cli | opened | **opened** | — | — |-| claude-code | opened | **opened** | opened | **opened** |--Each touched only the root it owns (claude-code reads both), which is a second, independent-confirmation of the mapping above — and the reason `deep` had to stop meaning "the agent-chose to go deeper" (§1).+| claude-code | 2.1.245 | 2.1.317 | `.claude/skills/` | yes | yes — via `cd …/<name> && … cat "$f"` (D115) |+| codex | 0.149.0 | 0.166.0 | `.agents/skills/` | yes | yes |+| agy | 1.1.20 | 1.4.12 | `.agents/skills/` | yes | yes |++Re-read from the binaries 2026-09-22 (claude-code 2.1.280, codex 0.155.1, agy 1.2.2) through the+harness's own launch path: same roots, all activate the routed bundles; agy's+`--disable-slash-commands`, which the harness passes, only stops `/name` expansion of the prompt.+Two things this settled that had only been argued: `Skill` was absent from `ALLOWED_TOOLS`+(claude-code would have denied its own skill tool — added, and `3dcode doctor --skills` checks+it); and the negative control — a bundle whose description does not match, "do not read any+files, do not activate any skill" — found every CLI opening `SKILL.md` **and** `references/`,+each only under the root it owns (claude-code both roots): the independent confirmation of the+root mapping, and the reason `deep` cannot mean "the agent chose to go deeper" on atime alone (§1).---## 8. How to add a skill-1. `codeverse3d/skills/<name>/SKILL.md` — `name` equals the directory name, lowercase- `a-z0-9` and single hyphens; `description` (≤ 1024 chars) says WHAT and WHEN, because it- is the only text a CLI matches on; **no `<` or `>` anywhere in the frontmatter** (the- spec's prompt-injection rule); only spec keys at the top level, everything of ours under- `metadata`; `metadata.evidence` and `metadata.verified` (a *quoted* ISO date) required.-2. `codeverse3d/skills/<name>/references/*.md` — **required**, one level. Without it the- bundle can never score on the read metric.-3. `codeverse3d/skills/_claims/<name>.toml` — one row per number the body quotes from live- code, so moving the constant breaks the test that ships the sentence:+1. `codeverse3d/skills/<name>/SKILL.md` — `name` equals the directory name (lowercase `a-z0-9`,+ single hyphens); `description` (≤ 1024 chars) says WHAT and WHEN — the only text a CLI matches+ on; no `<` or `>` in the frontmatter (the spec's prompt-injection rule); spec keys only at the+ top level, everything of ours under `metadata`, where `evidence` and `verified` (a *quoted*+ ISO date) are required.+2. `codeverse3d/skills/<name>/references/*.md` — required, one level: without it the bundle can+ never score on the read metric.+3. `codeverse3d/skills/_claims/<name>.toml` — one row per number the body quotes from live code,+ so moving the constant breaks the test that ships the sentence:```toml[[claim]]@@ -637,105 +344,51 @@ chose to go deeper" (§1).format = "{:.0f} mm"```A row may also pin a sentence in a bundle file (`file = "references/x.md"`), a JS constant- (`js = "runtime_js/lib/backdrop.mjs:GROUND_SPAN_M"`, a numeric `export const`), a whole- sequence (`sep = ", "`, optional `last_sep = " and "`), and the line it must stand on- (`context = "WARN below"`); the text counts only as its own token. The prompt corpus pins- its numbers the same way in `codeverse3d/prompts/_claims.toml` (`file` relative to prompts/;- `check_prompt_claims`, also run by `3dcode skills validate`).-4. Add its rows to `ROUTES` in `codeverse3d/skills/registry.py`, in the same commit.-5. Add a `Target` row to `codeverse3d/addons/skill_targets.py` and the matching `target_*` keys to- the frontmatter — the ONE deterministic quantity the bundle claims to move, its- direction, and its baseline from `python eval/bench/skill_targets.py eval/bench/out`. If no- deterministic instrument can see the claim, say so with `measurable=False` and a- `caveat`: that is a finding about the bundle, not a gap to paper over with a judged- criterion. Maintainers may also record the measurement in their local skills ledger.+ (`js = "runtime_js/lib/backdrop.mjs:GROUND_SPAN_M"`), a sequence (`sep`, `last_sep`) and the+ line it must stand on (`context = "WARN below"`). The prompt corpus pins its numbers the same+ way in `codeverse3d/prompts/_claims.toml` (`check_prompt_claims`, run by+ `3dcode skills validate`).+4. Its rows in `ROUTES` (`codeverse3d/skills/registry.py`), in the same commit.+5. A `Target` row in `codeverse3d/addons/skill_targets.py` and the matching `target_*` frontmatter+ keys — the ONE deterministic quantity the bundle claims to move, its direction, and its+ baseline from `python eval/bench/skill_targets.py eval/bench/out`; if no deterministic+ instrument can see the claim, say so with `measurable=False` and a `caveat`.6. `3dcode skills validate --strict` and `pytest tests/skills`.Body rules: ≤ 350 lines and ≤ 2,500 tokens; no code fence over 12 lines (D107); never restate a-frame, unit or naming rule (`conventions.py` owns those); every corpus percentage carries-its battery, its n and its date.+frame, unit or naming rule (`conventions.py` owns those); every corpus percentage carries its+battery, its n and its date.---## 9. Known open-Updated 2026-08-25 (curate wave). Items the three measurement waves **closed** are marked-so; the rest are live.--### Closed this wave--* ~~**The scene track never delivered its skills.**~~ **FIXED.** `SceneTrack.prepare()` (the- pre-round hook then; the track's declared `stages` since 2026-09-22) generated the whole baseline (`_env_stage`, `_zones_stage`, `_assemble_stage`) without ever- reaching `skills_hook.attach_for_round`, which lived only in `steps.run_round`; and- `baseline_tasks` returned `[]`, so round 0 listed four bundles to a session that did not- exist. Measured at **0 opens / 30 listings**. The generating stages now attach and record with- their own kind (`scene.js` is assembled without a session since the compose fallback went,- 2026-09-22). The prompt manifest's scene scenarios pin each stage's `skills.attached`- event and its routed bundles. **The scene bundles' read rate against working delivery- is UNMEASURED — that is the next wave's first experiment.**-* ~~**`api-agent` has no skill affordance.**~~ **FIXED earlier**, and re-measured here: with- `read_skill` it opens **58%** of listed bundles (51/88) across sessions that are offered- the tool, against 0/5 before the tool existed.-* ~~**The A/B rig scored four of seven languages 0.0.**~~ **FIXED.**- `compare_backends._run_harness` gated on `src/model.py`, so a glsl/threejs/scene/opengl run- that finished successfully was recorded `no_code`. Both arms scored 0.0 and the rig would- have reported "no effect" for a switch it never tested.--### Live--* **The mean-count readout is the wrong statistic, and it is what everything was measured- on.** Target metrics are spiky (38–83% of runs at 0). `P(run has ≥1 contract finding)` is- 63% and halving *that* needs ~24 pairs; halving the *mean* needs ~449. **Converting the- readout to a hit rate is the single highest-value cheap change available** and it is what- makes any effect A/B affordable. Not done.-* **Pinning the plan does not collapse the gate-metric noise.** Verified in production (one- planner call per pair, both arms cache-HIT), and it is still cheaper than not pinning — but- `contract_findings` regressed on `n_parts` gives R² = 0.103, and a pinned A/A pair scored- 0.450 vs 0.654 on identical arms. The spread is *generation* noise at fixed plan size.- Do not budget an A/B on the assumption that pinning buys power.-* ~~**`shader_preflight` is not a gate.**~~ **Fixed 2026-09-22**: the scene build's- `scene_probe` + `shader_preflight` reports ride `BuildResult.gates`, and every round- (a failed build's included) appends them to `rounds[].gates`, so R21 can fire and- `c3d-threejs-shader-traps` is instrumented from that date on. Runs recorded before it carry- no `shader_preflight` gate: a readout across the date must not mix the two.-* ~~**`eval/bench/_fixed_eval.FixedEvaluator` is blender-only.**~~ Fixed 2026-08-26; the entry- lives in the local skills ledger (not distributed) §4.-* ~~**The atime probe is blind in any git workspace, and its control proves it.**~~ **Worked- around 2026-09-22**: the probe reads each CLI's own tool calls first (§4) and falls back to- atime only for a session that left no trace. The atime probe itself is as blind as ever —- on the four 2026-09-22 sessions it called every reference "read" and the control opened,- where the transcripts show no reference read and the control untouched. (`.agents/` is- not in the workspace `.gitignore`, so every `git add -A` reads that root; `.claude/` is.)-* ~~**CLI backends' read rate is UNMEASURED — not 5/5.**~~ **Measured 2026-09-22** from the- transcripts: 4 of 4 surfaced on every CLI, 0 deep (§4). claude-code now runs- `--output-format stream-json`, so its `Skill` calls are on record.-* **`codex` can drop a round from the denominator.** It emitted `skills.attached` with no- `skills.read` and no `skills.jsonl` row, so an attached round can silently vanish from- `3dcode skills report`.-* **Read rate confounds with routed-set size.** Every 1-skill session (glsl, opengl) read its- bundle 4/4 while 5-skill object sessions averaged ~55%. Any wording A/B must hold the- routed set size fixed or it measures the cap, not the copy.-* **The prompt corpus no longer contradicts the connectivity gate on weld overlap (D107).**- `tracks/generate_static.j2` ("overlap neighbours by ≥ 0.002 m"), `tracks/assemble_static.j2`,- `generate_static_part.j2` (both deleted by D110), `system/harness_contract.md` ("seams overlap by ≥ 2 mm") and three- cookbooks taught 2–5 mm, at or above `PENETRATION_WARN_M`; all of them now state the gate's- band (gap ≤ 2 mm, overlap ≤ 2 mm) and the cookbooks are gone. It was the single most likely- reason `c3d-part-contact` failed to move the number it targets: re-run that A/B on a- post-D107 tree, and do not pool it with one from before.-* **`ENV_RECIPES` / `ZONE_RECIPES` inlining in `tracks/scene.py` is GONE (D106, 2026-10-15)** —- it was up to 6,000 unconditional tokens per env/zone session, never A/B'd against a routed- skill, and iteration 0 of the argon loop showed what it bought: 60 scenes of one shape. The- scene skills are now the only craft a scene session is offered, as short optional practices.-* **Corpus percentages are checked for provenance, not recomputed.** A body's "47 graded- blender runs" is a *slice*; without each claim declaring its query, a tight recomputation- compares two different populations. Enforced today: a claim may not exceed what `eval/bench/out`- holds, `measured` needs n ≥ 20, every rate names its battery / n / date.-* **`cadquery` and `threejs` bundles are unranked** until each language reaches 20 graded- runs. As of this wave: cadquery **4**, threejs **3**. (Routed by default since 2026-09-22,- with every other `inherited-unverified` bundle; `C3D_SKILLS_UNVERIFIED=0` drops them.)-* **Router PRECISION is not reported.** The read rate says how often a listed skill was read;- nothing says how often a skill was read whose defect class never fired in that run. That- number decides whether the cap of 5 is too generous.+* **The mean-count readout is the wrong statistic.** Target metrics are spiky (38–83 % of runs+ at 0); `P(run has ≥ 1 contract finding)` is 63 %, and halving *that* needs ~24 pairs where+ halving the mean needs ~449. Converting the readout to a hit rate is the cheapest change that+ makes an effect A/B affordable. Not done.+* **Pinning the plan does not collapse the gate-metric noise.** Verified in production (one+ planner call per pair, both arms cache-HIT): `contract_findings` against `n_parts` gives+ R² = 0.103, and a pinned A/A pair scored 0.450 vs 0.654. The spread is generation noise at a+ fixed plan size; do not budget an A/B on pinning buying power.+* **`codex` can drop a round from the denominator** — `skills.attached` with no `skills.read`+ and no `skills.jsonl` row, so an attached round can vanish from `3dcode skills report`.+* **Read rate confounds with routed-set size.** 1-skill sessions (glsl, opengl) read their+ bundle 4/4; 5-skill object sessions averaged ~55 %. A wording A/B must hold the set size fixed+ or it measures the cap.+* **`c3d-part-contact`'s A/B predates D107.** The corpus taught 2–5 mm of overlap, at or above+ `PENETRATION_WARN_M`, until D107 put every prompt on the gate's band (gap ≤ 2 mm, overlap+ ≤ 2 mm) — the single most likely reason the bundle failed to move its number. Re-run on a+ post-D107 tree; never pool with before.+* **Runs before 2026-09-22 carry no `shader_preflight` gate** (it rides `BuildResult.gates`+ since then, so R21 fires): a readout across that date must not mix the two.+* **Corpus percentages are checked for provenance, not recomputed.** A body's "47 graded+ blender runs" is a slice; enforced: a claim may not exceed what `eval/bench/out` holds,+ `measured` needs n ≥ 20, every rate names its battery / n / date.+* **`cadquery` and `threejs` bundles are unranked** until each language reaches 20 graded runs+ (4 and 3 at the last count; `C3D_SKILLS_UNVERIFIED=0` drops them).+* **Router precision is not reported** — how often a skill was read whose defect class never+ fired in that run; the number that decides whether the cap of 5 is too generous.* **`scene_frames/content_*` has no classified kind**, so frame coverage — half of what- `c3d-scene-composition` teaches — neither routes nor counts. Adding a kind widens routing,- which is an effect claim and needs its own evidence.+ `c3d-scene-composition` teaches — neither routes nor counts; adding a kind is an effect claim+ and needs its own evidence.