Official Codebase of 3DCodeVerse
Python
docs: skills default ON (2026-09-22) — per-CLI discovery ground truth, transcript read probe, live evidence
gaoypeng2e32389652f7
diff --git a/harness/docs/ARCHITECTURE.md b/harness/docs/ARCHITECTURE.mdindex 87e3ec1b..20e4bf2a 100644--- a/harness/docs/ARCHITECTURE.md+++ b/harness/docs/ARCHITECTURE.md@@ -141,7 +141,8 @@ codeverse3d/model.py (Skill/Selection + the SKILL.md loader), prompting.py(per-backend delivery policy + the index/mandate text),materialize.py,- telemetry.py (the read probe), config.py (call-time switches)+ telemetry.py (the read probe: the CLIs' own tool calls first, atime as+ fallback), config.py (call-time switches; skills ON by default since 2026-09-22)cost/ types.py (CallCost/Stage/Role) ledger.py (append-only telemetry/cost.jsonl + price provenance)context.py (per-call > ambient attribution) instrument.py (MeteredChatModel /MeteredAgent — one row per ChatModel.generate; one session row only for a backend@@ -192,7 +193,7 @@ codeverse3d/eval/docs/EVAL.md §8.1),depth.py,skills_hook.py (the round's view of codeverse3d/skills: attach before generating,- probe reads after — a no-op unless C3D_SKILLS is on)+ probe reads after — a no-op under C3D_SKILLS=0; on by default since 2026-09-22)record/ what every run WRITES: record.py (finalize_record, load_record, iter_runs,best_round_record), deliverable.py, telemetry.py, _git.py (files at a round's commit)addons/ optional tools that READ finished runs; outside cli/ nothing imports themdiff --git a/harness/docs/INSTALL.md b/harness/docs/INSTALL.mdindex f1465ba2..2776fee4 100644--- a/harness/docs/INSTALL.md+++ b/harness/docs/INSTALL.md@@ -501,11 +501,12 @@ that the keys work end to end):| `pool sharing` | WARN | this process's configured concurrency does not fit beside sibling harness processes | wait for the siblings or set `C3D_MAX_IN_FLIGHT` to the headroom printed in the row || `gemini pool` (`--live`) | WARN | a key is benched as dead | that key 401/403'd; rotate or remove it — the pool re-probes it after an hour. The row also shows in-flight, RPM/TPM headroom used and this process's 429/5xx counts || `storm gate` (`--live`) | WARN | a capacity storm is running | provider-side (`503 high demand`), not an install problem; the gate is off by default (docs/COST.md §21) and the row only appears when something enabled it |-| `skills switch` | WARN | skills are disabled | expected by default; set `C3D_SKILLS=on` only when you want skill routing |+| `skills switch` | WARN | skills are disabled (`C3D_SKILLS=0`) | skills are on by default since 2026-09-22; unset `C3D_SKILLS` unless this is an A/B control arm || `skills library` | WARN / FAIL | no bundles were found, or one or more bundles are invalid | use a full checkout and validate the named bundle under `codeverse3d/skills/<name>/` || `skills routing` | WARN | a routed skill has no installed bundle | restore the missing bundle from the checkout or update the stale route || `skills discovery` | OK | always informational | shows the agent-native directories where bundles are materialised || `claude-code Skill tool` | FAIL | the Claude backend would deny native skill activation | update/reinstall the harness so `Skill` is present in Claude Code's allowed tools |+| `gemini-cli skills setting` | FAIL | the per-session gemini-cli system settings do not pin `skills.enabled` | update/reinstall the harness; without the pin a user's `skills.enabled: false` hides every bundle from gemini-cli |---diff --git a/harness/docs/SKILLS.md b/harness/docs/SKILLS.mdindex ecdf6d82..68a04c32 100644--- a/harness/docs/SKILLS.md+++ b/harness/docs/SKILLS.md@@ -14,9 +14,12 @@ themselves** — proven live, see §7. (The in-process `api-agent`, the one bacneeded harness-side injection, was deleted 2026-08-28; bundles now reach a session onlythrough its vendor's native skill loader.)-Everything is behind `C3D_SKILLS`. **It ships OFF**, because `api-agent` read 0 of 5-routed bundles in the measured A/B while the three subscription CLIs read all five.-§6 has the numbers.+Everything is behind `C3D_SKILLS`, and **since 2026-09-22 it is ON by default** —+`C3D_SKILLS=0` turns it off (an A/B's no-skills arm must now say so). It shipped OFF from+2026-08-25 because `api-agent` read 0 of 5 routed bundles in the measured A/B; that backend+was deleted 2026-08-28, and on 2026-09-22 all four vendor CLIs were watched, through their+own transcripts, indexing the bundles and activating the matching ones unprompted. §6+keeps the history; its dated note has the reasons and the evidence.---@@ -35,8 +38,18 @@ one file, `skills/prompting.py`:| `claude-code` | `.claude/skills` || `codex`, `gemini-cli`, `agy` | `.agents/skills` |-It was read out of the shipped binaries, not assumed. Bundles reach a session only-through its vendor's native skill loader: the harness writes them under the discovery root and+It was read out of the shipped binaries, not assumed — and re-read 2026-09-22 against the+installed versions, with what each CLI needs to see the bundles and what its own record of+a session says it did with them:++| CLI (2026-09-22) | workspace roots · user roots | what must hold | what the model is shown | activation, as its own record logs it |+|---|---|---|---|---|+| `claude-code` 2.1.280 | `.claude/skills` · `~/.claude/skills` (+ claude.ai-synced skills, plugins) | `Skill` in `--allowedTools`; never `--disable-slash-commands` ("Disable all skills") | name + description list; stream-json `init` event's `skills` | `Skill` tool `{"skill": name}`, then `Read` of `references/` |+| `gemini-cli` 0.53.0 | `.gemini/skills`, `.agents/skills` — **only in a trusted folder** · `~/.gemini/skills`, `~/.agents/skills` | `skills.enabled` (default true; the harness pins it in the per-session system settings, merged last) and folder trust (`folderTrust.enabled=false` ⇒ trusted) | `<available_skills>` XML (name, description, location) in the system prompt + `activate_skill` with an enum of names | `activate_skill` `{"name": name}` returns the body; `read_file` for `references/` |+| `codex` 0.155.1 | `.agents/skills` of the working root (`-C <ws>`, also the repo root) · `$CODEX_HOME/skills` (+ `.system`) | nothing — listed from an untrusted `/tmp` workspace too (`codex debug prompt-input`) | `## Skills` section: name, description and an aliased path `r1/<name>/SKILL.md` with a `### Skill roots` table | a shell `sed`/`cat` of `SKILL.md` (an `exec --json` `command_execution` item) |+| `agy` 1.2.2 | `.agents/skills` under the repo root · a global customization root under `~/.gemini/` | nothing; `--disable-slash-commands` only stops `/name` expansion of the prompt (1.1.9 changelog) | a skills section of its system prompt (paths included: its first calls were the four absolute `SKILL.md` paths, with no listing before them) | `view_file` of `SKILL.md` (a step in its conversation database) |++Bundles reach a session only through its vendor's native skill loader: the harness writes them under the discovery root andhands the agent no second index (the loader's own would list the same skills twice) and no tool;the body file gets one MANDATORY sentence.@@ -66,7 +79,14 @@ The spec's three tiers, and what each costs us per turn:| 2 · body | the whole `SKILL.md` | on activation | 1,081–2,223 tokens, once || 3 · depth | `references/*.md` | when the agent chooses to go deeper | as read |-The same three tiers are the read probe. `materialize_skills` writes every file with+**Since 2026-09-22 the probe reads the CLI's own record first** (§4): every backend folds+its session's tool calls into `transcript.jsonl`, and a bundle is *surfaced* when a call+activated it or named its `SKILL.md`, *deep* when a call named a file under its+`references/` (`SkillsUsage.evidence == "transcript"`). The atime probe below is the+fallback for a session that left no trace — and the rest of this section is why it can only+be a fallback.++The same three tiers are the atime read probe. `materialize_skills` writes every file with`atime == mtime`; the root filesystem is ext4 with `relatime`, so the first read afterwardsbumps `atime`:@@ -126,10 +146,13 @@ This is the difference between a metric and a number that would have read 100% f| `c3d-scene-night` | scene | scene_threejs | **inherited-unverified** | 56 | 656 | R27 || `c3d-scene-materials` | scene | scene_threejs | **inherited-unverified** | 78 | 832 | R28 |-**All seventeen ship OFF** (`C3D_SKILLS` unset). Zero have a measured effect; see-`docs/SKILLS_LEDGER.md` §5 for why that is the honest default and what flips one on.-Shipping a subset needs no new machinery — `C3D_SKILLS_ONLY` restricts the library before-routing, so `C3D_SKILLS=1 C3D_SKILLS_ONLY=<names>` is the vehicle when a bundle earns it.+**All seventeen route by default since 2026-09-22** (`C3D_SKILLS` and+`C3D_SKILLS_UNVERIFIED` both default on; `C3D_SKILLS=0` turns the system off,+`C3D_SKILLS_UNVERIFIED=0` drops the six `inherited-unverified` bundles). The cap of 5 per+session still holds. Zero have a measured effect — they route because the owner made skills+part of the product and every backend now loads them natively (§6, 2026-09-22), not because+an A/B promoted them. Measuring a subset needs no new machinery: `C3D_SKILLS_ONLY` restricts+the library before routing, measured against a `C3D_SKILLS=0` arm.### Retired@@ -151,9 +174,9 @@ declares **one deterministic quantity it claims to move**, in its own frontmatte`python eval/bench/skill_targets.py eval/bench/out` prints them all — per battery or paired across anA/B's two arms — and **`docs/SKILLS_LEDGER.md` is the row-by-row maintenance surface**.-`cadquery-forms` and `threejs-forms` stay `inherited-unverified` and routed off: their-languages have **4 (cadquery) and 3 (threejs) graded runs** against the 20 the rule-requires (design §5.2 law 3).+`cadquery-forms` and `threejs-forms` stay `inherited-unverified` — routed by default since+2026-09-22, but unranked: their languages have **4 (cadquery) and 3 (threejs) graded runs**+against the 20 the rule requires (design §5.2 law 3).## 3. Routing@@ -195,9 +218,30 @@ index and never helps, and a test fails on both halves of that.## 4. TelemetryPer round, `RoundRecord.skills` carries a `SkillsUsage`: what was listed, what was-surfaced, what was read deep, `index_tokens`, `body_tokens_read`. Each attached body's hash-goes into `record.prompt_hashes` as `skill:<name>`, so "what text decided this run" stays-answerable. Per session, `telemetry/skills.jsonl` gets one line.+surfaced, what was read deep, `index_tokens`, `body_tokens_read`, and `evidence` — where the+reads came from. Each attached body's hash goes into `record.prompt_hashes` as+`skill:<name>`, so "what text decided this run" stays answerable. Per session,+`telemetry/skills.jsonl` gets one line.++### Ground truth: the CLI's own record (2026-09-22)++Each backend now writes what its CLI says it called into `trajectories/<label>/transcript.jsonl`+— one `tool_call` row per call (arguments compacted to 400 characters a value; the CLI's own+failure verdict as `failed`), then one `tool_trace` row that closes the session+(`agents/cli_common.record_tool_calls`). Where each one reads it:++| backend | source | skill activation looks like |+|---|---|---|+| claude-code | `--output-format stream-json --verbose` (was `json`, which carried no tool stream); the `init` event's `skills` list is kept as `skills_index` | `Skill {"skill": name}` |+| gemini-cli | its chat record, `~/.gemini/tmp/<project>/chats/session-*.jsonl`, found by the project's `.project_root` + mtime — so a session the watchdog killed before it printed JSON still has one. `--output-format json` stays: it has per-tool COUNTS only, and `stream-json` drops the `thoughts` tokens pricing needs | `activate_skill {"name": name}` |+| codex | the `exec --json` events it already streamed (`command_execution`, `mcp_tool_call`, `file_change`) | a `sed`/`cat` of `…/<name>/SKILL.md` |+| agy | its conversation database, `~/.gemini/antigravity-cli/conversations/<conversation_id>.db` — a schema-free walk of each step's protobuf for (call id, tool, JSON args) | `view_file` of `…/<name>/SKILL.md` |++`skills/telemetry.probe_reads` credits only sessions logged after the attach+(`SkillsMaterialized.attached_at`), and uses the transcript only when **every** such session+closed a trace — a partial trace under-counts exactly as the blind probe over-counts. Under+transcript evidence `control_read` means the AGENT opened the control: counted by+`3dcode skills report` (and flagged), never dropped as blind.```3dcode skills report eval/bench/out/<battery> # deep-read rate per skill per backend@@ -212,7 +256,23 @@ The read rate is the FIRST readout, before any score: `read_cookbook` was calledzone sessions on `scenes_v1_flash` although the prompt named five chapters by title, andshipping a nicer file format without measuring reads would repeat that at a new price.-### Measured, on an ordinary build task+### Measured, on an ordinary build task — 2026-09-22, from the transcripts++A harness-built static_object/blender workspace (four routed bundles + the control), each+CLI launched by its own backend's `run()`, a chair brief that never mentions skills:++| CLI · model | index the CLI showed the model | activated (surfaced) | deep | control | atime probe on the same session |+|---|---|---|---|---|---|+| gemini-cli 0.53.0 · gemini-3.7-flash (×2) | `gemini skills list`: the 4 + control | **4 / 4** in both sessions, all in the first turn (`activate_skill`); both later killed by a provider storm | 0 | untouched | 4/4 deep, control "read" → blind |+| claude-code 2.1.280 · sonnet | `init.skills`: the 4 + control (+ 26 bundled / user-level) | **4 / 4** (`Skill`) | 0 | untouched | 4/4 deep, control "read" → blind |+| codex 0.155.1 · gpt-5.6-luna | `## Skills`: the 4 + control (+ 5 `.system`) | **4 / 4** (`sed` of `SKILL.md`) | 0 | untouched | 4/4 deep, control "read" → blind |+| agy 1.2.2 · gemini-3.7-flash-medium | its first four calls were the four `SKILL.md` paths | **4 / 4** (`view_file`) | 0 | untouched | 4/4 deep, control "read" → blind |++Every CLI read every routed body and no reference file; the atime probe on the same four+sessions reported every reference "read" and the control opened — git, not the agent.+Details, costs and commands: `docs/SKILLS_LEDGER.md` §0b.++### Measured 2026-08-25, on the atime probeNot a compliance prompt — "write `src/model.py` and `src/parts/*.py` for a four-leggeddining chair, follow whatever conventions this workspace documents", five routed bundles@@ -296,7 +356,35 @@ Two contradiction checks are worth separating, because they answer different que## 6. Does it help? The A/B-**Verdict: ship OFF behind `C3D_SKILLS`.**+### 2026-09-22: default ON++**The owner's call, and the evidence behind it.** Everything below this note is the+2026-08-25/26 history, kept as written. What changed since it was written:++1. **The reason for OFF is gone.** Reason 1 of the verdict below is `api-agent`'s 0 % read+ rate — and `api-agent` was deleted 2026-08-28. Every backend left is a vendor CLI with a+ native skill loader, and the ground-truth channel this section wished it had (§9, "CLI+ backends' read rate is UNMEASURED") now exists: each backend logs its CLI's own tool calls.+2. **All four CLIs load the bundles, unprompted, and it is proven from their transcripts**+ (§4 table; `docs/SKILLS_LEDGER.md` §0b): gemini-cli, claude-code, codex and agy each+ activated **4 of 4** routed bundles on an ordinary chair brief, none touched the control,+ none read a reference file. The discovery wiring was re-read from the installed binaries+ and two gaps were closed: gemini-cli's `skills.enabled` is now pinned in the per-session+ system settings (a user's `false` would have hidden every bundle), and claude-code streams+ (`stream-json`) so its `Skill` calls are on record.+3. **An effect is still unmeasured, and that is now the A/B's job, not the default's.** The+ shipping bar in reason 2 cannot be cleared by anything at n = 8 (its own A/A fails it);+ skills are part of the product. An A/B of skills now names its no-skills arm+ explicitly: `eval/bench/ab_plan.py --variant-env C3D_SKILLS=0` — ab_plan's control is the+ driver's env minus the variant's keys, so `--variant-env C3D_SKILLS=1` is now two+ identical arms, and the OFF arm has to be the variant (its delta reads OFF − ON).++What default ON costs: the mandate sentence (41 tokens) in every session's body file, plus+the bodies the CLI chooses to activate — measured 2026-09-22 at all four of four, i.e. up+to the cap of five bodies (1,081–2,223 tokens each) per session. `C3D_SKILLS_MAX` is the+cost knob; `C3D_SKILLS=0` the off switch.++**Verdict (2026-08-25/26, superseded 2026-09-22): ship OFF behind `C3D_SKILLS`.**### The rig@@ -399,7 +487,7 @@ unprompted (§4) and codex's output carried the skill's number. The difference iaffordance: a CLI has a first-class skill tool its runtime surfaces, and `api-agent` has aline in a long markdown file plus a generic `read_file`.-### Verdict+### Verdict (2026-08-25/26 — superseded by the 2026-09-22 note above)**Ship OFF behind `C3D_SKILLS`**, and not because the number was negative:@@ -467,6 +555,14 @@ the read probe itself plus a magic word that only lives in `references/`.| claude-code | 2.1.245 | `.claude/skills/` | yes | yes | yes || agy (antigravity) | 1.1.20 | `.agents/skills/` | yes | yes | yes |+Re-checked 2026-09-22 on gemini-cli 0.53.0, claude-code 2.1.280, codex 0.155.1 and agy+1.2.2, through the harness's own launch path and read from each CLI's own transcript (§4):+same roots, all four activate the routed bundles. Two more things that could only be+settled against the binaries: gemini-cli skips BOTH workspace roots in an untrusted folder+(`SkillManager.discoverSkills`), which the harness's `folderTrust.enabled=false` already+avoids; and agy's `--disable-slash-commands`, which the harness passes, only stops `/name`+expansion of the prompt — its model still reads the bundles.+All four discover the library natively, with no injection from us. Two things this settledthat had only been argued statically:@@ -570,16 +666,15 @@ so; the rest are live.`rounds[].gates` before spending anything on that bundle.* ~~**`eval/bench/_fixed_eval.FixedEvaluator` is blender-only.**~~ Fixed 2026-08-26; the entrylives in `docs/SKILLS_LEDGER.md` §4.-* **The atime probe is blind in any git workspace, and its control proves it.** The control- bundle came back "opened" in 27 of 33 sessions; the only 6 sessions where it stayed clean- were the 6 scene sessions, where nothing was delivered. So the shipped report is honest- only where there was nothing to see. Trajectory `read_skill` calls were the ground truth;- that channel went with `api-agent` (2026-08-28).-* **CLI backends' read rate is UNMEASURED — not 5/5.** `agents/backends.py` runs claude-code with- `--output-format json`, which returns only the final result and no tool stream; the logs- contain zero occurrences of any `c3d-` name. Switching to `--output-format stream-json`- would give the CLI arm the same exact ground truth api-agent had. The earlier "5 of 5"- claim rested on the atime probe and should not be repeated.+* ~~**The atime probe is blind in any git workspace, and its control proves it.**~~ **Worked+ around 2026-09-22**: the probe reads each CLI's own tool calls first (§4) and falls back to+ atime only for a session that left no trace. The atime probe itself is as blind as ever —+ on the four 2026-09-22 sessions it called every reference "read" and the control opened,+ where the transcripts show no reference read and the control untouched. (`.agents/` is+ not in the workspace `.gitignore`, so every `git add -A` reads that root; `.claude/` is.)+* ~~**CLI backends' read rate is UNMEASURED — not 5/5.**~~ **Measured 2026-09-22** from the+ transcripts: 4 of 4 surfaced on every CLI, 0 deep (§4). claude-code now runs+ `--output-format stream-json`, so its `Skill` calls are on record.* **`codex` can drop a round from the denominator.** It emitted `skills.attached` with no`skills.read` and no `skills.jsonl` row, so an attached round can silently vanish from`3dcode skills report`.@@ -600,8 +695,9 @@ so; the rest are live.blender runs" is a *slice*; without each claim declaring its query, a tight recomputationcompares two different populations. Enforced today: a claim may not exceed what `eval/bench/out`holds, `measured` needs n ≥ 20, every rate names its battery / n / date.-* **`cadquery` and `threejs` bundles are routed off** until each language reaches 20 graded- runs. As of this wave: cadquery **4**, threejs **3**.+* **`cadquery` and `threejs` bundles are unranked** until each language reaches 20 graded+ runs. As of this wave: cadquery **4**, threejs **3**. (Routed by default since 2026-09-22,+ with every other `inherited-unverified` bundle; `C3D_SKILLS_UNVERIFIED=0` drops them.)* **Router PRECISION is not reported.** The read rate says how often a listed skill was read;nothing says how often a skill was read whose defect class never fired in that run. Thatnumber decides whether the cap of 5 is too generous.diff --git a/harness/docs/SKILLS_LEDGER.md b/harness/docs/SKILLS_LEDGER.mdindex eeb5c999..caf9d8af 100644--- a/harness/docs/SKILLS_LEDGER.md+++ b/harness/docs/SKILLS_LEDGER.md@@ -7,8 +7,10 @@ called earned.**Library: 17 bundles** (13 + the 2026-09-01 scene-graphics port: atmosphere, water,night, materials — routes R25-R28, evidence `inherited-unverified` from the-scene_multifile_graphics reference ledger). All 17 ship **OFF** (`C3D_SKILLS` unset). Zero have a measured-effect. §5 says why that is the honest default and what would change it.+scene_multifile_graphics reference ledger). **All 17 route by default since 2026-09-22**+(`C3D_SKILLS` and `C3D_SKILLS_UNVERIFIED` default on; `C3D_SKILLS=0` is the off switch) —+the owner's call, on the live evidence in §0b. They shipped OFF from 2026-08-25 to+2026-09-22; §5 records why that was the honest default then. Zero have a measured effect.Three files hold it up:@@ -65,6 +67,73 @@ gate_valve, turbocharger, marimba, jacobs_chuck redo) land in `results.jsonl` pemerge with `python eval/bench/skill_targets.py` or the paired readout (scratch `fancy_readout.py`— rows are only paired when both arms are `scored`).+## 0b. 2026-09-22 — all four vendor CLIs load the bundles; default ON++**Question.** With `api-agent` gone (2026-08-28), does each vendor CLI the harness drives+actually see the routed bundles and load them — and can we tell from the CLI's own record+instead of from atime?++**Rig** (scratch, not a battery). One workspace per CLI, built through the harness's own+code path: `Workspace.create` → `spec.json` (static_object / blender) →+`agents.materialize.materialize_workspace` → `skills.attach_skills(kind="baseline")` with a+chair plan (4 parts, 4 mirrored legs, 3 slats). The router picked **four** bundles —+`c3d-blender-forms` (R8), `c3d-part-contact` (R1), `c3d-bbox-contract` (R3),+`c3d-repeats-and-mirrors` (R6) — and materialised the never-routed control+`zz-c3d-read-control` beside them in both roots. Each session was launched by+`get_coding_agent("<kind>:<model>").run(AgentJob(..., kind="baseline", spatial_tools=True))`+— the metered backend and the argv/env/settings builders the round loop uses — with the+blender `system.md` as `system_append` and a chair brief that never mentions skills (the+only skills text in the session is the harness's one mandate sentence in the body file).+Workspaces under an untrusted `/tmp` path on purpose. Reads come from the tool trace each+backend now writes (`agents/cli_common.record_tool_calls`); the atime probe was run on the+same workspace afterwards for comparison.++**Static checks first, no model call:** `gemini skills list` under the harness's per-session+system settings listed the four + the control, all `[Enabled]`, from+`<ws>/.agents/skills`; `codex debug prompt-input` in an untrusted workspace rendered a+`## Skills` section with the four + the control as `r1/<name>/SKILL.md`, `r1` =+`<ws>/.agents/skills` (plus codex's five `.system` skills).++| CLI · model | exit | index the model was shown | activated (surfaced) | refs (deep) | control | atime probe, same session | cost |+|---|---|---|---|---|---|---|---|+| gemini-cli 0.53.0 · gemini-3.7-flash, session 1 | timeout 1 202 s — a `fetch failed` / 503 storm inside the CLI's retry loop | the 4 + control (`gemini skills list`) | **4 / 4** — `activate_skill` ×4 in its FIRST turn | 0 | untouched | 4/4 deep, control "read" (blind) | harness $0 (no envelope); its chat record: 1.19 M prompt (0.92 M cached), 19.0 k out+thoughts ≈ **$0.34** |+| gemini-cli 0.53.0 · gemini-3.7-flash, session 2 | timeout 902 s — same storm | same | **4 / 4** — `activate_skill` ×4 in its first turn | 0 | untouched | 4/4 deep, control "read" (blind) | harness $0; chat record ≈ **$0.34** |+| claude-code 2.1.280 · sonnet (served claude-sonnet-5) | completed, 294 s, 25 turns, 20 calls | `init.skills`: the 4 + control (+ 26 bundled / user-level) | **4 / 4** — `Skill` ×4, its first four calls | 0 | untouched | 4/4 deep, control "read" (blind) | $0.955 reported (subscription) |+| codex 0.155.1 · gpt-5.6-luna (effort high) | completed, 545 s, 29 calls | `## Skills`: the 4 + control (+ 5 `.system`) | **4 / 4** — `sed` of each `SKILL.md`; its first `sed` mis-expanded `r1` as `r0`, exited 2 and is NOT credited (`failed`) | 0 | untouched | 4/4 deep, control "read" (blind) | ≈ $0.085 at API prices (subscription) |+| agy 1.2.2 · gemini-3.7-flash-medium | completed, 102 s, 22 calls | its first four calls were the four absolute `SKILL.md` paths, no listing before them | **4 / 4** — `view_file` ×4 | 0 | untouched | 4/4 deep, control "read" (blind) | $0 (subscription) |++**Read-out.**+* **Every CLI loaded every routed bundle, unprompted, and never the control** — 20 of 20+ activations across five sessions, all before the agent wrote a line of code. codex said+ so in its first message ("I'm using the chair-relevant 3D skills: Blender form construction,+ bbox/contract control, part-contact checks, and mirrored/repeated-part handling").+* **No session opened a single `references/` file.** The depth tier is unused on a baseline;+ the old "deep" 100%s were git (§4 of docs/SKILLS.md). This is now a measured 0, not an+ unmeasured one — the first honest reading of the design's depth metric.+* **The atime probe was wrong on every session** and its control said so each time — the+ reason the probe now reads transcripts first.+* The two gemini sessions were killed by a provider storm, not by skills; both still left+ their tool trace (found by `.project_root` + mtime, not `session_id`, for exactly this+ case). They also expose a cost gap outside this lane: a killed gemini-cli session is+ recorded at $0 although its chat record holds ~$0.34 of tokens.+* codex's MCP server did not start in this rig (codex hands MCP servers a scrubbed env; the+ lane worktree is on `PYTHONPATH`, not installed), so it spent its session probing the+ server by hand — a rig artefact, not a skills result.++**Gaps closed** (code, 2026-09-22): gemini-cli `skills.enabled` pinned in the per-session+system settings (merged last; a user's `false` would have hidden every bundle); claude-code+switched to `--output-format stream-json --verbose` (its `json` output carried no tool+calls); every backend writes its CLI's tool calls into `transcript.jsonl` and the read probe+reads them first; agy's `AgentResult.tool_calls` is the trace count (was always 0).+**Verified not to be gaps:** gemini-cli folder trust (off ⇒ trusted ⇒ both workspace roots+load; ON would skip them), codex trust (none needed), agy's `--disable-slash-commands` (only+`/name` prompt expansion), claude-code's `Skill` in `--allowedTools` (present since 2026-08-25).++**Decision:** `C3D_SKILLS` and `C3D_SKILLS_UNVERIFIED` default ON (docs/SKILLS.md §6, the+2026-09-22 note). An A/B of skills now names its no-skills arm explicitly:+`eval/bench/ab_plan.py --variant-env C3D_SKILLS=0` (ab_plan's control inherits the driver's+env minus the variant keys, so `C3D_SKILLS=1` would be two identical arms).+## 0. What this wave decided (2026-08-25, curate)Three measurement waves ran: a description A/B read out on **read rate** (ground truth from@@ -273,7 +342,7 @@ it lands these four rows stay `mixed`, not `measured`.----## 5. What ships, and why the answer is "nothing"+## 5. What ships, and why the answer was "nothing" (2026-08-25; superseded 2026-09-22, §0b)The library does **not** have to share one switch. `C3D_SKILLS_ONLY` restricts the library*before* routing, so shipping a subset is one env var: