Official Codebase of 3DCodeVerse
Python
docs: the A/B result — 1 scored pair at +0.002, and the reason: api-agent read 0 of 5 listed bundles while making 52 read_file calls
gaoypengfe6fcd72d4ff
diff --git a/harness/docs/RELEASE_NOTES.md b/harness/docs/RELEASE_NOTES.mdindex fa9c759f..d21c484f 100644--- a/harness/docs/RELEASE_NOTES.md+++ b/harness/docs/RELEASE_NOTES.md@@ -116,7 +116,35 @@ the seat", which is `cv3d-part-contact`'s number, not the prompt corpus's 2–5wheel holds all 14 `SKILL.md`, all 14 `references/` and all 9 `_claims`.* **Corpus** recomputes each bundle's claimed evidence from `bench/out`.-<!-- AB-RELNOTES -->+### The A/B, and why it ships OFF++`bench/out/ab_skills`, eight `static_objects_v2` prompts (the same eight+`bench/out/plan_loop/C0` used for its A/A), `--rounds 1`, paired, fixed judge, arms+differing only in `CV3D_SKILLS=on`. A Gemini capacity outage held the pool at 0–4 of 6 keys+for most of the window: **1 pair scored, 2 cells `infra_failed`** (excluded, never scored 0).++* mean delta **+0.002** (control 0.937, variant 0.939) · CI not computable at n=1 · sign+ test 1/0, p = 1.000 · the rig printed `inconclusive`, `separated from noise: NO`.+* **The read rate on that cell was 0 of 5, with ground truth.** `api-agent` owns its+ `read_file`, so this is a log and not a probe: the agent made **52 `read_file` calls+ across 4 sessions and none of them was a skill**, while calling `read_cookbook` twice.+ The atime probe said 5 of 5 "deep" and its control simultaneously said `control_read:+ true` — the probe was blind, exactly as V-8 predicts, and the calibration arm supplied+ the number it could not.++So the two arms differed by 356 tokens of index that nobody opened, and +0.002 is what that+is worth. **Ship OFF**, for three reasons that do not depend on more pairs: the read rate on+the harness's own generator is 0 %; the bar it was given (mean ≥ 0, no prompt regressed past+0.03) is one the A/A of two *identical* arms fails twice over (−0.038, −0.206); and the+deterministic gate counts inherit the planner's spread directly (an A/A of them swings++5.67 penetrating pairs, `bench/ab_gate_rates.py`).++Next, in order: **give `api-agent` a first-class skill affordance** (a `read_skill` tool+beside `read_cookbook`, or inline the top body) — with 0 % reads no A/B can measure the+library at all; then pin the plan (`docs/EVAL.md` §8.1); then reconcile the prompt corpus on+weld overlap; then re-run. The three subscription CLIs read all five bundles unprompted and+codex's output carried the skill's number, so this is a delivery problem for one backend,+not a verdict on the library.### Known opendiff --git a/harness/docs/SKILLS.md b/harness/docs/SKILLS.mdindex 10ffba7e..7977e170 100644--- a/harness/docs/SKILLS.md+++ b/harness/docs/SKILLS.md@@ -12,7 +12,9 @@ names the cookbook section to fetch. Two libraries of truth would be worse thannot ours to bend: **claude-code, codex, gemini-cli and agy discover these bundlesthemselves** — proven live, see §7. Only `api-agent` needs harness-side injection.-Everything is behind `CV3D_SKILLS`. **It ships OFF.** §6 says why, with the numbers.+Everything is behind `CV3D_SKILLS`. **It ships OFF**, because `api-agent` read 0 of 5+routed bundles in the measured A/B while the three subscription CLIs read all five.+§6 has the numbers.---@@ -293,7 +295,63 @@ that refuses to start against a foreign `codeverse`, and`tests/compare_bench/test_worktree_import.py` — because a workaround protects whoeverremembers it, not the run.-<!-- AB-NUMBERS -->+### What this A/B actually measured++`bench/out/ab_skills`, still running when this was written. Weather was hostile: the Gemini+pool sat at 0–4 of 6 keys answering for most of the window, so the rig parked on its+preflight for two hours and lost a pair to a 503 storm mid-cell.++| | |+|---|---|+| paired prompts scored | **1** of 8 (`ctrl_med_dining_chair`) |+| mean delta | **+0.002** (control 0.937, variant 0.939) |+| 95% CI | **not computable at n = 1**; the A/A band on this battery is ±0.165 |+| sign test | 1 up / 0 down, p = 1.000 |+| `infra_failed` cells | **2** (both arms of `ctrl_med_toaster`), excluded, never scored 0 |+| verdict printed by the rig | `inconclusive`, `separated from noise: NO` |++**And the reason that delta is +0.002 is not that the skills did not help. It is that the+variant arm never read them.**++### The read rate, with ground truth: 0 of 5++`api-agent` is the calibration arm — we own its `read_file` tool, so+`telemetry/skill_reads.jsonl` is not a probe, it is a log. On the scored variant cell:++| | |+|---|---|+| skills listed in message 0, marked MANDATORY | **5** |+| index cost | 356 tokens |+| `read_file` calls the agent made, across 4 sessions | **52** |+| of those, on a skill path | **0** |+| `read_cookbook` calls instead | 2 |+| what the atime probe reported | 5 of 5 "deep" |+| what the control reported | `control_read: true` — **the probe was blind**, exactly as §1 predicts |++So on the backend the entire bench runs on, the mechanism is inert: the agent read 52 files+and not one of them was a skill. The two arms differed by 356 tokens of index that nobody+opened, and +0.002 is what that is worth.++This is not the same answer for every backend. The three subscription CLIs read all five+unprompted (§4) and codex's output carried the skill's number. The difference is the+affordance: a CLI has a first-class skill tool its runtime surfaces, and `api-agent` has a+line in a long markdown file plus a generic `read_file`.++### Verdict++**Ship OFF behind `CV3D_SKILLS`**, and not because the number was negative:++1. On `api-agent`, the read rate is **0 %** with ground truth. Shipping a default-on feature+ that its main generator does not use would pay 356 tokens a turn for nothing.+2. The shipping bar it was given — mean delta ≥ 0 and no prompt regressed by more than 0.03+ — **cannot be cleared by anything on this rig at n = 8**: the A/A of two identical arms+ fails it, twice over (−0.038, −0.206).+3. The deterministic readouts are not an escape hatch either (below).+4. Two of fourteen bundles have no graded runs behind them at all and are already routed off.++Nothing here says the library is wrong. The CLI evidence says the opposite. It says the+delivery mechanism for `api-agent` is a pointer nobody follows, and that is fixable.+### The deterministic readouts are not the escape hatch either@@ -321,6 +379,12 @@ So at n = 8 with a free plan, neither readout can separate this switch from noth### What to try next, in order+0. **Give `api-agent` a first-class affordance.** This is now the top item and it is not a+ measurement problem. Either a `read_skill(name)` tool beside `read_cookbook` — which the+ agent *does* call — or, for the routed set, inline the one highest-priority body the way+ the single-shot arm already does. A pointer in message 0 competes with everything else in+ message 0; a tool in the tool list does not. Re-measure the read rate before anything+ else: with 0 %, no A/B of this switch on `api-agent` can measure the library at all.1. **Pin the plan** (`docs/EVAL.md` §8.1). The skills switch is generation-side — theplanner runs in the `plan` stage before anything is attached — so plan-once-write-both isvalid here, and it removes the dominant variance term from *both* readouts instead of@@ -405,6 +469,9 @@ its battery, its n and its date.## 9. Known open+* **`api-agent` has no skill affordance, and does not read them.** 0 of 5, ground truth,+ §6. It has `read_cookbook` as a tool and skills only as a line of markdown. Until that+ changes, the switch is a no-op for the backend the whole bench runs on.* **Corpus percentages are checked for provenance, not recomputed.** A body's "47 gradedblender runs" is a *slice* (one battery, one track, one round kind) and `bench/out` holds139 blender runs overall; without each claim declaring the query that produced it, a