Official Codebase of 3DCodeVerse
Paper writing: everything measured, and where it came from
Paths in this document.
bench/…is relative toeval/(this folder's parent);codeverse3d/…,tests/…,runtime_js/…and the otherdocs/…files are relative toharness/. Recorded battery output (bench/out/…) is run data and is not in git.
Everything a paper about this harness would need in one file: what the system is, what was built, every experiment with its setup and numbers, the null results, the mechanism findings, the methodology, and a claim → evidence index. Written 2026-09-03/04 over the work of 2026-08-25 → 09-04; each number names the run directory that produced it.
This file is the LEDGER, not a second copy of the measurements. Each entry says what
was claimed, which script recomputes it, which data it reads and what its status is; the
measurement itself lives once, in COST.md (§23, §29, §30, the cost measurements),
DECISIONS.md (D36–D52, the decision log), EVAL.md (§8–§9, the protocol and the noise
floor) or ARCHITECTURE.md. A table repeated here is a table that will drift from the one
that owns it — §5.2 was exactly that until 2026-09-03 (review) and is now a pointer. Tools referenced throughout:
bench/prompts/{compare_v4,articulated_v2}.yaml (batteries), bench/ab_plan.py (paired
A/B with an A/A mode), bench/compare_backends.py (harness vs one-shot),
bench/paired_compare.py (paired statistics), bench/plan_stage_bench.py +
bench/plan_stage_report.py (the loss-event channel, data in bench/data/plan_stage/),
3dcodeverse flywheel refine (the transition corpus) and finetune/llamafactory/ (its training
format).
0. The system, in one paragraph
A request becomes an iterated loop: plan → generate code → build → deterministic gates →
render → VLM judge → refine, for a fixed N rounds under a wall-clock budget, recording every
round and handing over the best-scored one after the run. The generator is a coding agent writing real
source (Blender bpy, CadQuery, three.js, GLSL, or a URDF robot plus a bpy link builder);
the gates are program analysis, not model calls (lint, connectivity, dimensional contract,
and — on the articulated track — a joint sweep that collides every link pair in every sampled
pose, plus a planned-motion check); the judge is a VLM scoring rendered views against a
rubric. The recorded rounds are themselves the training corpus the project is built to
produce. The articulated track is the hardest and the one this ledger studies: its
artefact has kinematics a static object does not, and it was the harness's weakest track at
the start of this work — 5 of 14 prompts died at the planner, and 213 of 214 gate errors in a
full battery were joint_sweep.
0.1 Contributions a paper could claim
- A measured advantage for an iterated, gated, judged loop over one-shot generation, on two tracks, with paired confidence intervals and a judge-free corroboration (§3).
- An instrument section that most agent papers do not have: an A/A calibration that states the noise floor and the number of prompts needed to resolve a given effect, and four honest null results reported as null (§2, §4).
- A cost model for agentic code loops, with per-turn and per-round token accounting, the quadratic-in-turns growth, and the cache term that decides which savings are real (§6).
- A defect class specific to tool-using agents — a tool's negative verdict delivered as a protocol error — with its retry cost, a 507× token amplification when the result carries an image, and the before/after measurement of the fix (§5.2).
- A cheap methodology for changes whose effect the judge cannot see: isolate the stage, count the loss event, run n in the hundreds (§5.1).
- Coupled-mechanism support for URDF articulation with a mechanism-level readout (§5.3).
- A refine corpus: (judge issues + gate findings + instructions) → code diff → score
delta;
finetune/llamafactory/converts those rows into the finetune pipeline's format, which the harness itself never writes (§7).
0.2 Evaluation-protocol work that predates the measurements
These are prerequisites, and several are results in their own right (decision log D36–D43):
- Tie-breaking on an even
n_samplesfollowed a fixed direction, which made then_samples=2fixed evaluator strictly harsher than n=1 or n=3 and not comparable with either; ties now follow the representative sample and the broken ids are recorded (D36). judge_prompt_hash(role prompt + rig rules + wire schema) is stored in every score, so a rubric revision cannot be silently compared across (D37).harness_git_shawas empty in every record because the probe looked for a.gitthat does not exist at that level (D38).- Failures of the arm are zeros; failures of the provider are dropped (D41). Before this, an outage scored a one-shot arm 0 while the harness arm hit by the same outage was dropped — biasing every comparison in the harness's favour. Extended in this work with three more provider modes (§5.4).
- Storm-degraded cells (budget stop, ≤1 completed round, ≥40 min, money left) are
flagged and excluded from an
all −degradedrow (D40); the first 40-prompt battery ran inside an all-day 503 storm and 37 of 40 harness cells hit the wall clock with 0–2 rounds. - A lint that raises kills the round: every Python lint now parses through
safe_parse(D43). - The offline suite is every test directory: the removed CI job listed 7 of 24 (D39).
1. The battery and the arms
bench/prompts/articulated_v2.yaml — 14 prompts, 4 medium / 10 hard: architect lamp, tool
chest, camera tripod, dutch door, rolltop desk, folding workbench, parallel clamp, sash
window, grand piano, umbrella, casement window, step ladder, draw-leaf table, scissor mirror.
Three of them (umbrella, scissor mirror, folding workbench) are one-input coupled mechanisms
and are the lowest scorers throughout. The static battery is compare_v4.yaml, 40 prompts.
Arms, as the drivers name them: harness:<agent>:<model> (the loop),
oneshot:<model> (one call, one answer), oneshot+repair:<model> (one call plus one repair
call on the build error). The generator moved from an in-process api-agent to
gemini-cli:gemini-3.7-flash upstream on 2026-08-27; comparisons that cross that boundary
are marked as confounded wherever they appear below.
2. The instrument, and why it is the first result
Everything below is judged by a fixed evaluator: the delivered GLB re-rendered on a
14-view rig and scored by gemini:gemini-3.1-pro-preview with n_samples=3 against the
articulated rubric. Arms are paired per prompt. --pin-plan runs the planner once
and seeds both arms, removing the planner's spread from the difference.
A/A calibration (2026-09-02, bench/out/aa_articulated). Two identical arms on
articulated_v2 (14 prompts, generator gemini-cli:gemini-3.7-flash, 3 rounds, 60 min per
cell, plan pinned). 12 pairs survived (2 lost at the pinned plan to provider read timeouts
and redone; 0 budget-exhausted):
| quantity | value |
|---|---|
| mean Δ (identical arms) | −0.046 |
| median Δ | −0.048 |
| paired sd | 0.225 |
| SE | 0.065 |
| 2 SE band | ±0.130 |
| sign | 3 up / 8 down (p = 0.227) |
| separated from noise | no |
| pairs needed to resolve ±0.02 | ~506 |
The rig printed "revert" from two runs of the same code. That is the number every other result on this battery has to be read against: on 12–14 paired prompts the judge mean cannot decide a single-switch change. It also explains the four null results in §4 without appealing to the changes themselves.
The consequence: two channels.
- The judge channel (mean score, paired) is reserved for comparisons big enough to clear ±0.13 — harness vs one-shot, not switch vs switch.
- The loss-event channel counts the events that destroy a run or a round: a plan that never validated, a tool call the model retried, a pose the mechanism cannot reach, a round with gate errors left. These are cheap enough to measure at n in the hundreds, and they are what actually moved.
3. What the harness is worth (judge channel, where it resolves)
| comparison | battery | n pairs | means | Δ | 95 % CI | verdict |
|---|---|---|---|---|---|---|
| harness − one-shot + repair | compare_v4_calm (static, 40) |
40 | 0.633 vs 0.487 | +0.146 | [+0.077, +0.215] | supported |
| harness − one-shot | compare_v4_calm |
40 | 0.633 vs 0.232 | +0.401 | [+0.297, +0.505] | supported |
| harness − one-shot + repair | compare_art_v2 (articulated, 14) |
14 | 0.309 vs 0.316 | −0.007 | [−0.288, +0.273] | parity — 5/14 harness runs died at the planner |
| harness − one-shot + repair | compare_art_v3 (after the planner repair) |
14 | 0.468 vs 0.316 | +0.152 | [+0.061, +0.243] | supported (sign p 0.039) |
| harness − one-shot + repair | compare_art_v4 (3 harness arms) |
13–14 | — | +0.19 … +0.33 | all exclude 0 | supported in every arm |
Judge-free, same runs: on the static battery the harness builds 100 % of the time with 1.5 gate errors on average (75 % of cells at zero) against one-shot+repair's 100 % / 17.3 / 12 % and one-shot's 50 % / 17.9 / 0 %.
The articulated parity → +0.152 transition is itself a result: the harness's articulated advantage was hidden by planner mortality, not by the loop's quality.
4. The four null results (judge channel, where it does not resolve)
Every one of these is a real change, measured, that could not be resolved on this battery. They are recorded so nobody re-runs them expecting a different answer.
| lever | battery / method | n pairs | Δ | 2 SE | sign | disposition |
|---|---|---|---|---|---|---|
Plan-time geometry re-ask (C3D_PLAN_GEOMETRY, D49) |
compare_art_v4 pf vs pf0 |
14 | +0.064 | ±0.25 | 6/6/2 | shipped OFF; code removed 2026-09-21 (D49), the numbers stand |
Pro planner (gemini-3.1-pro vs 3.7-flash) |
compare_art_v4 pp vs pf |
13 | +0.090 | ±0.23 | 9/2/2 | keep flash: 2–10× plan cost, 5/14 cells lost to provider limits |
Deterministic repairs (C3D_ART_REPAIRS, a bundle since removed; its axis flip lives on as upstream's C3D_AXIS_REPAIR, ON by default) |
ab_repairs, plan pinned |
12 | −0.039 | ±0.105 | 2/5/5 | the bundle does not ship; the axis flip fired in 1 of 12 cells |
Fewer turns (C3D_FEWER_TURNS) |
ab_fewer_turns, plan pinned |
14 | +0.015 | ±0.126 | 6/5/3 | ships OFF; −12 % tool calls, $/cell unchanged |
Lean prompt (C3D_LEAN_PROMPT) |
wave2_lean, plan pinned |
12 | +0.030 | ±0.076 | 8/3/1 | ships OFF; −45.7 % generate prompt chars, $3.51 vs $3.66 per cell; code removed 2026-09-03 in review (D52); the numbers stand, the second prompt path does not. |
Two of these carry a second, non-score reading worth keeping:
- The geometry re-ask fired in 7 of 14 cells and retro-fires on 8 of 14 recorded
compare_art_v3 plans (mean score 0.362 for the plans it complains about vs 0.610 for the
clean ones) — it identifies bad plans, it just does not fix enough of them to show. With the
check ON vs OFF: final gate errors 0.00 vs 0.25, round-0 distinct
joint_sweeptargets 0.54 vs 0.25 (D49). C3D_FEWER_TURNSandC3D_LEAN_PROMPTboth cut tokens without cutting dollars. The reason is in §6: the token growth is quadratic in turns but lands in the cache, so the bill is not where the tokens are.
5. What did move: loss events
5.1 Planner mortality (the one significant single-switch result)
Method. bench/plan_stage_bench.py runs the plan stage alone — no build, no
judge — at ≈$0.03 and ≈85 s per call, so 280 calls per arm is affordable where 14 judged
cells is not. Both arms run in the same window against the same provider. The 560 rows
are in the tree (bench/data/plan_stage/*.jsonl, bench/plan_stage_report.py prints the
table): a paper cites the p-value, so the data it comes from is committed.
Failure class, measured first. 200 calls over two code trees showed one class: the planner writes one top-level part and hangs its joints off links it never lists, and then reproduces that answer through both in-context re-asks. The two re-asks shipped the week before changed nothing (29/30 vs 28/30 valid, same window) — the model conditions on its own broken output.
The change (C3D_PLAN_RESTART, ON with a kill switch): on that failure the
conversation is rebuilt from the original request plus the rule the answer broke, instead
of appending the broken plan and a complaint.
| arm | plan calls | valid | PlanningError |
provider | failure rate | 95 % CI |
|---|---|---|---|---|---|---|
| restart ON | 280 | 274 | 2 | 4 | 0.7 % | [0.2 %, 2.6 %] |
| restart OFF | 280 | 264 | 13 | 3 | 4.7 % | [2.8 %, 7.9 %] |
Fisher exact two-sided p = 0.0067. The restart fires on 14 % of calls and recovers 39 of 40. Cost per call is unchanged ($0.0345 vs $0.0337) and wall time is slightly lower (82 s vs 86 s): one extra sample replaces two doomed re-asks. Failures under OFF are spread over six prompts, so this is not one pathological request.
Re-measured after review narrowed the trigger (2026-09-03; the trigger now also
requires a collapsed plan, and a restart no longer spends one of the two validation re-ask
slots). Three arms in ONE window, 700 calls,
bench/data/plan_stage/trigger_{off,wide,narrow}.jsonl:
| arm | judged | losses | rate | dangling_link |
parent_eq_child |
other |
|---|---|---|---|---|---|---|
| restart off | 140 | 4 | 2.9 % | 3 | 1 | 0 |
| trigger as measured above | 275 | 6 | 2.2 % | 4 | 2 | 0 |
| trigger narrowed | 276 | 5 | 1.8 % | 0 | 4 | 1 (budget ceiling) |
No pair separates on the overall rate (Fisher 0.45–0.75) — and this window's own
control loses 2.9 %, not 4.7 %, so the headline above is a property of its window as much
as of the switch. The separation is inside the class the mechanism targets: 3/140 off
vs 0/276 narrowed, p = 0.038. Order matters for how that is read: the three overall-rate
tests were run first and came back null, and the class split was examined afterwards, so
this is exploratory evidence on a pre-specified mechanism rather than a pre-registered
test. bench/plan_stage_report.py prints the class table and the per-class p. All four of the old trigger's dangling-link deaths carry
restarts=1 and died at the validation cap — the re-ask slot the restart used to consume.
The narrowed trigger fires on 28 of 280 calls against 38 and recovers 27 of 28 (96 %)
against 33 of 38 (87 %).
A second failure class, unaddressed. The residue in every arm is a joint whose
parent and child are the same link (1 / 2 / 4, flat). Those calls never restart — the
plan lists every link it names — and the model rewrites the same joint through all three
re-asks, which is the same shape as the collapsed-plan class before it was handled. Two
lessons for a paper: report a loss rate WITH its window, because the class mix moves; and
report the class, because the total dilutes a mechanism that only removes one of them.
5.2 Tool calls the model retried, and the token blow-up behind them
docs/COST.md §30 is the measurement; this is the ledger entry. The tables live there once — the per-tool error rates, the base64 arithmetic, the before/after table and the selector — and repeating them here is how four copies of one number drift apart. What a paper needs from this section:
- The fault.
spatial/mcp_server.pyreturnedis_error = not obs.ok, andobs.okis the tool's verdict, so a joint sweep that ran correctly and reported a penetration reached the model as a broken call it then retried. The per-tool rates are COST §30's first table; they replicate across five recorded batteries (bench/session_stats.py), so it is the mechanism and not one window. - The amplifier. gemini-cli's error path stringifies the whole result, so a 275 kB articulation sheet arrives as ≈261 k prompt tokens of base64 text instead of ≈516 as an inline image (≈507×), escaping the vendor's own truncation. ~10 % of requests carried ~225 k uncacheable tokens — two thirds of the entire uncached bill.
- Not a caching defect. Non-blob requests already cached as well as the deleted
in-process agent (4 946 vs 5 639 uncached tokens per turn); nothing about settings, tool
ordering or
GEMINI.mdneeded to change. A negative finding that saved an A/B budget. - The change.
Observation.failed("the tool could not run") besideok(the verdict); MCPis_errorisfailedalone; every affected tool leads its text with the verdict; the MCP boundary bounds text characters, image count and encoded image bytes. - The readout (
python bench/session_stats.py bench/out/aa_articulated bench/out/wave2_lean, both columns printed by the script): calls reported as errors 26.6 % → 1.3 %, cache 69 % → 90 %, uncached prompt tokens per request 38 521 → 13 480, generator dollars per round median 1.572 → 0.950. The residual 1.3 % are genuineObservation.errorcases. An earlier hand-computed after column (1.4 %, 13 736, $0.967) is withdrawn: the script does not reproduce it and no tested mechanism explains its shape — COST §30 carries that note. - Why it matters here: this is the largest effect found in the whole effort, and it has no signature in the judge mean at n = 14.
5.3 Coupled mechanisms
The fault. <mimic> was a lint WARN reading "ignored by the harness", and the sweep
drove every movable joint independently. An umbrella was therefore posed with each rib at
a different angle — a state the mechanism cannot reach — and the judge scored that, while
the fix hints told the agent to move links that are not free to move. The three
lowest-scoring prompts on the battery (umbrella, scissor mirror, folding workbench) are all
one-input mechanisms.
The change. Joint carries a Mimic(joint, multiplier, offset) parsed from <mimic>;
load_urdf refuses a dangling name, a fixed driver, a zero multiplier and a cycle;
resolve_q expands one input through the chain and fk applies it (a value passed for a
driven joint is ignored); pose_samples drives independent_joints() only; the motion
check probes a driven joint through its driver; JointPlan.mimic makes the coupling a
planner decision and the skeleton writes it; the lint checks it instead of dismissing it.
Measured (wave2_lean, both arms; python bench/coupling_stats.py bench/out/wave2_lean --per-prompt recomputes every number here from the recorded URDFs — what the sampler used
to drive is every movable joint, what it drives now is the independent ones, and both are
properties of the file):
| prompt | coupled URDFs | movable joints (median) | driven per pose (median) |
|---|---|---|---|
| umbrella | 12 | 7 | 1 |
| step ladder | 12 | 5 | 1 |
| folding workbench | 12 | 6 | 2 |
| scissor mirror | 12 | 6 | 3 |
| grand piano | 12 | 4 | 3 |
| architect lamp | 20 | 5 | 4 |
| rolltop desk | 12 | 8 | 5 |
| casement window | 6 | 7 | 6 |
8 of 14 prompts declared a coupling at all (98 of 176 built URDFs); across those the
median goes 6 → 3 driven joints, and 24 URDFs are fully one-input. (The absolute
counts in this table and this line are per URDF file, taken before coupling_stats.py
was made per-run on 2026-09-04 — each run holds the file three times, so they are about
3× the run counts; the medians and the 8-of-14 are unaffected. Not recomputed: the
corpus is not in the tree.) The mechanisms the support was built for — umbrella, step
ladder, folding workbench — are the ones that collapse to 1–2. The pre-mimic batteries
(aa_articulated, compare_art_v4_pf0) contain zero couplings, so nothing else on
the battery changed shape. The score effect on the three coupled prompts is inside the
noise band at n = 2 per side (umbrella 0.35/0.65 → 0.54/0.64, scissor 0.46/0.54 →
0.49/0.60, workbench 0.91/0.60 → 0.60/0.60); the pose count is the readout that
resolves.
Caveat on every judge- and gate-side number in this section (2026-09-04). Until the
fix in spatial/joints_export.robot_scene, the per-pose GLBs behind the articulation
sheet posed every <mimic> follower at rest — only fk (and so the collision sweep)
resolved the coupling. The mechanical counts above come from the URDFs and stand; the
score readouts and the joint_sweep gate rates below were produced on sheets where the
followers did not move, and need a re-run on the corrected export before either is cited
as evidence for or against <mimic>.
A battery built for the question (coupled_v1, ten one-input linkages, 2026-09-04)
separates what the planner does from what the agent does. Plan stage alone, 200 calls:
196 valid plans, zero planning losses, 193 declare a coupling; the ten
mechanisms each come back with the coupling declared, and the free joints left over are
the input itself (eight of ten) or a real second degree of freedom — the pram's four wheel
spins, the pantograph mirror's tilt. (An earlier reading of the same table treated
movable − mimic as undeclared couplings and reported a branching-coupling gap; joint by
joint there is none.) Then two
build-and-judge arms, baseline round only, differing by two edits — the mimic field
typed to None so the planner cannot fill it, and the paragraph asking for it removed
from codeverse3d/prompts/tracks/plan_articulated.j2 (bench/data/coupled/; the nomimic
umbrella cell scored nothing — no verdict in any round, the shape bench run now reports
as an error — so the score row has nine pairs, not ten):
| readout | planner may declare | planner may not |
|---|---|---|
| URDFs declaring a coupling | 10 / 10 | 3 / 10 |
| left with one degree of freedom | 8 / 10 | 3 / 10 |
joint_sweep rounds failed |
5 / 10 | 4 / 10 |
| ERROR findings per round | 2.60 | 2.30 |
| fixed-judge score (9 pairs) | 0.248 | 0.254, Δ −0.006 [−0.193, +0.181], unsupported |
Three things follow. The score cannot resolve it — nine pairs against an A/A band of
±0.13, exactly as §2 predicts, and the battery was not run for that. The mechanical
difference is structural. And the three couplings in the second arm are the agent's,
not the planner's: the field is gone, so it wrote <mimic> into robot.urdf by hand —
for the umbrella, the venetian blind and the step ladder, the three most textbook
mechanisms on the battery, and nothing else. Making the coupling a plan field is not
teaching a concept the model lacks; it turns something it volunteers in the obvious cases
into a question it answers every time.
Why the gate rate did not move. In the coupled arm every sampled pose is reachable by
construction — eight of ten models have exactly one free joint — so each overlap the gate
reports is one the mechanism can reach. Across both arms' 20 runs it raised 49 ERROR
findings over 47 distinct link pairs at a median 5.8 mm penetration, some already present
at rest; all of those are printed by bench/penetration_thresholds.py. The errors were
never mostly artefacts of unreachable poses: these mechanisms genuinely self-intersect, and
the defect the battery exposes is in generation rather than in the sampler or the gate.
Three caveats, two of them corrections (2026-09-06). Reachable means reachable in the
KINEMATIC MODEL; wrong limits still let a model reach what the object could not. "Every
sampled pose is reachable" holds for the eight one-DOF models — for the pram and the mirror
the sampler still combines the second freedom at random. And the sweep's tolerance is
2 mm, with a rest-pose ERROR line at 5 mm (sweep_collisions(tol_m=) and
sweep_findings(rest_max_m=), both defaults every caller takes); an earlier version of this
paragraph said 1 mm. The judge's "the deck does not rise" was cited here as corroboration
and has been removed: until joints_export.robot_scene resolves <mimic> followers, the
articulation sheet posed every follower at rest, so that complaint is evidence about the
sheet, not about the generation. The sweep poses through fk, which does resolve followers,
so the sweep-side evidence stands on its own.
What is NOT shown: that the sweep gate's failure rate moved. Two comparisons, both
null and both too small to be equivalence claims. On coupled_v1 the gate fails 5/10
rounds with the coupling declared against 4/10 without (Wilson 0.24–0.76 vs 0.17–0.69, a
hand count over the two arms' records). Across batteries, on the eight coupled prompts,
it fails 30 % of the rounds with couplings declared (11 of 37, wave2_lean) against 31 %
without (12 of 39, aa_articulated; ab_fewer_turns 19 % of 37, compare_art_v4_pf0
25 % of 8 — the spread across pre-mimic batteries is wider than the before/after
difference). Neither number is printed by a script: coupling_stats.py --per-prompt
reports the gate per battery, not per prompt subset, and the batteries are not in the
tree. The sampler drives fewer joints and every pose it drives is reachable, but the gate
still finds links overlapping in them — and the cross-battery comparison is confounded at
the source, since a pre-mimic URDF declares no couplings at all. What can be said is the
mechanical claim (which poses are sampled); "not detectably different at this n" is what
the gate data carries, and the claim that the gate's findings became truer is not
measured.
5.4 Provider-failure accounting
Three failure modes were reclassified from "the arm scored 0" to "the provider failed, drop
and redo" (D41): a Gemini stream exceeding its attempt budget, a 504 "Deadline expired",
and a finish_reason=PROHIBITED_CONTENT (the content filter tripping mid-JSON on a
furniture plan). Without this the batteries above would have carried zeros that measure the
weather. Related: gemini-cli gets a third rotated attempt on a quota 429 (one battery lost
4 of 10 cells to two per-minute 429s in a row while other keys were answering), and
render_glb retries once on a transient failure (two cells had lost their in-loop judge to
a render timeout under load).
6. Where the money goes (the cost model)
Measured over 133 recorded generation rounds, before the fix in §5.2: ~4.0 M input tokens per round (75 % cached), 62 k output + 36 k thoughts, 33 tool calls, ≈$1.39. At list prices the input side is ~68 % of the bill.
Per-turn telemetry from 1 063 in-process sessions (30 401 turns) gives
input(t) = 12 364 + 1 822·t and a session total ∝ n^1.60 — pure history re-send, not a few
huge tool results (the largest single per-turn increment ever recorded is 25 026 tokens; the
top 1 % of increments carry 7 % of appended tokens). Halving the turns by exact truncation
of those transcripts cuts tokens 68 % but dollars only 55 %, because the quadratic part
lands in the 10×-cheaper cache. That is why both turn/prompt-shape levers in §4 moved
tokens and not dollars, and why the §5.2 fix — which removes uncached tokens — is the one
that moved the bill.
7. The corpus the loop produces
3dcodeverse flywheel refine exports every refine round as the transition it was: the previous
round's judge issues and improvement plan, the gate ERROR findings with their fix hints, the
instruction lines build_refine_instructions compiled for the round (the task list, not the
per-group prompt text rendered from it), the unified src/ diff between the two
recorded commits, and the score delta with an improved/unchanged/regressed label. On the
recorded corpus, re-exported through the shipped CLI on 2026-09-03 (bench/out, 21
batteries): 254 rows from 87 runs across 13 batteries, the only drops being 54
cross-battery symlink aliases (dropped then as duplicate_run, a reason the exporter no longer
has — history, not a current code path), median diff 26.2 kB over 2 changed files,
112 improved / 81 unchanged / 59 regressed / 2 unscored, median gain on an improved
round +0.197, 119 rows carrying gate ERRORs. Both sides come from the round's own recorded
commit, not from HEAD. (The same export read 205 rows on 2026-09-02; the corpus grows
with every battery, so the selector is the date.)
finetune/llamafactory/build_refine_sft.py turns the improved transitions into the message
shape the finetune pipeline consumes: 112 samples (92 articulated, 20 static; median
answer 25.4 kB, median brief 28.2 kB), answered with the files the round produced in the
=== FILE: path === envelope. This is the one thing the harness produces that a one-shot corpus cannot: given
a judged, gated object and a list of what is wrong with it, write the corrected files.
The pre-existing flywheel/pairs.py (now addons/dataset/pairs.py) saw 57 % fewer transitions and carried neither the
instructions nor a diff.
8. Conclusions
- The harness's advantage over one-shot generation is real and resolvable: +0.146 on 40 static prompts, +0.152 to +0.33 on 14 articulated ones, with judge-free corroboration (gate errors 1.5 vs 17.3; build rate 100 % vs 50 %).
- A 12–14-prompt battery cannot resolve a single-switch change. The A/A puts the noise floor at ±0.13 with a paired sd of 0.225 and asks for ~506 pairs to resolve ±0.02. Four plausible levers landed inside that band; none of them is thereby shown to do nothing.
- The changes that mattered were found by counting loss events, not by scoring artefacts. Planner mortality 4.7 % → 0.7 % (p = 0.0067); retried tool calls 26.6 % → 1.3 %; uncached tokens −65 %; dollars per round −40 %; driven joints per sampled pose on a coupled mechanism 6 → 3 (1 on the umbrella and the step ladder). Every one of these is invisible to the judge mean at this n.
- The biggest single defect was in the harness's own plumbing, not in the model or the prompt: one line mapping a tool's verdict to a protocol error cost a quarter of all tool calls to retries and two thirds of the uncached token bill. It was found by reading recorded tool statistics against the harness's own gate outcomes.
- Cost intuitions about agent loops need the cache term. Tokens grow ∝ n^1.6 in turns while dollars grow ∝ n^1.2–1.3; cutting turns or trimming prompts moves the first and not the second. Removing uncacheable payload moves the bill.
- Secondary, uncontrolled: the paired sd of a battery on the fixed code is 0.131 against the A/A's 0.225 on the old code (~172 vs ~506 pairs for ±0.02). Different runs and different arms, so it is an observation — but a retry storm is variance, and the direction is the one the fix predicts.
9. What a paper can claim from this, and what it cannot
Can:
- The harness beats one-shot and one-shot+repair on both tracks, with paired CIs and a judge-free corroboration that does not depend on the VLM.
- A measured account of why an agentic 3D loop is expensive (history re-send, cache behaviour, tool-call retries) with per-request and per-round numbers.
- A significant, cheap fix to planner mortality, and the method that made it measurable: isolate the stage, measure the loss event, run n in the hundreds.
- Coupled-mechanism support with a mechanism-level readout (poses per mechanism), not a score claim.
- An honest instrument section: the A/A calibration, the n_for_power arithmetic, and four null results reported as null.
Cannot:
- Attribute any of the four §4 levers a score effect.
- Credit the §5.2 fix with a score improvement — it is a cost and a retry result; the score on the same battery moved inside noise.
- Compare across the api-agent → gemini-cli generator change (the static regression in the wave-2 battery is confounded by exactly that, plus ~140 upstream commits and a rubric wording change), or across judge rubric revisions.
- Treat the mimic score deltas as evidence: n = 2 per side.
9.1 Claim → evidence index
Four of these are recomputed by a script in the repo, so a reviewer can re-derive the
number rather than trust the row: bench/plan_stage_report.py (planner mortality, the failure-class split and a Fisher
p for every pair of arms, over the committed bench/data/plan_stage/*.jsonl), bench/session_stats.py
(tool-error, cache and dollar columns), bench/coupling_stats.py (the mechanism counts)
and bench/paired_compare.py (every paired score row). The rest — the base64
amplification arithmetic and the in-process token-growth model — were measured once, on
data the repo does not carry; they are marked in the caveat column.
| claim | number | where the data is | caveat |
|---|---|---|---|
| harness > one-shot+repair, static | +0.146 [+0.077, +0.215], n=40 | bench/out/compare_v4_calm |
calm provider window; the storm run of the same battery gives +0.028 with a CI crossing 0 |
| harness > one-shot, static | +0.401 [+0.297, +0.505] | same | — |
| harness > one-shot+repair, articulated | +0.152 [+0.061, +0.243], n=14 | compare_art_v3 |
after the planner repair; before it, parity |
| harness > one-shot+repair, articulated, three arms | +0.19 … +0.33 | compare_art_v4_{pf,pf0,pp} merged with compare_art_v2 one-shot rows |
one-shot rows come from an earlier run of the same battery/judge |
| judge-free advantage | 1.5 vs 17.3 gate errors; 100 % vs 50 % build | compare_v4_calm |
— |
| noise floor of the battery | paired sd 0.225, 2 SE ±0.130, ~506 pairs for ±0.02 | aa_articulated |
identical arms |
| geometry re-ask | +0.064 ±0.25, fired 7/14 | compare_art_v4_pf vs pf0 |
— |
| pro vs flash planner | +0.090 ±0.23 | compare_art_v4_pp vs pf |
5/14 pro cells lost to provider limits |
| deterministic repairs | −0.039 ±0.105, fired 1/12 | ab_repairs |
plan pinned |
| fewer turns | +0.015 ±0.126, −12 % tool calls, $/cell flat | ab_fewer_turns |
plan pinned |
| lean prompt | +0.030 ±0.076, $3.51 vs $3.66 | wave2_lean |
plan pinned |
| planner mortality | 4.7 % → 0.7 %, Fisher p = 0.0067, n = 560 calls | bench/data/plan_stage/restart_{on,off}.jsonl (in the repo) |
plan stage only, both arms same window |
| tool calls reported as errors | 26.6 % → 1.3 % | aa_articulated vs wave2_lean stats envelopes |
same battery/config, different weather |
| cache hit | 69 % → 90 % | same | same |
| uncached tokens per request | 38 521 → 13 480 (main-role) | same | same |
| generator $ per round | median 1.572 → 0.950 | round records of both runs | same |
| base64 amplification | 261 k tokens vs 516 for one 275 kB sheet | measured on a recorded blob + vendor bundle | arithmetic + matching recorded prompt jumps; no repo script |
| token growth in turns | input(t) = 12 364 + 1 822·t; total ∝ n^1.60; dollars ∝ n^1.19–1.34 | 1 063 in-process sessions, 30 401 turns | no repo script — the in-process agent, since gemini-cli emits no per-turn usage |
| coupled mechanisms declared | 8 of 14 prompts; driven joints per pose 6 → 3 (1–2 on the one-input mechanisms) | bench/coupling_stats.py bench/out/wave2_lean --per-prompt |
recomputed from the recorded URDFs |
| refine corpus | 254 transitions, 112 SFT samples (2026-09-03) | 3dcodeverse flywheel refine, finetune/llamafactory/build_refine_sft.py over bench/out |
corpus grows with every battery |
9.2 Open questions
Everything in §4 is unresolved, not refuted. Resolving ±0.02 on this battery needs ~172–506 paired prompts depending on the code; a paper that wants those effects has to pay for the prompts or find a judge-free readout for each.
The judge is one model. No cross-judge check has been run (it needs a non-Gemini key).
judge_prompt_hashmakes the comparison auditable but does not remove the dependence.The mimic result is mechanical, not scored.Answered 2026-09-04:coupled_v1is that battery, and the score still cannot resolve it (9 pairs, Δ −0.006, CI ±0.19). What the battery did produce is the planner/agent split in §5.3 — the agent supplies a coupling for 3 of 10 one-input mechanisms on its own, the planner for 10 of 10.The static regression in wave 2 is confounded by the api-agent → gemini-cli change; a clean static before/after on one generator has not been run.
The refine corpus has not been trained on. 112 samples is a probe, not a training set; the interesting experiment is whether a model finetuned on refine transitions repairs better than one finetuned on one-shot pairs.
Planner mortality is now 0.7 %, not 0. The residue is two calls in 276 that still failed after a restart; both died at the validation cap, which the 2026-09-03 change (a restart no longer spends a re-ask slot) addresses without being measured yet.
What kills a plan call now is
parent == child, not a collapsed plan. The 2026-09-03 three-arm run leaves that class at 1 / 2 / 4 across the arms with nothing acting on it (§5.1). It never restarts and it survives all three re-asks — the same shape the collapsed-plan class had before it was handled, and the obvious next measurement.The scene track's recorded numbers are not usable, and the first battery says why. The paper's "does not yet work" rests on a loop value of −0.009 at n = 4 whose records are gone from both clones — no script in the tree recomputes it. The first battery that CAN be recomputed (
bench/out/scene_baseline, 12 prompts, 2026-09-05) returned no verdict for four of its first ten cells, and every one of those four is the machine: three lost the render tab to Chrome under memory pressure, and the fourth, recorded as[?] scene did not bootthrough three repair attempts, has its own round artifact readingok: true, boot.ok: true, stage: readyand boots today unchanged in 600 ms. Worse for a quality claim, the instrument was manufacturing defects: 15 of 19 recordedscene_placementERRORs were volumetric passes counted as solid matter, the judge quotes gate findings back as defects (3 of 24 issues, one at critical), and a false "camera in geometry" retreats the authored lens — cozy_cabin's two cameras were moved back 4 m for standing in a moonbeam, which is what took their frames from mean luminance 0.263 / 0.298 down to 0.126 / 0.149 and intodark_frameERROR. Since "the frame is too dark" is this track's most-cited defect (153 findings over 32 runs), some share of that history is the instrument; the share is unknowable because those records are gone, so no number should be put on it. The fixes are onziyao/scene-placement(D53–D58); arm B (local/scripts/run_scene_fixed.sh) is the measurement, and until it lands the paper should not cite a scene result. Handover note:local/docs/PAPER-scene-track-correction.md.A loss rate is a property of its window. The same control arm measured 4.7 % on 2026-09-02/03 and 2.9 % on 2026-09-03; single-arm comparisons across windows are not safe, which is why the re-measurement ran all three arms at once.
9.3 The switches this work added, and their state
Every one is a Settings field, and tracks/plan_features derives from those fields which
names are live (since 2026-09-22; before, a hand-kept registry and a test grepping the tree); a
switch nothing reads once produced "keep, mean delta +0.344" on two byte-identical arms, which
is why ab_plan refuses one.
| switch | what it does | default | why |
|---|---|---|---|
C3D_PLAN_GEOMETRY |
plan-time geometry re-ask (attachment gap, hinge pivot, swept collision) | off | +0.064 ±0.25, no measurable gain (§4); removed from the tree 2026-09-21 (D49) |
C3D_PLAN_RESTART |
re-sample a collapsed plan (one top-level part and dangling links) from the original request | on, kill switch | 4.7 % → 0.7 % planner mortality, p = 0.0067 (§5.1); narrowed 2026-09-03 and re-measured three-arm: dangling-link deaths 3/140 off vs 0/276, p = 0.038 (exploratory, after three null overall-rate tests — §5.1) |
C3D_AXIS_REPAIR (upstream, tracks/articulated_object.py) |
axis flip on a reversed joint + buried-link check | on (upstream default; the A/B ran it against off) | −0.039 ±0.105, fired 1/12 (§4) |
C3D_LEAN_PROMPT |
drop duplicated contract/tool cards, select cookbook chapters, focus the refine prompt | off | +0.030 ±0.076, no cost saving (§4); removed from the tree 2026-09-03; last carried on ziyao/articulated-wave-2 before commit 4cbb28c. |
C3D_FEWER_TURNS (pre-existing) |
fold gate checks into build, inline refine files | off | +0.015 ±0.126, dollars flat (§4) |
C3D_AXIS_REPAIR (upstream) |
deterministic axis rewrite from the measured motion | on | upstream's, kept |
C3D_SCENE_TEXTURES |
the scene texture pack stage before env/zones | off | never measured; removed from the tree 2026-11-03 with all image generation (owner: not a feature for now) |
Not switched, because they are bug fixes rather than levers: the MCP verdict/failed split
(§5.2), <mimic> support (§5.3), the provider-failure markers and retries (§5.4), the
truncation and degenerate-plan re-asks around the planner (§5.1).
10. Reproduction
cd eval # the bench scripts live here since 2026-09-22 (they were harness/bench/)
# the measurements above ran on `gemini-cli:*`, deleted in D103 (2026-10-13); its successor for
# Google's models is `agy:*`, so a rerun is a new measurement, not a replay
# the instrument
python bench/ab_plan.py --prompts bench/prompts/articulated_v2.yaml --aa --pin-plan \
--generator agy:gemini-4-argon --judge gemini:gemini-4-argon \
--n-samples 3 --rounds 3 --out bench/out/aa_articulated
# a switch A/B (generation-side switches only; plan-side is refused for --pin-plan). The arm
# measured in §4 was C3D_FEWER_TURNS=1, a switch deleted since; name a live Settings flag instead
python bench/ab_plan.py --prompts bench/prompts/articulated_v2.yaml \
--variant-env C3D_<SWITCH>=1 --pin-plan --out bench/out/<switch>
# harness vs one-shot
python bench/compare_backends.py --prompts bench/prompts/articulated_v2.yaml \
--arms harness:agy:gemini-4-argon,oneshot+repair:gemini:gemini-3.7-flash \
--judge gemini:gemini-4-argon --judge-samples 3 --out bench/out/art
# the loss-event channel: the plan stage alone, one row per call
python bench/plan_stage_bench.py --tree ../harness --label restart_on --reps 20 \
--out bench/data/plan_stage/restart_on.jsonl --env C3D_PLAN_RESTART=1
python bench/plan_stage_report.py bench/data/plan_stage/*.jsonl
# the corpus the loop produces
3dcodeverse flywheel refine bench/out refine.jsonl --with-code
python ../finetune/llamafactory/build_refine_sft.py refine.jsonl --out refine_sft.jsonl # finetune/ is at the repo root
Recorded runs referenced above (not committed): bench/out/{compare_v4_calm, compare_art_v2, compare_art_v3, compare_art_v4_pf, compare_art_v4_pf0, compare_art_v4_pp, ab_repairs, ab_fewer_turns, aa_articulated, wave2_lean, wave2_static}.