Skip to content

Official Codebase of 3DCodeVerse

Python

docs: fixed rounds, every round kept, the pick after the run

gaoypeng00352bd316aa
diff --git a/eval/docs/EVAL.md b/eval/docs/EVAL.md
index 7152e85f..4b9b4b60 100644
--- a/eval/docs/EVAL.md
+++ b/eval/docs/EVAL.md
@@ -63,17 +63,22 @@ deterministic contract check.
3dcode bench run bench/prompts/static_objects_v1.yaml --generator gemini-cli:gemini-3.7-flash --judge gemini:gemini-3.1-pro-preview --out bench/out/static_v1_gemcli
3dcode bench report bench/out/static_v1_apiagent
```
-Each item = one full `3dcode make` run (its own workspace under `--out`); `results.jsonl`
-rows carry `score_baseline`, `score_final`, `passed`, `rounds`, `cost_usd`, `minutes`,
-`status`, `generator`, `judge`. `report.md/html` tabulates per tier and category:
-mean baseline → final (the harness delta), pass rate, cost, time. Keep `--judge` fixed
-across arms; vary only `--generator` (and `--planner` if that is the variable).
-Resumable: re-running the same `--out` skips finished ids.
-
-Reading the numbers: *baseline* (round 0) is the raw-model-with-contract result;
-*final − baseline* is what the refine loop buys; the pass rate at the rubric threshold
-(0.72 for `static_object_v1`) is the headline. Observed so far (flash judge,
-n=1, single runs): chair 0.67 → 0.74, bench 0.64 → 0.89, cabinet 0.68 → 0.93,
+Each item = one full `3dcode make` run (its own workspace under `--out`) followed by the
+hand-over `make` does — `addons/select.pick` by score, then `package` (deliverable/ +
+selection.json); `results.jsonl` rows carry `score_baseline`, `score_picked` (rows written
+before 2026-09-22 say `score_final`), `picked_round`, `rounds`, `cost_usd`, `minutes`,
+`status` (the stop reason), `generator`, `judge`. No `passed` column: since 2026-09-22 a run
+is not passed or failed. `report.md/html` tabulates per tier and category: mean baseline →
+picked (the harness delta), cost, time. Keep `--judge` fixed across arms; vary only
+`--generator` (and `--planner` if that is the variable). Resumable: re-running the same
+`--out` skips finished ids.
+
+Reading the numbers: *baseline* (round 0) is the raw-model-with-contract result; *picked −
+baseline* is what the refine loop buys — but the picked round is the best-scored of the
+baseline + `--rounds` refine rounds, so that delta is ≥ 0 by construction and carries the
+judge's noise as a maximum over n draws: compare arms on the picked score itself, at the
+same `--rounds`. Observed before 2026-09-22 (flash judge, n=1, single runs, runs that
+stopped on a pass): chair 0.67 → 0.74, bench 0.64 → 0.89, cabinet 0.68 → 0.93,
scene 0.56 → 0.58 after one refine.
## 4. Harness vs raw one-shot (`bench/compare_backends.py`)
@@ -159,8 +164,8 @@ Pending: rerun the parked sol battery, then terra, then luna, then the claude-co
## 5. Comparing backends inside the harness
Same battery, same `--judge`, different `--generator` (`api-agent:*`, `single-shot:*`,
-`gemini-cli:*`, `claude-code:*`, `codex:*`, `agy:*`). Report baseline, final, delta,
-pass rate, cost, time per tier; optionally feed the best rounds of two arms through
+`gemini-cli:*`, `claude-code:*`, `codex:*`, `agy:*`). Report baseline, picked, delta,
+cost, time per tier; optionally feed the picked rounds of two arms through
`PairwiseJudge.compare(spec, renders_a, renders_b)` (`flywheel pairs` already emits
cross-backend candidate pairs keyed by prompt hash).
@@ -638,7 +643,8 @@ Until this date a profile that injected a `judge_samples>1` policy (economy, qua
silently zeroed the static track's surface-detail round while balanced kept it; all
three profiles now get it, and `C3D_DETAIL_ROUNDS` remains the A/B switch. Any
economy/quality-vs-balanced comparison straddling this commit compares different
-round counts.
+round counts. (2026-09-22: the detail round is gone with the other judgement-driven rounds;
+`C3D_DETAIL_ROUNDS` is a dead switch, and every run is the baseline + `--rounds`.)
**2026-08-29 — per-language system prompts: NULL, three independent A/Bs.** The
one-line system prompts were replaced with evidence-grounded ones mined from each
diff --git a/eval/docs/PAPER_WRITING.md b/eval/docs/PAPER_WRITING.md
index cc193e4a..5578fa62 100644
--- a/eval/docs/PAPER_WRITING.md
+++ b/eval/docs/PAPER_WRITING.md
@@ -25,8 +25,8 @@ format).
## 0. The system, in one paragraph
A request becomes an iterated loop: **plan → generate code → build → deterministic gates →
-render → VLM judge → refine**, for N rounds under a wall-clock and dollar budget, delivering
-the best round and recording every round. The generator is a coding agent writing real
+render → VLM judge → refine**, for a fixed N rounds under a wall-clock budget, recording every
+round and handing over the best-scored one after the run. The generator is a coding agent writing real
source (Blender `bpy`, CadQuery, three.js, GLSL, or a URDF robot plus a `bpy` link builder);
the gates are program analysis, not model calls (lint, connectivity, dimensional contract,
and — on the articulated track — a joint sweep that collides every link pair in every sampled
diff --git a/harness/CLAUDE.md b/harness/CLAUDE.md
index 966639bc..ed127460 100644
--- a/harness/CLAUDE.md
+++ b/harness/CLAUDE.md
@@ -89,6 +89,8 @@ cd /home/yipeng/3dcodeverse/harness
3dcodeverse make "..." --track scene --language scene_threejs --rounds 2 --max-minutes 60
3dcodeverse make "neon rain on a window" --track graphics --language glsl_shader
3dcodeverse make "..." --image ref.png --candidates 3 --rounds 2 --dim height=0.45 --must "three legs" --texture --no-run
+3dcodeverse make "..." --no-pick # a run = baseline + --rounds refine rounds (fixed); make then picks a round → deliverable/ unless --no-pick
+3dcode pick <slug> [--by score|pairwise] [--round N] [--texture] [--judge MODEL] # hand over a round (codeverse3d/addons/select.py)
3dcode resume <slug> · 3dcode status <slug> · 3dcode render <slug> [--mode wire] · 3dcode judge <slug> [--model ... --n 3]
3dcode texture pass <slug> [--no-judge] · 3dcode texture scene-pack <slug> · 3dcode texture show <slug>
3dcode tools list · 3dcode tools measure --workspace runs/<slug> · 3dcode mcp --workspace runs/<slug>
diff --git a/harness/docs/ARCHITECTURE.md b/harness/docs/ARCHITECTURE.md
index b8512b53..ff7adb96 100644
--- a/harness/docs/ARCHITECTURE.md
+++ b/harness/docs/ARCHITECTURE.md
@@ -31,15 +31,18 @@ layered on top of the language — for four **tracks**:
of files, no tools) handled inside `tracks/generation.py`.
Every run is: **spec → plan → skeleton → [scene stages] → baseline round
-(optionally best-of-N candidates) → refine rounds → finalise → record**, with an
-optional post-hoc **texture pass**. The record (plus git history of `src/`) is
-the flywheel unit.
+(optionally best-of-N candidates) → `max_rounds` refine rounds → finalise → record**.
+Which round to hand over is decided AFTER the run, by the `addons/select` reader
+(`3dcode pick`; `3dcode make` calls it by score): it writes `deliverable/` for that
+round and runs the optional **texture pass** on it. The record (plus git history of
+`src/` and every round's kept build) is the flywheel unit.
## 1. Design laws (from the reference post-mortems)
1. **Code is truth; artifacts are derived.** Raw code in `src/` is the
- deliverable; GLB/renders are regenerated from it (finalise restores the best
- commit and rebuilds). Textures are a *derived asset pack*
+ deliverable; GLB/renders are regenerated from it. Every round is a commit and
+ keeps its own built files under `artifacts/rNN/`, so any round can be handed over
+ without a rebuild; the workspace ends at the last round. Textures are a *derived asset pack*
(`object_textured.glb` + `artifacts/textures/`); `object.glb` is never touched.
2. **Concrete beats abstract.** Contracts are delivered as skeleton code, exact
numbers and copyable snippets (cookbooks), never as clauses.
@@ -167,19 +170,19 @@ codeverse3d/
(world-metre unwrap, PBR map set, application + material normalisation),
materials.py (named material library), run.py (texture_pass); __init__.py is
docstring-only — import from the submodules
- orchestrator.py the round loop's LIBRARY, not the loop: round POLICIES (RoundPolicy /
- StopPolicy / pick_best_round), refine-task compilation + grouping,
+ orchestrator.py the round loop's LIBRARY, not the loop: the round knobs (RoundPolicy —
+ no stop knob besides max_rounds), refine-task compilation + grouping,
StageRunner + RunState (resume), BudgetGuard. The loop itself is
tracks/lifecycle.py:_round_loop → tracks/steps.py:run_round. A best-of-N
candidate IS steps._run_round(kind='candidate') in a _cand/c<k> sub-workspace
with two knobs (render=candidates.quick_render, geometry_views=False), its
own _cand/c<k>/events.jsonl and a one-sample judge
tracks/ __init__.py (get_track(track, **options) → a lifecycle.BaseTrack),
- lifecycle.py, steps.py, candidates.py (best-of-N + the pure candidate/pairwise
- decision logic), generation.py (agent + single-shot strategies + the file
+ lifecycle.py, steps.py, candidates.py (best-of-N baseline candidates and
+ their ranking), generation.py (agent + single-shot strategies + the file
envelope), repair.py, planner.py, prompting.py (prompt helpers, split from common),
- common.py (RunContext, Services), static_object.py (+ the detail round and the
- reference-image gates), articulated_object.py (+ the planned-motion gate),
+ common.py (RunContext, Services), static_object.py (+ the reference-image
+ gates), articulated_object.py (+ the planned-motion gate),
scene.py, scene_assets.py (+ cheap single-shot asset generation),
zone_layout.py (L2 zone director: per-zone structured layout calls + deterministic validator),
graphics.py (the whole graphics track: planner hooks, prompt context, frame
@@ -195,7 +198,9 @@ codeverse3d/
skills_hook.py (the round's view of codeverse3d/skills: attach before generating,
probe reads after — a no-op under C3D_SKILLS=0; on by default since 2026-09-22)
record/ what every run WRITES: record.py (finalize_record, load_record, iter_runs,
- best_round_record), deliverable.py, telemetry.py, _git.py (files at a round's commit)
+ effective_judgment), deliverable.py (every round's kept build under
+ artifacts/rNN/, and deliverable/ for ONE round — built by addons/select),
+ telemetry.py, _git.py (files at a round's commit)
addons/ optional tools that READ finished runs; outside cli/ nothing imports them
(tests/core/test_addons_boundary.py). dataset/ = export.py, pack.py, pairs.py,
refine.py, captions.py, index.py, sample.py, quality.py (tiers + code fingerprints +
@@ -217,7 +222,7 @@ codeverse3d/
prompts/ EVERY piece of prompt material the harness writes, and the only place it
lives: <lang>/{system,contract,cookbook}.md (incl. glsl_shader/,
opengl_python/), system/* (harness contract, single-shot envelope,
- role_{scope,detail,repair}.j2), texturing/*.md, tracks/*.j2.
+ role_{scope,asset,repair}.j2), texturing/*.md, tracks/*.j2.
catalog.py answers "what exists and how does each piece reach the model"
— including the ONE language-id → prompts/<dir> mapping, which used to be
copied four times and missing in a fifth. sections.py splits that
@@ -268,6 +273,8 @@ tests/ agents blender_cadquery core cost flywheel_cli gallery gra
```
runs/<slug>/
spec.json plan.json run_state.json record.json events.jsonl
+ selection.json (after a pick) the round deliverable/ holds, how it was chosen, every round's score
+ deliverable/ (after a pick) that round's code + artifacts + sheet + manifest.json (RUN_LAYOUT.md)
src/ agent-authored RAW code (git repo; commits: spec, skeleton, pre:/agent:<label>, rNN <kind>)
public/ (scene) compiled assets public/assets/<snake>.glb; (textured scenes) public/textures/*.png + manifest.json
_assets/<snake>/ (scene) sub-workspaces for blender_glb assets (gitignored)
@@ -281,11 +288,14 @@ runs/<slug>/
alias; runs before 2026-08-23 have the root file only)
run_state.json stages + rounds done; extra carries budget_snapshot and spec_fingerprint only
artifacts/ object.glb object.stl|step robot.urdf meshes/ articulation.json build.json census.json
- measurement.json … ; graphics: frames/fNN_tT.png frames_sheet.png preview.gif metrics.json
- texturing: object_textured.glb textures/{<id>.png, texture_plan.json, texturing.json, gate/}
+ measurement.json … — the LAST round's build; graphics: frames/fNN_tT.png frames_sheet.png
+ preview.gif metrics.json; texturing (a pick's --texture): object_textured.glb
+ textures/{<id>.png, texture_plan.json, texturing.json, gate/}
+ rNN/ the round's own build, copied when it built: object.glb object.stl|step robot.urdf
+ meshes/*.glb preview.gif frames_sheet.png (only what a hand-over needs)
renders/rNN/ view_<name>.png sheet.png views.json (judge flags) (+ poses/ articulated; <cam>_t<t>.png metrics.json scenes)
gates/rNN/ lint_<lang>.json connectivity.json contract.json joint_sweep.json motion_direction.json … (+ *_tool.json)
- judge/rNN.json (+ rNN_cli.json from `3dcode judge`)
+ judge/rNN.json (+ rNN_cli.json from `3dcode judge`; rAA_vs_rBB_pairwise.json from `3dcode pick --by pairwise`)
tool_renders/rNN_<hash>/
trajectories/<label>_rNN/ prompt.md transcript.jsonl stdout.json stderr.log result.json
(a retried label lands in <label>.a2_rNN — first attempt preserved)
@@ -408,7 +418,7 @@ citation auditable. Floors,
deterministic caps from gate findings (`data["kind"]`), console errors, missing
must-acceptance and
`missing_views` rules apply on top; degraded verdicts are glitches, not scores.
-`PairwiseJudge` (position-swapped, tie on disagreement); `ReferenceJudge` for
+`PairwiseJudge` (position-swapped, tie on disagreement; only `3dcode pick --by pairwise` asks it); `ReferenceJudge` for
image-conditioned specs.
Rubrics: `static_object_v1` (0.72), `articulated_v1` (requires pose sheet),
`scene_v1` (frame-gate caps), `asset_v1`, `reference_v1`, `shader_v1` (0.70).
@@ -435,14 +445,11 @@ round 0 "baseline": generate → build_with_repair → measure → gates → ren
session that owns the entry file, placement and the connectivity/contract gates. $C3D_SCOPED_PARTS=off
restores the single whole-object session; single-shot always uses it.)
(--candidates N: N parallel baselines in <ws>/_cand/c<k>, quick 4-view judge, crashed candidate retried once,
- selection build_ok → quick score → fewer gate errors with pairwise tie-break; winner copied back, normal r00 follows)
-repeat while StopPolicy says continue (≤ max_rounds refine rounds, plateau_window=2 / min_delta=0.02,
- target = rubric threshold, budget ok):
- money stops, sized by the judge's MEASURED noise σ (cost/routing.JUDGE_NOISE: pro 0.030, flash 0.083):
- regression — last round scored < best − 1σ ⇒ never another round of the same shape: the first one
- switches strategy (ONE whole-artifact rewrite, kind "rewrite"), a second stops the run
- diminishing_returns — from r03 on, only start when the last gain > 1.5σ AND best < target
- refine tasks = gate ERRORS (fix hints, authoring-frame numbers) ∪ failed must-acceptance ∪ judge improvement plan
+ selection build_ok → quick score → fewer gate errors → the earlier candidate; winner copied back, normal r00 follows)
+then max_rounds refine rounds, EACH built on the round before it — FIXED rounds (owner, 2026-09-22): no pass,
+plateau, regression or diminishing-returns stop, no rewrite or surface-detail round, no refine-from-best:
+ refine tasks = the previous round's gate ERRORS (fix hints, authoring-frame numbers) ∪ failed must-acceptance
+ ∪ its judge improvement plan
(≤ 6 tasks, ≤ 6 compacted instruction lines each; reference runs add an IoU task when silhouette IoU < 0.6)
fan out when ≥ 2 file-disjoint groups AND every task maps to files (threejs/blender parts, scene zones/assets/env)
generate (no HARNESS turn cap by default — claude-code runs under AgentJob.max_turns=60 (+6-turn wrap-up),
@@ -450,19 +457,18 @@ repeat while StopPolicy says continue (≤ max_rounds refine rounds, plateau_win
a cap the machine sets (C3D_AGENT_MAX_TURNS / limits.agent_max_turns; no profile sets one) still buys a wrap-up session
that lands a final build + summary instead of being killed) → build+repair (error-focused,
escalates on identical signatures) → gates → … → judge (SKIPPED only where the verdict is never
- bought at all: no judge/renders or budget already exceeded)
- pick_best_round: highest score, tie → fewer gate errors; |Δ| < pairwise_margin (0.03) → position-swapped
- PairwiseJudge decides (replace only at confidence ≥ 0.6; note persisted in rNN.json)
+ bought at all: no judge or no renders — a round that finishes past the clock is still judged)
+ → commit src/ (the round's commit) and copy its build to artifacts/rNN/
+ a round left without a verdict (judge outage / degraded) is re-judged once before the next is planned
every round emits cost.round {stage → $, judge $, agent turns, wasted flag}; a round that raises mid-way
still reports what it burned (rounds/aborted_rNN.json + record.extra["aborted_rounds"])
- a plateau / diminishing_returns stop on a CLEAN artifact (built, judged, no gate ERROR, score ≥ 0.45 and
- within σ of the best) is converted into ONE round of kind "detail" instead: surface detail only — bevels,
- panel lines, fasteners, wear, material variation — with the silhouette, placement and part list frozen and
- a deterministic `detail_drift` gate that ERRORs if any part box moved > 5 mm. Measured on 88 refine-round
- pairs: the part count never changed once and mean Δgeometry_detail was +0.003, so detail needed its own
- round; the rounds that added geometry DURING repair lost 0.075 assembly_fit. $C3D_DETAIL_ROUNDS=0 is off.
-stop reasons: pass | plateau | budget | max_rounds | no_change | no_refine_tasks | regression | diminishing_returns
-finalise: restore best commit, rebuild so artifacts match delivered code, finalize_record → record.json
+stop reasons: max_rounds | budget (the wall clock — BudgetGuard / max_minutes) | agent_quota | no_change (a refine
+ round's sessions changed nothing) | no_refine_tasks | judge_unavailable | failed
+ (a record from before 2026-09-22 may say passed / plateau: it loads as `stopped`)
+finalise: the workspace ENDS AT THE LAST ROUND — src/ a cut round left dirty is put back and that round rebuilt;
+ finalize_record → record.json. No best round, no deliverable/, no texture pass in the run.
+after the run (`3dcode make`, unless --no-pick): addons/select.pick(by="score") — highest effective score, ties →
+ fewer gate errors → the earlier round — then package(): deliverable/ + selection.json (+ --texture)
```
Track-specific gates: static `connectivity` + `contract` (+ `reference_silhouette`),
articulated + `joint_sweep` + `motion_direction` (URDF axis vs plan motion text),
@@ -484,7 +490,10 @@ a discarded load does not), graphics `gl_frames` (NaN/black/blown/static/flicker
## 8. Texturing (derived asset pack)
-`texturing.run.texture_pass(ws, spec, plan, model_id=…)` — after (or outside) a run:
+`texturing.run.texture_pass(ws, spec, plan, model_id=…)` — never inside a run: `3dcode pick
+--texture` (and `make --texture`, which picks after the run) texture the PICKED round's kept
+`artifacts/rNN/object.glb`; a standalone `3dcode texture pass` textures the canonical one, i.e.
+the last round:
one VLM **material plan** (parts → shared texture ids, family, projection,
`tile_size_m`; cached) → text-to-image tiles (`GeminiImageModel`,
gemini-3.1-flash-image; mirror cross-fade makes them tileable, `seam_score ≤ 0.08`
@@ -526,7 +535,7 @@ $0.02–0.03 (flash) / ~$0.2 (pro); api-agent generation 3–6 min per object ro
* Scene judging is deliberately narrow: the judge sees the scene's OWN cameras (first,
and the detail crop is taken from one of them) plus at most three overview-rig tiles;
the harness's eye-level rig is a diagnostic only. A `must` acceptance item the harness
- cannot verify caps a run at 0.60 AND fails it, so on the scene track only the spec's
+ cannot verify caps a round at 0.60 AND fails its verdict, so on the scene track only the spec's
`must_have` list keeps that priority (the plan's own checklist is `should`).
* Articulated: candidate selection uses the quick 4-view sheet (not pose views);
mimic joints are honoured (the sweep drives independent joints only and resolves
@@ -536,8 +545,8 @@ $0.02–0.03 (flash) / ~$0.2 (pro); api-agent generation 3–6 min per object ro
derived pack); `userData.tick` cannot survive export.
* agy exposes no per-workspace MCP, cost or served model.
* Anthropic / OpenAI backends are mock-tested only (no keys on this box).
-* Budget checks run between steps: a refine round that finishes its judge and then
- trips the budget is not promoted to best — give scenes `--max-minutes 60`.
+* Budget checks run between steps: a round the clock cuts mid-way is rolled back (its
+ spend stays in `aborted_rounds`) — give scenes `--max-minutes 60`.
* Gemini flash 503 storms happen; dead keys and 429s rotate freely now, but a
sustained outage can still fail a round (`3dcode resume` re-uses cached stages).
* A few single-file wrappers were once over the old ~400-line guideline; the rule
@@ -547,8 +556,9 @@ $0.02–0.03 (flash) / ~$0.2 (pro); api-agent generation 3–6 min per object ro
`record.json` per run + git history; `3dcode flywheel export` writes sample folders
(`<out>/<track>/<language>/<slug>/{code.<ext>, src/**, robot.urdf, meta.json,
-captions.json, renders/}`) with **quality tiers** (A passed & 0 gate errors, B
-passed, C best ≥ 0.6, D else), acceptance checklists, gate summaries and
+captions.json, renders/}`) of the PICKED round (`addons/select`; the last built round
+when none was judged) with **quality tiers** from that round's verdict (A passed & 0 gate
+errors, B passed, C score ≥ 0.6, D else), acceptance checklists, gate summaries and
`(code fingerprint, prompt)` dedupe (`--drop-duplicates`; side-car captions via
`--captions-dir`); `metadata.parquet`/`.jsonl` + sqlite index carry tier, gate
errors, cost, fingerprints, `duplicate_of`; `--pack` tars with byte-range locators.
diff --git a/harness/docs/COST.md b/harness/docs/COST.md
index 69c89638..23dc8825 100644
--- a/harness/docs/COST.md
+++ b/harness/docs/COST.md
@@ -45,6 +45,10 @@ superseded compare battery) is excluded.
| recorded vs measured | `record.total_usage` says $85.08; the ledger finds **$86.30** (§6) |
| re-priced with the corrected table | **$89.10** (+3.2%, §7) |
+(The pass count and the $ per passing artifact are this audit's. Since 2026-09-22 a run is
+not passed or failed — it runs its fixed rounds and a pick hands one over — so `3dcode cost`
+reports $ per run and no longer counts passes.)
+
**The money is generation, not judging.** Nine dollars in ten are the coding
agent writing and rewriting code; the judge is a rounding error next to it
(7.2%), the planner is noise (0.8%). Any cost programme that starts with "use a
@@ -189,6 +193,15 @@ model — **88% of the clock is waiting for a model**, 12% is the harness
| repair loop that never converged (run did not pass) | 4 | $1.31 | 1.5% |
| **total** | 59 | **$27.35** | **31.7%** |
+> **2026-09-22 — three of these kinds are gone.** "Regression", "zero-delta round" and
+> "unpromoted judge" measured money against the IN-RUN best round, and the owner removed
+> the in-run best with the judgement stops: a run is now the baseline + `--rounds` refine
+> rounds, each built on the round before it, every round is kept, and which one to hand
+> over is picked after the run (`addons/select`). A round that scored lower is a candidate
+> for that pick, not waste, so `3dcode cost` no longer reports those kinds (nor the
+> `cost.round` `regression` / `zero_delta` flags). "Repair loop that never converged" is
+> now repair money spent in a round that still did not build; post-budget stays.
+
Concrete examples:
* `tool_med_hand_drill` r01: **$2.48** to go from 0.576 to **0.560**.
@@ -349,7 +362,7 @@ Savings are estimated **on this data set** (61 runs, $86.30) unless stated.
| # | change | est. saving | confidence | where |
|---|---|---|---|---|
-| 1 | **Stop refining after r01 unless the last delta ≥ 0.05.** r02+r03 cost $12.16 and bought +1.44 score points across 8 of 20 rounds. | **$9–12 (10–13%)** | high (measured) | `RoundPolicy` (min_delta 0.02 → 0.05 from r02, or plateau_window 1 after r01) |
+| 1 | ~~**Stop refining after r01 unless the last delta ≥ 0.05.** r02+r03 cost $12.16 and bought +1.44 score points across 8 of 20 rounds.~~ **Removed 2026-09-22**: the round count is fixed (owner) — spend less with a lower `--rounds` or a profile, not with a judgement stop. | $9–12 (10–13%) | high (measured) | `--rounds` / `cost/profiles.py` |
| 2 | **Check the budget *before* starting a round, against the estimated round cost**, not only between steps. | **$8.76 (9.7%)** | high (measured waste) | `orchestrator.py` (`BudgetGuard`) + `tracks/steps.py`; use `codeverse3d.cost.estimate_call` / median round cost |
| 3 | ~~**Cap agent turns at ~25 and compact old tool results.** 39.3% of the agent bill is turn ≥20; 19 of the 33 sessions that ran ≥50 turns were cut off by their own budget.~~ **Tested, rejected: +$0.02 and −0.21 score** (A/B, n=3 per arm, §17) — the cap is off by default. | est. $8–15, **measured $0** | high (A/B) | `DEFAULT_AGENT_MAX_TURNS=0`; `$C3D_AGENT_MAX_TURNS` / `Settings.limits.agent_max_turns` still set one |
| 4 | **Fold the off-record spend into the budget** (cut rounds, post-hoc texture passes). | $0 saved, **$1.22 of blindness removed** | high | §6; the ledger (`codeverse3d.cost.record_call`) makes it automatic |
@@ -698,7 +711,8 @@ tool is registered for every object track, so the coding agent could buy a pass
any run. That is why the quality run below shows a ledger 9.5 % above its record:
the pass ran twice, once from inside a round-2 agent session.
`codeverse3d.texturing.run.texture_requested(spec)` is now the single owner;
-`tracks.lifecycle.finalise` and the tool both ask it, and the tool refuses with a
+the CLI hand-over after the run (`make`/`resume` → `addons.select.package`, which textures the
+PICKED round since 2026-09-22 — finalise no longer does) and the tool both ask it, and the tool refuses with a
usage error (costing $0) in a run whose spec says no. `3dcode texture pass <slug>`
is an explicit user instruction and is unaffected.
@@ -805,7 +819,7 @@ out of tree (no such test ships here):
|---|---|---|
| `no file change` | **removed** | unreachable: a generation result is `ok` only when a file changed, so `run_generation_tasks` raises `RoundFailed` (→ the loop's `no_change` plateau) before any judge question is asked |
| `build not repaired within the repair budget` | **removed** | a broken build never reaches the judge at all (`run_round` judges only when `build.ok`), so it only ever fired on a round that BUILT but still lint-failed — and the loop's `rejudge_round` then bought the same verdict one iteration later. Measured on the reproduction: 3 rounds "skipped", 2 verdicts re-bought as `judge.retry`, and the last round left **without a score — it was the best of the three (0.7)** and could not be promoted |
-| `budget already exceeded` | **kept** | the loop's next `budget_ok` check ends the run, so the verdict is never bought later. Guard, not a measured saving: 0 of the 108 recorded rounds bought a verdict after the budget ended, so this branch has saved $0 so far — it is kept because the state is knowable in advance and the run is over either way (§5) |
+| `budget already exceeded` | kept, then **removed 2026-09-22** | the loop's next `budget_ok` check ends the run, so the verdict was never bought later. Guard, not a measured saving: 0 of the 108 recorded rounds bought a verdict after the budget ended. Removed with the in-run best: a round's verdict is now what lets a pick choose it after the run, and the generation it scores is already paid for |
| `no judge` / `no renders` | **kept** | pre-existing guards (nothing to buy). The third pre-existing reason, `gate errors` behind `judge_on_gate_errors=False`, went 2026-09-21: no caller ever set it |
Two changes were needed to make the kept branches real:
@@ -813,16 +827,16 @@ Two changes were needed to make the kept branches real:
1. `rejudge_round` re-buys a verdict only after the judge was *tried* and failed
or came back degraded — never one that policy deliberately skipped. Otherwise
every skip is a `judge.retry` with the same price tag.
-2. `pick_best_round` never promotes a round that did not build, and when no round has
- a score at all it falls back to the built round with the **fewest gate errors**
- (later on a tie) instead of simply the last one — an unjudged round that broke
- the gates can no longer displace the clean artifact that came before it.
-
-The two controls that DID survive are unchanged and re-verified after this
-correction: the regression "switch then stop" rule and the r03+ marginal stop
-still cut 2 runs / **$1.14** with **0 best rounds lost** over the 107 recorded
-rounds (`waste-and-accounting/replay_stops.py`), and every round still emits its
-`cost.round` event.
+2. `pick_best_round` never promoted a round that did not build. (Removed 2026-09-22 with
+ the in-run best; `addons/select.pick` picks among JUDGED rounds only, so an unbuilt or
+ unjudged round is never handed over by score.)
+
+The two controls that DID survive this correction — the regression "switch then stop"
+rule and the r03+ marginal stop, which cut 2 runs / **$1.14** with **0 best rounds lost**
+over the 107 recorded rounds (`waste-and-accounting/replay_stops.py`) — were **removed on
+2026-09-22** with every other judgement stop: the owner fixed the round count (baseline +
+`--rounds`, cut short only by the clock or a hard failure) and moved the choice of round to
+after the run. Every round still emits its `cost.round` event.
---
diff --git a/harness/docs/INTERFACES.md b/harness/docs/INTERFACES.md
index 200f3bee..ccc8bebf 100644
--- a/harness/docs/INTERFACES.md
+++ b/harness/docs/INTERFACES.md
@@ -389,24 +389,14 @@ rec = get_track(spec.track, **options).run(spec, ws, resume=False) -> RunRecord
# services=, judge=, agent=, model=, runtime=, policy=RoundPolicy, settings=, planner_model=, n_candidates=
# (CLI --candidates > spec.options.candidates > settings.default_candidates); StaticObject | Articulated | Scene | Graphics
BaseTrack.run(spec, ws, *, resume=False, force=False) -> RunRecord # get_track returns a tracks.lifecycle.BaseTrack
-from codeverse3d.orchestrator import RoundPolicy, StopPolicy, StopDecision, pick_best_round, judge_sigma, \
- best_score, last_gain, REWRITE_KIND, build_refine_instructions, compact_instructions
-RoundPolicy(max_rounds=4, plateau_window=2, min_delta=0.02, target=0.8, max_refine_tasks=6,
- max_instructions_per_task=6, parallel_min_tasks=2, n_candidates=1, pairwise_margin=0.03,
- pairwise_min_confidence=0.6, judge_samples=1,
- judge_model="", regression_sigma=1.0, # money stops (docs/COST.md §5)
- marginal_sigma=1.5, marginal_from_round=3, # r03+ must beat 1.5σ
- detail_rounds=None, detail_min_score=0.45, detail_bbox_tol_m=0.005 # surface-detail round, tri-state:
- # None = the track default (lifecycle.DEFAULT_DETAIL_ROUNDS=1 when supports_detail_round, else 0),
- # 0 = off, N = N; $C3D_DETAIL_ROUNDS overrides all (the A/B switch). Injected policies keep it.
- ) # lifecycle.build_context binds n_candidates + judge_model with ONE dataclasses.replace;
- # RoundPolicy.target is ALWAYS overridden by the rubric pass threshold in BaseTrack.after_plan
-policy.sigma / .regression_delta / .marginal_delta # judge_sigma() reads cost.routing.JUDGE_NOISE — THE σ table
-StopPolicy(policy).evaluate(history, budget_ok=True) -> StopDecision(reason, strategy="same"|"switch"|"detail", detail)
- # (.reason is the StopReason); strategy "switch" = ONE whole-artifact rewrite round (kind REWRITE_KIND),
- # "detail" = ONE surface-detail round (kind DETAIL_KIND) — a plateau/diminishing stop is converted into it
- # only when detail_blocked(history, policy) == "" (clean gates, built, judged, within σ of best, budget left)
-from codeverse3d.orchestrator import DETAIL_KIND, KIND_FOR_STRATEGY, detail_blocked # THE strategy → kind dict
+from codeverse3d.orchestrator import RoundPolicy, build_refine_instructions, compact_instructions, gate_error_count
+RoundPolicy(max_rounds=4, max_refine_tasks=6, max_instructions_per_task=6, parallel_min_tasks=2,
+ n_candidates=1, judge_samples=1) # lifecycle.build_context binds n_candidates with ONE dataclasses.replace
+ # Δ 2026-09-22 FIXED rounds: a run is the baseline + max_rounds refine rounds, EACH built on the round before
+ # it; only the clock (BudgetGuard) or a hard failure ends it early. Gone with the judgement stops: StopPolicy /
+ # StopDecision, pick_best_round, judge_sigma, best_score, last_gain, REWRITE_KIND / DETAIL_KIND /
+ # KIND_FOR_STRATEGY / detail_blocked, RunState.best_* / update_best, and the RoundPolicy fields plateau_window,
+ # min_delta, target, pairwise_*, judge_model, regression_sigma, marginal_*, detail_*.
from codeverse3d.orchestrator import BudgetGuard, usage_delta
BudgetGuard(budget, start_time=None, *, soft_fraction=1.0) # no run= / ledger=
.charge(usage, *, stage="other", enforce=True) # (Δ 2026-08-30: role/label/round_index/outcome gone)
@@ -418,27 +408,25 @@ BudgetGuard(budget, start_time=None, *, soft_fraction=1.0) # no run=
# add(...) = charge(enforce=False) — "not enforced" never means "not seen".
.by_stage / .stage_summary() / .mark() # what a round burned, live (by_round[i] deleted — no reader;
# per-round money is CallCost.round in telemetry/cost.jsonl)
-from codeverse3d.tracks.candidates import CandidateRecord, rank_candidates, decide_best # pure decision logic
-from codeverse3d.tracks.candidates import run_best_of_n, choose_best_round, quick_render # N parallel baselines in <ws>/_cand/c<k>
+from codeverse3d.tracks.candidates import CandidateRecord, rank_candidates # the one in-loop choice left
+from codeverse3d.tracks.candidates import run_best_of_n, quick_render # N parallel baselines in <ws>/_cand/c<k>
# each candidate IS steps._run_round(kind="candidate") in its sub-workspace: render=quick_render(ctx, round_index, build,
# measurement, *, pipeline) (4 views + the articulated pose views), geometry_views=False, its own events.jsonl and a
# one-sample judge → _cand/c<k>/judge/r00.json (a degraded verdict stays score None); crashed candidate retried once;
-# selection by build_ok → quick score → fewer gate errors, pairwise within margin (booked stage=Stage.PAIRWISE);
+# selection by build_ok → quick score → fewer gate errors → the earlier candidate (no pairwise since 2026-09-22);
# winner copied back, normal r00 pipeline follows; rounds/candidates.json IS record.extra["candidates"] (n, selected,
-# candidates[], pairwise)
+# candidates[])
from codeverse3d.tracks.articulated_object import default_motion_checks, expected_direction # gate "motion_direction"
from codeverse3d.tracks.static_object import silhouette_gate, reference_refine_tasks # gate "reference_silhouette" (IoU<0.6 → WARN + refine task)
from codeverse3d.tracks.depth import depth_budget, DepthBudget, scope_groups, PartScope, interfaces_text, \
scoped_generation_enabled # complexity-aware budgets + per-part scoped generation
depth_budget(plan, *, build_timeout_s=300) -> DepthBudget # min/target/max triangles + max_build_s sized from
# the plan's LEAF count (parts × instances × children); .as_prompt() is the DETAIL BUDGET block every
- # generate/refine/detail template shows in place of a flat "≤ 300k tris"
+ # generate/refine template shows in place of a flat "≤ 300k tris"
scope_groups(plan, *, files_for, max_groups=6, parts_per_scope=3, min_parts=8) -> [PartScope]
# [] = one session owns the object (small plan, no per-part file ownership, or $C3D_SCOPED_PARTS=off);
# otherwise attachment-subtree groups whose files are disjoint, so the sessions run in parallel
interfaces_text(plan, scope) -> str # the planned boxes of the neighbours this scope must weld to
-from codeverse3d.tracks.static_object import drift_gate, detail_instructions, DRIFT_GATE # gate "detail_drift":
- # ERROR when a detail round moved/resized/removed a part or changed the overall extents (tol from policy)
from codeverse3d.tracks.prompting import base_prompt_context, reference_images, file_for_target_factory, \
scope_context, budget_for, detail_budget_text # Δ split out of
from codeverse3d.tracks.prompting import select_cookbook_chapters, is_always_chapter
@@ -509,7 +497,8 @@ generation prompts.
```python
from codeverse3d.texturing.run import texture_pass, texture_requested, load_report
texture_requested(spec) -> bool # THE owner of "does this run texture?" (Spec.options.texture, or
- # the legacy "texture" tag). Asked by tracks.lifecycle.finalise AND by the texture_pass spatial
+ # the legacy "texture" tag). Asked by the CLI hand-over after the run (make/resume → addons.select.package(
+ # texture=…), which textures the PICKED round — Δ 2026-09-22, finalise no longer does) AND by the texture_pass spatial
# tool, which refuses in a run that did not ask — the tool is registered for every object track,
# so `texture: false` used to be bypassable from inside an agent session.
texture_pass(ws, spec, plan, *, model_id, image_model=None, judge=True, judge_model_id=None, rubric=None,
@@ -526,15 +515,38 @@ from codeverse3d.texturing.plan import scene_texture_pack, texture_pack_prompt
## record/ + addons/ + cli/
```python
-from codeverse3d.record.record import finalize_record, load_record, iter_runs, best_round_index, best_round_record
-from codeverse3d.addons.gallery.index import hero_view # (ws, rec) -> (rel, label, n_views): the card image, rebased via ws
-from codeverse3d.record.record import complexity_block, round_complexity # objective complexity of what shipped
- # finalize_record fills record.extra["complexity"] = the BEST round's vector + plan_parts /
- # parts_per_plan_part / by_round; every rounds_summary row gains "complexity" (the index or None)
+from codeverse3d.record.record import finalize_record, load_record, iter_runs, effective_judgment, effective_score
+ # Δ 2026-09-22: best_round_index / best_round_record are gone, and fill_derived no longer derives a best round,
+ # baseline or final score — which round counts is addons.select's question
+from codeverse3d.record.deliverable import keep_round_artifacts, round_outputs, build_deliverable, load_deliverable
+ # keep_round_artifacts(ws, round_index) -> [names]: steps._run_round copies a BUILT round's hand-over files
+ # (object.glb object.stl object.step robot.urdf preview.gif frames_sheet.png meshes/*.glb) to artifacts/rNN/
+ # round_outputs(ws, rnd) -> Path | None: artifacts/rNN/, else artifacts/ for the best round a pre-2026-09-22
+ # record.json names, else None
+ # build_deliverable(ws, record, round_index, *, clean=True) -> RunDeliverable: deliverable/ for ONE round —
+ # its commit's code + its kept files + its sheet (+ graphics: its judged frames; + a shipped texture pack
+ # whose report names this round's GLB); never rebuilds. Only addons.select.package calls it.
+from codeverse3d.addons import select # which round of a FINISHED run to hand over (cli/ + addons only)
+select.round_rows(run_dir, *, record=None) -> [RoundRow{index, kind, score (effective), passed (the judge's own
+ # verdict for THAT round), gate_errors, build_ok, commit, cost_usd, minutes}]
+select.pick(run_dir, *, by="score"|"pairwise", pairwise_model=None, record=None, judge=None) -> int | None
+ # highest effective score → fewer gate errors → the earlier round; None when no round was judged. "pairwise":
+ # the top two within PAIRWISE_MARGIN=0.03 go to judges.pairwise.PairwiseJudge (position-swapped); the runner-up
+ # wins at confidence ≥ 0.6; one paid verdict per (pair, model), cached at artifacts/judge/rAA_vs_rBB_pairwise.json
+select.summarise(run_dir, *, record=None) -> RunSummary{rounds, stop_reason, baseline_score (r00's effective
+ # score), picked_round, picked_score, delta, method}: selection.json's round when the run was packaged, else pick()
+select.package(run_dir, round_index, *, texture=False, method="round", image_model=None) -> Path (deliverable/)
+ # + selection.json {round, method, scores, textured, selected_at}; texture=True runs the pass on that round's
+ # kept GLB unless a report already covers those bytes; a failed pass never fails the hand-over
+from codeverse3d.addons.gallery.index import hero_view # (ws, rec, picked) -> (rel, label, n_views): the card image
+from codeverse3d.record.record import complexity_block, round_complexity # objective complexity of what was built
+ # finalize_record fills record.extra["complexity"] = the LAST measured round's vector + plan_parts /
+ # parts_per_plan_part / by_round; every rounds_summary row gains "complexity" (the index or None); readers
+ # that show one round (gallery, dataset) read that round's own round_complexity
from codeverse3d.addons.dataset.export import export_samples # (runs_dir, out_dir, *, min_score=None, only_passed=False,
# include_unbuilt=False, captions_dir=None, drop_duplicates=False) -> ExportReport{…, n_duplicates, duplicates, tiers}
-from codeverse3d.addons.dataset.quality import quality_tier, prompt_hash, find_duplicates # tiers: A passed & 0 gate errors, B passed,
- # C best ≥ 0.6, D else; dedupe = (code fingerprint, prompt)
+from codeverse3d.addons.dataset.quality import quality_tier, prompt_hash, find_duplicates # tiers (of the exported round's verdict):
+ # A passed & 0 gate errors, B passed, C score ≥ 0.6, D else; dedupe = (code fingerprint, prompt)
from codeverse3d.addons.dataset.pairs import build_pairs # (runs_dir, out_jsonl, *, min_delta=0.05) -> n
from codeverse3d.addons.dataset.refine import build_refine, transitions, RefineTransition, REFINE_KINDS, outcome_of
# build_refine(runs_dir, out_jsonl, **kw) -> (rows written, Counter of drop reasons); writes via a .part file
@@ -573,7 +585,7 @@ from codeverse3d.addons.gallery.urls import safe_join # (root, rel) ->
from codeverse3d.addons.gallery.urls import content_type # .glb→model/gltf-binary, .py/.js/.frag→text/plain; charset=utf-8
from codeverse3d.addons.dataset.index import build_index, query, summary # sqlite + parquet: adds quality_tier, gate_errors, cost_usd,
# rounds, status, code_fingerprint, prompt_hash, duplicate_of, has_captions
-3dcodeverse make [--profile economy|balanced|quality]|resume|status|show|render|judge|tools|mcp
+3dcodeverse make [--profile economy|balanced|quality] [--no-pick]|resume [--no-pick]|pick|status|show|render|judge|tools|mcp
|texture {pass,scene-pack,show}|cost {<slug>,show,cache,prices,profiles,estimate}
|flywheel {export,pairs,refine,caption,index}|gallery {serve,build}
|bench {run,report}|doctor # alias: 3dcode
@@ -583,24 +595,27 @@ from codeverse3d.addons.dataset.index import build_index, query, summary # sql
`EventLog.emit(event, **data)` writes `{"t", "event", ...}` (**Δ** key is `event`).
Event names: `run.start`, `stage.start/done`, `plan.done`, `workspace.materialized`,
`round.start`, `generate.done`, `build.done`, `gates.done`, `judge.done`,
-`round.done`, `best.updated`, `refine.planned`, `recipes.seeded` (graphics: names written this call, present on disk, chapters), `asset.judged`, `assets.done`,
+`round.done`, `refine.planned`, `recipes.seeded` (graphics: names written this call, present on disk, chapters), `asset.judged`, `assets.done`,
`zones.done`, `assemble.done`, `round.no_change`, `candidates.start`,
-`candidate.start/done/failed/retry/selected`, `pairwise.done`,
+`candidate.start/done/failed/retry/selected`, `resume.reconciled` (…, `restored_last_round`),
`texture.start/plan/generated/applied/gate/done`, `budget.exceeded`,
-`finalise.rebuild`, `stop`, `run.done` / `run.failed`.
+`finalise.rebuild`, `stop` (reason, rounds), `run.done` (status, rounds, last_score) / `run.failed`;
+after the run: `pick.pairwise`, `pick.packaged`, `texture.skipped` / `texture.failed` (addons.select).
Cost events: **`cost.round`** (per round: `stages{stage → $}`, `judge_usd`, `total_usd`,
-`agent_turns` (= AgentResult.turns), `score`, `previous_best`, `wasted`, `waste_reason` ∈ aborted | build_failed |
-unjudged | regression | zero_delta, `run_usd`) — emitted for aborted rounds too;
-`judge.skipped` (reason), `generate.turn_cap` (label, max_turns, turns, cost_usd),
-`strategy.switch` (regression → whole-artifact rewrite), `budget.overrun` (a post-loop
-texture pass crossed the ceiling; the loop is already finished, so not `budget.exceeded`).
-`RoundRecord` gained `pairwise: PairwiseNote | None` (Δ 2026-08-30) — the paid tie-break
-verdict, stored so `reconcile_resume` REPLAYS it (`candidates.replay_best_round`) instead
-of re-ranking on score and reversing a comparison the run bought. `PairwiseNote` moved
-from `codeverse3d.tracks.candidates` to `codeverse3d.contracts.run` (it is pure data).
+`agent_turns` (= AgentResult.turns), `score`, `wasted`, `waste_reason` ∈ aborted | build_failed |
+unjudged, `run_usd`) — emitted for aborted rounds too (Δ 2026-09-22: `previous_best` and the
+regression / zero_delta reasons went with the in-run best: a lower-scored round is kept, not wasted);
+`judge.skipped` (reason), `generate.turn_cap` (label, max_turns, turns, cost_usd).
+Δ 2026-09-22: `best.updated`, `strategy.switch`, `round.refine_from_best`, `pairwise.done` and
+`budget.overrun` are gone, and so is `RoundRecord.pairwise` (the in-run tie-break note; an old
+rNN.json that carries one still loads, the key ignored). `PairwiseNote` lives in `addons.select`.
`RunRecord` (record.json): spec, plan, workspace, status, rounds[RoundRecord],
-best_round, baseline_score, final_score, total_usage, environment,
+total_usage, environment,
prompt_hashes{contract, cookbook, generate, refine}, error, extra{stop_reason,
rubric, budget, cost_by_stage, aborted_rounds?, rounds_summary, candidates? (the
rounds/candidates.json payload), texturing?, captions?}. `total_usage` is the BudgetGuard total whenever it
-exceeds the sum of the rounds (aborted rounds, retried sessions, texture pass).
+exceeds the sum of the rounds (aborted rounds, retried sessions); a texture pass bought by a pick is in
+telemetry/cost.jsonl, not in `total_usage`. Δ 2026-09-22: no `best_round` / `baseline_score` /
+`final_score` (old records carry them: ignored — `addons.select.summarise` answers), and `status` is only
+the stop reason: max_rounds | budget | agent_quota | no_change | no_refine_tasks | judge_unavailable |
+failed (an old `passed` / `plateau` loads as `stopped`).
diff --git a/harness/docs/RUNBOOK.md b/harness/docs/RUNBOOK.md
index abe36350..40fb30f2 100644
--- a/harness/docs/RUNBOOK.md
+++ b/harness/docs/RUNBOOK.md
@@ -56,11 +56,14 @@ python -m pytest tests -q -m live # OPT-IN: real API calls
```
Useful flags (`3dcode make --help`): `--generator`, `--planner`, `--judge`, `--captioner`
(backend ids, §3), `--image <png>` (repeatable; reference images → `ReferenceJudge` +
-silhouette gate + IoU refine tasks), `--rounds N` (refine rounds after the baseline),
+silhouette gate + IoU refine tasks), `--rounds N` (refine rounds after the baseline — a
+run does exactly N, each built on the one before it, unless the clock or a hard failure
+stops it; there is no pass / plateau / regression stop since 2026-09-22),
`--candidates N` (best-of-N baseline: N parallel candidates in `<ws>/_cand/`,
-quick-judged, pairwise tie-break, winner kept; multiplies baseline cost ≈ N;
-default from `settings.default_candidates`), `--texture` (run the texture pass after
-finalise; see §6), `--max-minutes`, `--dim height=0.45`, `--must`,
+quick-judged, the highest quick score kept (fewer gate errors on a tie); multiplies baseline
+cost ≈ N; default from `settings.default_candidates`), `--texture` (texture the PICKED round
+after the run; see §6), `--no-pick` (package nothing: no `deliverable/`, no `selection.json` —
+`3dcode pick` later), `--max-minutes`, `--dim height=0.45`, `--must`,
`--must-not`, `--style`, `--tag`, `--seed`, `--slug`, `--runs-dir`, `--force`,
`--no-run` (workspace + spec.json only — except that `--reference` still runs its
paid grounding pass first, since the grounded spec is what it writes).
@@ -108,9 +111,12 @@ artifacts in `artifacts/` (`object.glb`, `robot.urdf` + `meshes/`, scene
`metrics.json`, texturing `object_textured.glb` + `textures/`), per-round renders in
`artifacts/renders/rNN/`, gate JSON in `artifacts/gates/rNN/`, verdicts in
`artifacts/judge/rNN.json`, transcripts in `trajectories/<label>_rNN/` (retries in
-`<label>.a2_rNN`), events in `events.jsonl`, the flywheel record in `record.json`.
-`3dcode status <slug>` prints the rounds table (best round starred), cost and the last
-events.
+`<label>.a2_rNN`), events in `events.jsonl`, the flywheel record in `record.json`. The tree
+and `artifacts/` end at the LAST round; every round's own built files stay in
+`artifacts/rNN/`, and after the run `deliverable/` + `selection.json` hold the round a pick
+handed over (RUN_LAYOUT.md). `3dcode status <slug>` prints the stop reason, baseline → picked
+score, the rounds table (the picked round starred, each round's own judge verdict), cost and
+the last events.
### Looking at results locally
@@ -123,12 +129,13 @@ events.
**directories** too, so every link works: contact sheet, full-size renders, `record.json`,
`plan.json`/`spec.json`, the `src/` tree (browsable, line-numbered, `raw` = text/plain),
`object.glb` (orbit viewer on the vendored three.js — no network), articulation sheets,
-`preview.gif`, frames, textures. The page has a live summary strip (n, pass rate,
-mean/median score, total $, $ per passing artifact, wall clock — **recomputed per filter**),
+`preview.gif`, frames, textures. The page has a live summary strip (n = judged · unjudged ·
+error, mean/median score, total $, wall clock — **recomputed per filter**; no pass rate: a run
+is not passed or failed),
filters (track / language / tier / backend / verdict / battery + text search over
prompt+slug) and sort (score / cost / time / name) that never reload, a card ⇄ table
toggle, light/dark, and a detail page per run (`/run/<battery>/<slug>`) with every round,
-the best round's judge verdict + issues + improvement plan, the measurement table, the
+the picked round's judge verdict + issues + improvement plan, the measurement table, the
full render set, the cost breakdown and the code.
Filters live in the query string, so a view can be curled or bookmarked:
@@ -142,9 +149,20 @@ unless you type `--host` yourself; it never serves a path outside the declared r
## 5. Resume, re-render, re-judge, texture, export
```bash
-3dcode resume <slug> [--candidates N] # continues from run_state + stages/*.json (input-hash cached). The budget
+3dcode resume <slug> [--candidates N] [--no-pick]
+ # continues from run_state + stages/*.json (input-hash cached) and from the
+ # LAST round (a run recorded before 2026-09-22 that restored its best round
+ # is put back on its last). A max_rounds run resumes only with a raised
+ # --rounds, a budget run with any raised cap; an agent_quota or failed run
+ # resumes as is; the other stops are finished. The budget
# SNAPSHOT is restored: money/calls/active-minutes already spent still count,
# so a raised --max-minutes grants only the difference (downtime never counts)
+3dcode pick <slug> [--by score|pairwise] [--round N] [--texture] [--judge MODEL]
+ # hand over a round: deliverable/ + selection.json (addons/select). score =
+ # highest effective score, ties → fewer gate errors → the earlier round;
+ # pairwise = the pairwise judge between the top two when within 0.03 (one
+ # paid verdict per pair, cached); --round N = that round; --texture = the
+ # texture pass on it first. `make`/`resume` run `pick --by score` for you
3dcode render <slug> [--round N] [--mode shaded|wire|normals|clay|silhouette] [--out dir]
3dcode judge <slug> [--round N] [--rubric static_object_v1] [--model gemini:gemini-3.1-pro-preview] [--n 3]
# re-judges a round's recorded renders → artifacts/judge/rNN_cli.json
@@ -216,9 +234,10 @@ scoreless cells afterwards — delete the row from `results.jsonl` and the run d
then `bench run --id <prompt>`.
A refine session that dies on the VENDOR's usage limit ("You've hit your usage limit … try again at
-Sep 14th", `RESOURCE_EXHAUSTED`, `insufficient_quota`) stops the run at `agent_quota` with status
-`budget`: the best round ships, and the record says the agent's budget ended, not the code's
-improvement (cmp8, 2026-09-09, filed three such runs as `plateau` before this).
+Sep 14th", `RESOURCE_EXHAUSTED`, `insufficient_quota`) stops the run at `agent_quota`: every round
+so far is kept and picked from as usual, the record says the agent's budget ended, not the code's
+improvement (cmp8, 2026-09-09, filed three such runs as `plateau` before this), and `3dcode resume`
+continues it once the limit resets.
**A cell can also lose its renders to the box rather than to itself.** When the machine
runs out of memory Chrome reaps the render tab and the driver reports
@@ -244,7 +263,7 @@ asserts that the specific fixes the arm exists to measure are present.
**A new worktree needs `runtime_js/node_modules` before it can run a battery.** Without it
`render_glb` dies on every round, the judge is skipped for want of renders, and the cells
-come back `status=plateau` with `score=None` — an arm that reads as healthy and measures
+came back `status=plateau` (today: `judge_unavailable`) with `score=None` — an arm that reads as healthy and measures
nothing (2026-09-04, the mimic-off arm). `npm ci` in `runtime_js/`, or COPY the
directory from a worktree that has it (the `package.json` is the same file). The row now (a symlink gave two 404s on every render of one worktree on 2026-09-07 — `serve.cjs` refuses a real path outside its root — while another probe served through one; copy and be sure)
says `no verdict in any of N round(s)` when this happens.
@@ -255,7 +274,10 @@ stages are reused — this also recovers from Gemini 503 storms). Ctrl-C is saf
## 6. Texture pass
-`3dcode texture pass <slug>` (or `--texture` on `make`): one VLM material plan →
+`3dcode pick <slug> --texture` (or `--texture` on `make`, which picks after the run) textures the
+PICKED round's own `artifacts/rNN/object.glb` — once per round: a pass that already started from
+those bytes is re-used, never re-bought; `3dcode texture pass <slug>` textures the canonical
+`artifacts/object.glb`, i.e. the LAST round. Either way: one VLM material plan →
tileable texture images (gemini-3.1-flash-image, ~$0.07/tile, cached by prompt) →
world-metre UVs → `artifacts/object_textured.glb` → seam gate + before/after judge
gate (ships only when the score does not drop and the materials criterion improves).
diff --git a/harness/docs/RUN_LAYOUT.md b/harness/docs/RUN_LAYOUT.md
index 92061b15..0fa3080e 100644
--- a/harness/docs/RUN_LAYOUT.md
+++ b/harness/docs/RUN_LAYOUT.md
@@ -6,8 +6,8 @@ three separate buckets:
| bucket | question | contents |
|---|---|---|
-| `deliverable/` | **what did I get?** | the code snapshot of the best round, the canonical artifact (`object.glb` / `robot.urdf` + `meshes/` / `frames/` + `preview.gif` / the scene bundle), the best contact sheet, `captions.json`, `manifest.json` |
-| `evidence/` (= `artifacts/`) | **why should I believe it?** | `renders/rNN/`, `gates/rNN/`, `judge/rNN.json`, `measurement.json`, `articulation.json`, `census.json`, `textures/`, `tool_renders/` |
+| `deliverable/` | **what did I get?** | ONE round, the one a pick chose (`3dcode pick`; `3dcode make` picks by score after the run): its code snapshot, its artifact (`object.glb` / `robot.urdf` + `meshes/` / `frames/` + `preview.gif` / the scene bundle), its contact sheet, `captions.json`, `manifest.json` |
+| `evidence/` (= `artifacts/`) | **why should I believe it?** | every round's own build (`rNN/`), `renders/rNN/`, `gates/rNN/`, `judge/rNN.json`, `measurement.json`, `articulation.json`, `census.json`, `textures/`, `tool_renders/` |
| `telemetry/` | **what did it cost and how was it configured?** | `settings.json`, `cost.json`, `usage.jsonl`, plus `events.jsonl`, `run_state.json`, `stages/`, `trajectories/` |
The run's **identity** stays at the root, where every tool has always looked for it:
@@ -15,12 +15,13 @@ The run's **identity** stays at the root, where every tool has always looked for
```
runs/<slug>/
spec.json plan.json record.json identity: what was asked, planned, produced
+ selection.json which round deliverable/ holds and how it was chosen (after a pick)
run_state.json events.jsonl (physical home; also linked from telemetry/)
- src/ live working tree (git; every round is a commit)
+ src/ live working tree (git; every round is a commit; ends at the LAST round)
public/ (scene) compiled assets
- deliverable/ (a) the hand-over
+ deliverable/ (a) the hand-over — one round, written by a pick
evidence/ -> artifacts/ (b) the proof
- artifacts/ renders/ gates/ judge/ measurement.json …
+ artifacts/ the last round's build, rNN/ (each round's own), renders/ gates/ judge/ measurement.json …
telemetry/ (c) the accounting
settings.json cost.json usage.jsonl
events.jsonl -> ../events.jsonl
@@ -49,10 +50,11 @@ symlink. Which side is physical was not a matter of taste:
(tmp + rename) — a rename onto a *symlink* replaces the link, so the symlink
must be on the alias side, never on the side that gets written.
* **`deliverable/`, `telemetry/settings.json` and `telemetry/cost.json` are real
- files.** They are *derived*: rebuilt from the run's own git history,
- artifacts, trajectories and events every time the run is finalised, and on
- the fly by `3dcode show` / the exporters for a run that predates them — old
- runs are read as-is, never rewritten. `deliverable/` is self-contained so it can be zipped and handed to
+ files.** They are *derived*: `telemetry/` is rebuilt from the run's own
+ artifacts, trajectories and events every time the run is finalised,
+ `deliverable/` from a round's commit and its `artifacts/rNN/` every time a round
+ is picked, and both on the fly by `3dcode show` / the exporters for a run that
+ predates them — old runs are read as-is, never rewritten. `deliverable/` is self-contained so it can be zipped and handed to
someone (the flywheel exporter only falls back to it when git cannot answer).
`telemetry/usage.jsonl` is the one file that can be either: a real
reconstructed ledger, or a symlink to the run's live `telemetry/cost.jsonl`
@@ -64,32 +66,58 @@ symlink. Which side is physical was not a matter of taste:
(as well as `artifacts/`, `stages/`, `trajectories/`, …) so the derived buckets
never enter the code snapshot.
+## Every round, kept (`artifacts/rNN/`)
+
+The run ends at its LAST round (`src/` and the canonical `artifacts/` files), and since
+2026-09-22 no round is chosen during the run. So every round keeps what a hand-over of
+it needs, the moment it builds (`record.deliverable.keep_round_artifacts`, called by
+`tracks/steps._run_round` after the round's commit):
+
+```
+artifacts/rNN/ real copies, never hard links (the next build replaces the canonical files)
+ object.glb object.stl object.step (object tracks; whichever the language exports)
+ robot.urdf meshes/<link>.glb (articulated)
+ frames_sheet.png preview.gif (graphics)
+```
+
+The rest of a round is already per round: its code is its git commit (a scene's hand-over
+— `src/` + `public/`, GLB assets included — is nothing else), its renders and contact sheet
+are `renders/rNN/` (a graphics round's judged frames among them), its gates `gates/rNN/`,
+its verdict `judge/rNN.json` and `rounds/rNN.json`. The census, `build.json`, the `.blend`
+and the measurement are evidence of the LAST build only. A round that did not build keeps
+nothing. A run recorded before 2026-09-22 has no `rNN/`: its canonical files are the build
+of the best round its `record.json` still names (`round_outputs` reads it that way).
+
## `deliverable/`
-Built by `codeverse3d.record.deliverable.build_deliverable(ws, record)` from the
-**best round** (`record.best_round`, else the highest judged score):
+Built for ONE round by `codeverse3d.addons.select.package(run_dir, round)` →
+`record.deliverable.build_deliverable(ws, record, round)` — from that round's commit and its
+`artifacts/rNN/`, never a rebuild. `3dcode make` / `resume` package `select.pick(by="score")`
+after the run (highest effective judged score, ties → fewer gate errors → the earlier round;
+`--no-pick` skips it); `3dcode pick <slug> [--by pairwise] [--round N] [--texture]` re-packages
+any round:
```
deliverable/
- src/** raw code at the best round's commit (public/** too, for scenes)
+ src/** raw code at the round's commit (public/** too, for scenes)
object.glb object.stl object.step object_textured.glb
robot.urdf meshes/<link>.glb (articulated)
- frames/fNN_tT.png frames_sheet.png preview.gif (graphics)
- textures/<id>.png (when the texture pass shipped)
- sheet.png the best round's contact sheet
+ frames/*.png frames_sheet.png preview.gif (graphics: the round's judged frames)
+ textures/<id>.png (when a texture pass of THIS round shipped)
+ sheet.png the round's contact sheet
captions.json {detailed, instruction, factory} when captioned
manifest.json every file with role, size, sha256
```
-Every file is hashed with sha256. Small files (all code) are real copies, so
-editing the hand-over folder can never touch the evidence; binaries of 256 KB and
-up (frame stacks, GLBs, GIFs) are **hard-linked** to their evidence copy when the
-filesystem allows — same bytes, one block on disk, and `cp` / `tar` / `rsync`
-still produce a standalone copy (the six graphics bench runs grow from 140 MB to
-144 MB, not to 210 MB). A file over 128 MB (or a bucket over 512 MB) is left
-out and named in `manifest.skipped`. Rebuilding is idempotent: the manifest
-keeps its previous `generated_at` when the content is unchanged, so re-running
-the packager produces no diff.
+Beside `record.json`, `selection.json` says which round that is and why: `{round, method
+(score | pairwise | round), scores (every round's effective score), textured, selected_at}`.
+`addons.select.summarise` reads it, so every reader reports the round that was handed over.
+
+Every file is hashed with sha256 and is a real copy (never a hard link), so editing the
+hand-over folder can never touch the evidence. A file over 128 MB (or a bucket over
+512 MB) is left out and named in `manifest.skipped`. Rebuilding is idempotent: the manifest
+keeps its previous `generated_at` when the content is unchanged, so re-running the packager
+produces no diff.
## `telemetry/`
@@ -120,19 +148,22 @@ the packager produces no diff.
zones / baseline / refine / repair / judge / pairwise / texture / …`), per
role, per model, per round, plus `ledger_usd`, `unattributed_usd` (the
ledger's residual row) and `post_run_usd` (priced calls outside the run total,
- e.g. a texture pass that ran after the loop). `record.total_usage` stays the
- authority on what the run cost.
+ e.g. the texture pass or pairwise verdict a pick bought after the run).
+ `record.total_usage` stays the authority on what the run cost.
## `record.json`
-Two additive blocks, both `None` on older records:
+One additive block, `None` on older records:
-* `record.telemetry` — `RunTelemetry(settings, cost, environment, files)`;
-* `record.deliverable` — `RunDeliverable(best_round, commit, code_source, entry,
- files[path, role, bytes, sha256], total_bytes, skipped, generated_at)`.
+* `record.telemetry` — `RunTelemetry(settings, cost, environment, files)`.
-Everything else is unchanged, so a `record.json` written before this layout
-still validates and every consumer keeps working.
+The hand-over is described by `deliverable/manifest.json` (`RunDeliverable(round, commit,
+code_source, entry, files[path, role, bytes, sha256], total_bytes, skipped, generated_at)`) and
+`selection.json`, not by the record: since 2026-09-22 the record names no best round and
+carries no `deliverable` block. A `record.json` written before that still validates — its
+`best_round` / `baseline_score` / `final_score` / `deliverable` keys are ignored, a
+`passed` / `plateau` status reads `stopped`, and a manifest that says `best_round` loads as
+`round`.
## Reading a run
@@ -146,8 +177,8 @@ still validates and every consumer keeps working.
and settings block is computed on the fly (read-only) from the trajectories,
judge verdicts and events already on disk.
-For the money itself across many runs — waste, $ per passing artifact, price
-provenance — use the cost package's own command, `3dcode cost show <runs-dir>`.
+For the money itself across many runs — waste, $ per run, price provenance — use the cost
+package's own command, `3dcode cost show <runs-dir>`.
`3dcode show` is the single-run view; both read the same ledger rows.
## Back-compatibility contract
@@ -162,6 +193,6 @@ provenance — use the cost package's own command, `3dcode cost show <runs-dir>`
(`meta.code_source == "deliverable"`), and adds a compact `meta.telemetry`
digest when the run has one (`{}` otherwise).
* the gallery (`3dcode gallery build`) links the packaged
- `deliverable/object.glb` and `cost.json` when they exist and falls back to
- `artifacts/` otherwise; the per-stage cost line appears only for runs that
- carry telemetry.
+ `deliverable/object.glb` and `cost.json` when they exist, else the picked round's
+ `artifacts/rNN/` copy, else `artifacts/`; the per-stage cost line appears only for
+ runs that carry telemetry.