Skip to content

Official Codebase of 3DCodeVerse

Python

docs: the refine exporter, the git read path and the loss-event data, where they belong

ziyao12c23ada77b0
diff --git a/harness/docs/ARCHITECTURE.md b/harness/docs/ARCHITECTURE.md
index a5e9eebf..76ecffd6 100644
--- a/harness/docs/ARCHITECTURE.md
+++ b/harness/docs/ARCHITECTURE.md
@@ -552,7 +552,18 @@ passed, C best ≥ 0.6, D else), acceptance checklists, gate summaries and
errors, cost, fingerprints, `duplicate_of`; `--pack` tars with byte-range locators.
`flywheel pairs` emits preference pairs (round i < j by judge Δ ≥ τ) and round-level
repair pairs (in-session trajectory mining died with the api-agent, 2026-08-28 —
-vendor CLIs log raw stdout, not structured tool turns); `flywheel caption` adds
+vendor CLIs log raw stdout, not structured tool turns); `flywheel refine` emits the loop's OWN transitions — one row per (round i → round
+i+1) where the harness asked for a change: the failing round's gate findings and
+judge complaint, the refine instructions the harness wrote, both code snapshots (git
+`before`/`after` commits, `--with-code` inlines the files), the score before and
+after, and `outcome` (`improved` when the judge gained ≥ 0.05, `regressed` when it
+lost as much, `flat` otherwise, `build_failed` when the after-round did not build).
+Rows are dropped, with the reason recorded, when a predecessor has no commit, no
+verdict or did not build. That is the supervision the harness produces that a
+one-shot corpus cannot: what a failing artefact looked like, what was wrong with it
+in the harness's own words, and what the fix changed. `toolkits/llamafactory/`
+turns those rows into training files; the harness writes the measurement, not the
+trainer's format. `flywheel caption` adds
{detailed, instruction, factory} captions (image-grounded, brand-free, `--out` for
side-car mode); `flywheel gallery` is an alias of `3dcv gallery build --embed`
(the flywheel package has no renderer of its own).
diff --git a/harness/docs/DECISIONS.md b/harness/docs/DECISIONS.md
index cbfdfcfb..51e33fbe 100644
--- a/harness/docs/DECISIONS.md
+++ b/harness/docs/DECISIONS.md
@@ -667,7 +667,7 @@ written) that were accepted because the code works that way and the tests pin it
**0.225**, SE 0.065, **2 SE ±0.130**, sign 3 up / 8 down, p 0.227, separated from
noise **NO**; the rig's own n_for_power is ~506 pairs for ±0.02. Every phase-4/5
result sat inside that band, and the rig printed "revert" from identical arms.
- * **The channel that works.** `local/scripts/plan_degeneracy.py` runs the plan stage
+ * **The channel that works.** `bench/plan_stage_bench.py` runs the plan stage
ALONE (≈$0.03 and ≈85 s per call), so a 280-call arm is affordable where a 14-cell
battery is not. `CV3D_PLAN_RESTART` (a plan that names links it never lists is
re-sampled from the original request instead of edited in context) measured on 560
@@ -676,7 +676,9 @@ written) that were accepted because the code works that way and the tests pin it
39 of 40; cost per call unchanged ($0.0345 vs $0.0337). Ships ON with a kill switch.
The rule this sets: measure the loss event (a run that produced nothing, a retried
tool call, a pose the mechanism cannot reach), not the judge mean, unless the battery
- is large enough for the judge mean.
+ is large enough for the judge mean. The 560 rows live in
+ `bench/data/plan_stage/*.jsonl` with `bench/plan_stage_report.py`: a p-value whose
+ data is not in the tree is not reproducible (review, 2026-09-03).
* **What the judge channel could not have found.** `is_error = not obs.ok` (§30 of
docs/COST.md) made 62 % of 1 445 `joint_sweep` calls and 23 % of 2 144 `build` calls
arrive as broken calls the model retried, and — because gemini-cli stringifies an
@@ -697,6 +699,23 @@ written) that were accepted because the code works that way and the tests pin it
an inconclusive lever with no cost saving is not worth a second prompt path through
three templates, and the measurement above is the record of what it was worth. To
re-run it, restore the branch commit named in docs/PAPER_WRITING.md §9.3.
+ * **Reading an agent's repo is a sandbox boundary, and `git archive` is not inside it.**
+ Every flywheel read of a workspace already ran under `GIT_SAFE_FLAGS` (no repo config,
+ no hooks, no user config) plus `--no-ext-diff --no-textconv` for diffs, because a
+ `.gitattributes` the agent writes can name a `diff.<name>.textconv` command that git
+ RUNS on our side. `read_tree_at` still used `git archive`, which renders every blob
+ through `convert_to_working_tree` — so a planted `filter.<name>.smudge` executes, and
+ unlike textconv there is no flag that turns it off. It reads through `ls-tree -r -z`
+ + `cat-file --batch` now: same bytes, no filter path. Pinned by a test that plants a
+ smudge filter and fails on the old implementation (`tests/flywheel_cli`).
+ * **The refine transitions are the corpus this harness is for.** `3dcv flywheel refine`
+ emits one row per round the loop asked to change: the gate findings and judge
+ complaint that condemned round i, the instructions the harness wrote, both code
+ snapshots, and whether the score moved. A one-shot corpus cannot contain that pair;
+ the loop produces it as a by-product of running. Format conversion (LLaMA-Factory
+ messages) lives in `toolkits/llamafactory/`, not in the harness — the harness writes
+ the measurement, a toolkit writes whatever a trainer wants (owner's boundary,
+ 2026-09-03).
* **`<mimic>` is used and it changes the sweep, which is what it was for.** In the same
battery the planner and agent declared couplings in 8 of 14 prompts (umbrella 6 joints,
workbench 4, step ladder 4, scissor mirror 3, rolltop desk 3), and the sampled poses
diff --git a/harness/docs/INTERFACES.md b/harness/docs/INTERFACES.md
index 302ae319..9bc8a2b3 100644
--- a/harness/docs/INTERFACES.md
+++ b/harness/docs/INTERFACES.md
@@ -523,6 +523,19 @@ from codeverse.flywheel.export import export_samples # (runs_dir, out_dir, *,
from codeverse.flywheel.quality import quality_tier, prompt_hash, find_duplicates # tiers: A passed & 0 gate errors, B passed,
# C best ≥ 0.6, D else; dedupe = (code fingerprint, prompt)
from codeverse.flywheel.pairs import build_pairs # (runs_dir, out_jsonl, *, min_delta=0.05) -> n
+from codeverse.flywheel.refine import build_refine, transitions, RefineTransition, REFINE_KINDS
+ # build_refine(runs_dir, out_jsonl, *, with_code=False) -> n; transitions(rec, ws, *, with_code) -> list[RefineTransition]
+ # one row per round i -> i+1 the harness asked to change; outcome improved|regressed|flat|build_failed
+ # (threshold: pairs.MIN_PREFERENCE_DELTA); dropped rows carry the reason (no_predecessor / no_commit /
+ # predecessor_build_failed / predecessor_unjudged / git_read_failed / duplicate_run)
+from codeverse.flywheel._git import read_tree_at, diff_between, changed_files_between, commit_exists, GitReadError
+ # read_tree_at(ws, commit) -> {path: bytes} via ls-tree + cat-file --batch — NEVER `git archive`, which
+ # renders content through a planted filter.<name>.smudge and has no --no-filters (tests/flywheel_cli)
+ # diff_between(ws, before, after, *, max_bytes=None) -> (text, truncated); changed_files_between -> [path]
+ # both under GIT_SAFE_DIFF_FLAGS (--no-ext-diff --no-textconv) on top of workspace.GIT_SAFE_FLAGS
+from codeverse.workspace import GIT_SAFE_FLAGS, GIT_SAFE_DIFF_FLAGS, git_safe_env
+ # every read of an agent-written repo goes through these: no repo config, no external diff/textconv,
+ # no hooks, no user config (git_safe_env strips GIT_CONFIG*, GIT_DIR, GIT_EXEC_PATH, ...)
from codeverse.flywheel.captions import caption_sample # Δ (ws, record, model_id, *, model=None, out_dir=None) -> Captions;
# out_dir → side-car <out_dir>/<slug>.json, run untouched
from codeverse.gallery import build_index, default_roots, build_static, serve, GalleryApp # THE local gallery
@@ -537,7 +550,7 @@ from codeverse.flywheel.index import build_index, query, summary # sqlite + pa
# rounds, status, code_fingerprint, prompt_hash, duplicate_of, has_captions
3dcodeverse make [--profile economy|balanced|quality]|resume|status|show|render|judge|tools|mcp
|texture {pass,scene-pack,show}|cost {<slug>,show,cache,prices,profiles,estimate}
- |flywheel {export,pairs,caption,index,dedupe,gallery}|gallery {serve,build}
+ |flywheel {export,pairs,refine,caption,index,dedupe,gallery}|gallery {serve,build}
|bench {run,report}|doctor # alias: 3dcv
```
diff --git a/harness/docs/PAPER_WRITING.md b/harness/docs/PAPER_WRITING.md
index 5c03c6d5..ddf77e7c 100644
--- a/harness/docs/PAPER_WRITING.md
+++ b/harness/docs/PAPER_WRITING.md
@@ -10,8 +10,10 @@ The same facts live, in their own idiom, in `DECISIONS.md` (D36–D52, the decis
noise floor) and `ARCHITECTURE.md`. Tools referenced throughout:
`bench/prompts/{compare_v4,articulated_v2}.yaml` (batteries), `bench/ab_plan.py` (paired
A/B with an A/A mode), `bench/compare_backends.py` (harness vs one-shot),
-`bench/paired_compare.py` (paired statistics), `bench/refine_pairs.py` and
-`bench/refine_sft.py` (corpus export).
+`bench/paired_compare.py` (paired statistics), `bench/plan_stage_bench.py` +
+`bench/plan_stage_report.py` (the loss-event channel, data in `bench/data/plan_stage/`),
+`3dcv flywheel refine` (the transition corpus) and `toolkits/llamafactory/` (its training
+format).
## 0. The system, in one paragraph
@@ -163,9 +165,11 @@ Two of these carry a second, non-score reading worth keeping:
### 5.1 Planner mortality (the one significant single-switch result)
-**Method.** `local/scripts/plan_degeneracy.py` runs the **plan stage alone** — no build, no
+**Method.** `bench/plan_stage_bench.py` runs the **plan stage alone** — no build, no
judge — at ≈$0.03 and ≈85 s per call, so 280 calls per arm is affordable where 14 judged
-cells is not. Both arms run in the same window against the same provider.
+cells is not. Both arms run in the same window against the same provider. The 560 rows
+are in the tree (`bench/data/plan_stage/*.jsonl`, `bench/plan_stage_report.py` prints the
+table): a paper cites the p-value, so the data it comes from is committed.
**Failure class, measured first.** 200 calls over two code trees showed one class: the
planner writes **one top-level part** and hangs its joints off links it never lists, and
@@ -288,7 +292,7 @@ that moved the bill.
## 7. The corpus the loop produces
-`bench/refine_pairs.py` exports every refine round as the transition it was: the previous
+`3dcv flywheel refine` exports every refine round as the transition it was: the previous
round's judge issues and improvement plan, the gate ERROR findings with their fix hints, the
instruction lines the sessions were actually handed, the unified `src/` diff between the two
recorded commits, and the score delta with an improved/unchanged/regressed label. On the
@@ -297,8 +301,8 @@ cross-battery symlink aliases, 3 non-batteries, 2 empty), median diff 26 kB / 31
lines, **87 improved / 73 unchanged / 44 regressed**. Both sides come from the round's own
recorded commit — only 54 of 239 runs have `HEAD` at their last round.
-`bench/refine_sft.py` turns the improved transitions into the message shape the finetune
-pipeline consumes: **87 samples** (71 articulated, 16 static; median answer 26.6 kB, median
+`toolkits/llamafactory/build_refine_sft.py` turns the improved transitions into the message
+shape the finetune pipeline consumes: **87 samples** (71 articulated, 16 static; median answer 26.6 kB, median
brief 28.6 kB), answered with the files the round produced in the `=== FILE: path ===`
envelope. This is the one thing the harness produces that a one-shot corpus cannot: *given
a judged, gated object and a list of what is wrong with it, write the corrected files.*
@@ -376,7 +380,7 @@ instructions nor a diff.
| base64 amplification | 261 k tokens vs 516 for one 275 kB sheet | measured on a recorded blob + vendor bundle | arithmetic + matching recorded prompt jumps |
| token growth in turns | input(t) = 12 364 + 1 822·t; total ∝ n^1.60; dollars ∝ n^1.19–1.34 | 1 063 in-process sessions, 30 401 turns | the in-process agent, since gemini-cli emits no per-turn usage |
| coupled mechanisms declared | 8 of 14 prompts; poses drive 1–2 joints (was 5–6) | `wave2_lean` URDFs and round records | — |
-| refine corpus | 205 transitions, 87 SFT samples | `bench/refine_pairs.py`, `bench/refine_sft.py` over `bench/out` | corpus grows with every battery |
+| refine corpus | 205 transitions, 87 SFT samples | `3dcv flywheel refine`, `toolkits/llamafactory/build_refine_sft.py` over `bench/out` | corpus grows with every battery |
## 9.2 Open questions
@@ -429,10 +433,13 @@ python bench/ab_plan.py --prompts bench/prompts/articulated_v2.yaml \
python bench/compare_backends.py --prompts bench/prompts/articulated_v2.yaml \
--arms harness:gemini-cli:gemini-3.7-flash,oneshot+repair:gemini:gemini-3.7-flash \
--judge gemini:gemini-3.1-pro-preview --judge-samples 3 --out bench/out/art
-# the loss-event channel: the plan stage alone (see local/scripts/plan_degeneracy.py)
-# the corpus
-python bench/refine_pairs.py bench/out --summary
-python bench/refine_sft.py bench/out --out refine_sft.jsonl --summary
+# the loss-event channel: the plan stage alone, one row per call
+python bench/plan_stage_bench.py --tree . --label restart_on --reps 20 \
+ --out bench/data/plan_stage/restart_on.jsonl --env CV3D_PLAN_RESTART=1
+python bench/plan_stage_report.py bench/data/plan_stage/*.jsonl
+# the corpus the loop produces
+3dcv flywheel refine bench/out refine.jsonl --with-code
+python toolkits/llamafactory/build_refine_sft.py refine.jsonl --out refine_sft.jsonl
```
Recorded runs referenced above (not committed): `bench/out/{compare_v4_calm, compare_art_v2,
diff --git a/harness/docs/RUNBOOK.md b/harness/docs/RUNBOOK.md
index 43f61ff3..1efeb6b7 100644
--- a/harness/docs/RUNBOOK.md
+++ b/harness/docs/RUNBOOK.md
@@ -158,6 +158,14 @@ unless you type `--host` yourself; it never serves a path outside the declared r
3dcv flywheel export runs/ dataset/ [--min-score 0.7] [--only-passed] [--pack] [--include-unbuilt]
[--captions-dir caps/] [--drop-duplicates]
3dcv flywheel pairs runs/ pairs.jsonl [--min-delta 0.05]
+3dcv flywheel refine runs/ refine.jsonl [--with-code]
+ # one row per round i -> i+1 the harness asked to change:
+ # gate findings + judge complaint + the refine instructions,
+ # before/after commits (--with-code inlines the files),
+ # score_before/score_after, outcome improved|regressed|flat|
+ # build_failed, and dropped rows with the reason
+ # (no_predecessor / no_commit / predecessor_unjudged / ...).
+ # Training formats live in toolkits/llamafactory/, not here.
3dcv flywheel caption <slug> [--model …] [--out caps/] # --out = side-car mode, run untouched
3dcv flywheel gallery runs/ gallery.html [--title …] # alias of `3dcv gallery build --embed` (§4)
3dcv flywheel index runs/ runs_index.sqlite · 3dcv flywheel dedupe dataset/