Skip to content

Official Codebase of 3DCodeVerse

Python

refactor: consolidate LLaMAFactory tooling under finetune

Yipeng Gao0bea329baca2
diff --git a/CLAUDE.md b/CLAUDE.md
index d25786a5..fe964661 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -10,4 +10,7 @@ The harness lives in **`harness/`** — read `harness/CLAUDE.md` +
A/B rigs, reports; its tests: `cd eval && python -m pytest`), `eval/llm` evaluates a bare LLM/VLM
(was `finetune/3dcodeverse_eval`). Read `eval/README.md` first.
-Other top-level folders are separate components (`toolkits/` data tooling, `finetune/` training).
+Other top-level folders are separate components: `toolkits/` owns the general data CLI and source curation;
+`finetune/` owns training and its data preparation. The canonical LLaMA-Factory data scripts live in
+`finetune/llamafactory/`; `finetune/configs/`, `finetune/env/`, and the training launchers in
+`finetune/scripts/` retain their existing locations. Read `finetune/CLAUDE.md` before changing them.
diff --git a/README.md b/README.md
index 46ff10ce..803dbc27 100644
--- a/README.md
+++ b/README.md
@@ -73,8 +73,8 @@ on 805 fixed tasks:
| Directory | Description | Documentation |
| :-- | :-- | :-- |
| [`harness/`](harness/) | Agent-based 3D content generation, execution, rendering, and verification. | [Harness guide](harness/README.md) |
-| [`toolkits/`](toolkits/) | Dataset curation, format conversion, validation, and training-data preparation. | [Data toolkit guide](toolkits/README.md) |
-| [`finetune/`](finetune/) | Supervised fine-tuning and preference optimization for open models. | [Training guide](finetune/README.md) |
+| [`toolkits/`](toolkits/) | General data CLI and per-source corpus curation. | [Data toolkit guide](toolkits/README.md) |
+| [`finetune/`](finetune/) | LLaMA-Factory training-data preparation, supervised fine-tuning, and preference optimization for open models. | [Training guide](finetune/README.md) |
| [`eval/`](eval/) | Benchmarks and evaluation pipelines for the harness and standalone models. | [Evaluation guide](eval/README.md) |
## Citation
diff --git a/eval/docs/PAPER_WRITING.md b/eval/docs/PAPER_WRITING.md
index 59dbb1fb..fff40486 100644
--- a/eval/docs/PAPER_WRITING.md
+++ b/eval/docs/PAPER_WRITING.md
@@ -19,7 +19,7 @@ that owns it — §5.2 was exactly that until 2026-09-03 (review) and is now a p
A/B with an A/A mode), `bench/compare_backends.py` (harness vs one-shot),
`bench/paired_compare.py` (paired statistics), `bench/plan_stage_bench.py` +
`bench/plan_stage_report.py` (the loss-event channel, data in `bench/data/plan_stage/`),
-`3dcode flywheel refine` (the transition corpus) and `toolkits/llamafactory/` (its training
+`3dcode flywheel refine` (the transition corpus) and `finetune/llamafactory/` (its training
format).
## 0. The system, in one paragraph
@@ -53,7 +53,7 @@ full battery were `joint_sweep`.
count the loss event, run n in the hundreds (§5.1).
6. **Coupled-mechanism support for URDF articulation** with a mechanism-level readout (§5.3).
7. **A refine corpus**: (judge issues + gate findings + instructions) → code diff → score
- delta; `toolkits/llamafactory/` converts those rows into the finetune pipeline's format,
+ delta; `finetune/llamafactory/` converts those rows into the finetune pipeline's format,
which the harness itself never writes (§7).
## 0.2 Evaluation-protocol work that predates the measurements
@@ -424,7 +424,7 @@ round +0.197, 119 rows carrying gate ERRORs. Both sides come from the round's o
commit, not from `HEAD`. (The same export read 205 rows on 2026-09-02; the corpus grows
with every battery, so the selector is the date.)
-`toolkits/llamafactory/build_refine_sft.py` turns the improved transitions into the message
+`finetune/llamafactory/build_refine_sft.py` turns the improved transitions into the message
shape the finetune pipeline consumes: **112 samples** (92 articulated, 20 static; median
answer 25.4 kB, median brief 28.2 kB), answered with the files the round produced in the
`=== FILE: path ===` envelope. This is the one thing the harness produces that a one-shot corpus cannot: *given
@@ -511,7 +511,7 @@ data the repo does not carry; they are marked in the caveat column.
| base64 amplification | 261 k tokens vs 516 for one 275 kB sheet | measured on a recorded blob + vendor bundle | arithmetic + matching recorded prompt jumps; **no repo script** |
| token growth in turns | input(t) = 12 364 + 1 822·t; total ∝ n^1.60; dollars ∝ n^1.19–1.34 | 1 063 in-process sessions, 30 401 turns | **no repo script** — the in-process agent, since gemini-cli emits no per-turn usage |
| coupled mechanisms declared | 8 of 14 prompts; driven joints per pose 6 → 3 (1–2 on the one-input mechanisms) | `bench/coupling_stats.py bench/out/wave2_lean --per-prompt` | recomputed from the recorded URDFs |
-| refine corpus | 254 transitions, 112 SFT samples (2026-09-03) | `3dcode flywheel refine`, `toolkits/llamafactory/build_refine_sft.py` over `bench/out` | corpus grows with every battery |
+| refine corpus | 254 transitions, 112 SFT samples (2026-09-03) | `3dcode flywheel refine`, `finetune/llamafactory/build_refine_sft.py` over `bench/out` | corpus grows with every battery |
## 9.2 Open questions
@@ -606,7 +606,7 @@ python bench/plan_stage_bench.py --tree ../harness --label restart_on --reps 20
python bench/plan_stage_report.py bench/data/plan_stage/*.jsonl
# the corpus the loop produces
3dcode flywheel refine bench/out refine.jsonl --with-code
-python ../toolkits/llamafactory/build_refine_sft.py refine.jsonl --out refine_sft.jsonl # toolkits/ is at the repo root
+python ../finetune/llamafactory/build_refine_sft.py refine.jsonl --out refine_sft.jsonl # finetune/ is at the repo root
```
Recorded runs referenced above (not committed): `bench/out/{compare_v4_calm, compare_art_v2,
diff --git a/finetune/CLAUDE.md b/finetune/CLAUDE.md
index 49c099e3..40d8954f 100644
--- a/finetune/CLAUDE.md
+++ b/finetune/CLAUDE.md
@@ -1,6 +1,19 @@
# finetune/ (component notes for agents)
-LLM finetuning + execution-based eval for 3DCodeVerse. Read `README.md` here and `docs/REPORT.md` (conclusions + pitfalls) first.
-- Training is **unpatched LLaMA-Factory** driven by `configs/lf/*.yaml`; do not fork the framework — add a YAML + a dataset entry.
-- Evaluate by *executing* the code (`eval/`): 3DCodeBench (Blender → Chamfer/F vs GT), per-dialect held-out sets, pass@N. Greedy decoding collapses on boilerplate-heavy dialects (OpenSCAD/GLSL) — use sampling there.
-- Absolute paths point at the original machine; adapt `env.sh` + YAML path keys + tool paths in `eval/*runner*.py` before running.
+Training-data preparation and LLM fine-tuning for 3DCodeVerse. Read `README.md` here and
+`docs/REPORT.md` (conclusions and pitfalls) first.
+
+- Training is **unpatched LLaMA-Factory** driven by `configs/lf/*.yaml`; keep the upstream framework
+ external and add a recipe plus the corresponding dataset registry entries.
+- `llamafactory/` is the canonical home for this project's LF data conversion, QC, pair builders,
+ mix sampling, and publication scripts. It is not vendored framework code. Edit implementations there;
+ the seven Python entries in `scripts/` are compatibility forwarders.
+- Configs stay in `configs/`, environment snapshots in `env/`, and train/export/orchestration launchers
+ in `scripts/`. General source curation and the contributor CLI belong in `../toolkits/`.
+- Evaluate generated code with `../eval/llm/`: 3DCodeBench, held-out dialect suites, and pass@N. Follow
+ its protocol and decoding defaults. The old stop-DPO scripts still need adaptation to its generation
+ schema; do not assume replacing a path is sufficient.
+- Workstation paths remain in `env.sh`, launchers, configs, and data tools. Inspect them before running;
+ editing `env.sh` alone is insufficient. Current local training recipes also require external data
+ registries not fully represented by `data_specs/dataset_info.json`.
+- Keep historical reports and experiment records intact when reorganizing code.
diff --git a/finetune/README.md b/finetune/README.md
index d062b5cf..e985561a 100644
--- a/finetune/README.md
+++ b/finetune/README.md
@@ -1,48 +1,84 @@
-# finetune/ — LLM finetuning recipes + execution-based evaluation for 3DCodeVerse
+# finetune/ — training data and LLaMA-Factory recipes for 3DCodeVerse
-Everything needed to reproduce the Qwen3.5-9B finetuning study on 3DCodeVerse
-(text → Blender-Python / CadQuery / OpenSCAD / GLSL / three.js code), and the
-conclusions in `docs/REPORT.md`. The harness (`../harness`) generates data with
-API models; this folder trains and evaluates open models on that data.
+This component prepares training data and fine-tunes open models to generate Blender-Python,
+CadQuery, OpenSCAD, GLSL, and three.js code. The harness in [`../harness/`](../harness/) generates
+data with API models; standalone model evaluation lives in [`../eval/llm/`](../eval/llm/).
-**Nothing in LLaMA-Factory is patched** — all training is plain
-`llamafactory-cli` YAML (`configs/lf/`) + ShareGPT JSON registered in
-`data_specs/dataset_info.json`. Pinned environments are in `env/`.
+Training uses **unmodified upstream LLaMA-Factory**, pinned in `env/requirements-lf.txt`, through
+`llamafactory-cli` and ordinary YAML recipes. `llamafactory/` contains this project's data-preparation
+tools; it is not a copy or fork of the LLaMA-Factory framework.
## Layout
| path | what |
|---|---|
-| `scripts/` | data builders (`prep_data.py`, `build_multidialect*.py`, `build_md_big.py`, `build_md_xl.py`, `build_md_bal.py`, `build_boot_data.py`, `cache_ntok.py`), thin launchers around LLaMA-Factory (`lf_train.sh`, `lf_export.sh`, `make_lora_cfg.py`), experiment pipelines train→merge→eval (`run_exp.sh`, `run_md_exp.sh`, `run_full_ft*.sh`, `*_ckpt_sweep.sh`), execution-feedback / geometry-feedback DPO rounds (`run_dpo_exec.sh`, `dpo_round2.sh`, `md_dpo_round.sh`, `md_big_dpo_round.sh`, `md_xl_dpo_round.sh`, `geo_dpo_build.sh`), bootstrapping (`self_train_round*.sh`), report tooling (`build_report_html.py`, `contact_sheet_3dcodebench.py`) |
-| `configs/lf/` | one LLaMA-Factory YAML per experiment: LoRA SFT (`lora_*.yaml`), full-parameter ZeRO-3 SFT (`qwen35_9b_full_sft*.yaml`, `full_md_bal.yaml`, `ds_z3_offload.json`), DPO (`dpo_*.yaml`), LoRA merge/export (`export_*.yaml`) |
-| [`../eval/llm/`](../eval/llm/) | the execution-based evaluation (was `eval/` here): vLLM / API generation, 3DCodeBench execution in Blender, geometry metrics Chamfer / F-score vs GT, pass@k, per-dialect executors (CadQuery, OpenSCAD, GLSL, three.js), held-out dialect sets — see its README |
-| `data_specs/dataset_info.json` | LLaMA-Factory dataset registry for every SFT/DPO set used (ShareGPT format; the JSON files themselves are rebuilt by `scripts/`) |
-| `env/requirements-{llmft,lf,vllm}.txt` | pip freezes of the three conda envs (data/eval, LLaMA-Factory training, vLLM inference) |
-| `env.sh` | shell env (caches, secrets file) used by all scripts |
-| `docs/REPORT.md` | full experiment report (setup, data audit, ablations, DPO, multi-dialect, scale-up, training-amount curves, qualitative checks); `docs/PROJECT_README.md` is the original working README; `docs/assets/` renders |
+| [`llamafactory/`](llamafactory/) | canonical training-data scripts: ShareGPT conversion, quality checks, multimodal pairs, token counts, mix sampling, and Hub packaging; includes the per-source `runners/` |
+| [`scripts/`](scripts/) | training/export launchers and workstation orchestration; seven old Python data-script paths are compatibility forwarders to `llamafactory/` |
+| [`configs/lf/`](configs/lf/) | current SFT and stop-DPO YAML recipes, the LoRA export template, and a DeepSpeed offload template; see its README for the current recipe list |
+| `data_specs/` | recorded dataset registry and data statistics; each run must also have the registry and datasets named by its own `dataset_dir` |
+| `env/requirements-{llmft,lf,vllm}.txt` | snapshots of the original data/eval, LLaMA-Factory training, and vLLM inference environments |
+| `env.sh` | original workstation shell setup for environments, caches, and a local secrets file |
+| [`docs/REPORT.md`](docs/REPORT.md) | experiment conclusions; `REPORT_FULL.md`, `PROJECT_README.md`, and `llm_finetune_exps.md` preserve the historical study and run log |
-## Data on the Hub
+Configs, environments, and training launchers keep their existing locations. General corpus curation
+and the contributor CLI remain in [`../toolkits/`](../toolkits/).
-The training data in LLaMA-Factory (ShareGPT) format is published as **[`ilabai/3dcodeverse-llamafactory`](https://huggingface.co/datasets/ilabai/3dcodeverse-llamafactory)** (private; same CC-BY-NC-SA / non-commercial terms as the parent `ilabai/3dcodeverse`): the corpus sharded by dialect (`sft/<dialect>/`), the 15.7k execution-verified bootstrapped Blender samples, seven preference-pair sets (execution feedback and geometry feedback), the held-out test sets, a drop-in `dataset_info.json`, and `mixes.json` with the exact budget spec of every training mix in the report. Rebuild/refresh it with `scripts/build_hf_dataset.py`; regenerate any mix with `scripts/sample_mix.py`.
+## Data preparation
-## Results
-* **[`llm_finetune_exps.md`](llm_finetune_exps.md)** — the full experiment log: every training run with its config and wall time, every 3DCodeBench and per-dialect evaluation, and a summary of which levers moved the numbers (auto-generated tables from `scripts/build_exp_log.py` + curated commentary).
+Use the tools in [`llamafactory/`](llamafactory/README.md) for corpus conversion, validation,
+multimodal datasets, and mix sampling. For example, from this directory, after preparing the source
+`train.jsonl` and `val.jsonl`:
-## Install & background
-* **`docs/SETUP.md`** — step-by-step environment install (three conda envs with exact pins, LLaMA-Factory commit, flash-attn / fla / triton, vLLM, Blender / OpenSCAD / glslang / Playwright, caches on a network FS, smoke tests).
-* **`docs/LLAMA_FACTORY.md`** — how this project uses LLaMA-Factory (unpatched): dataset registry, template, stages (SFT / DPO), LoRA vs full + ZeRO-3, packing, launch/merge scripts, an annotated minimal YAML, and the pitfalls.
-* **`docs/REPORT.md`** — the experiment report (results + conclusions).
+```bash
+python llamafactory/sample_mix.py --src /path/to/md_max --out /path/to/mix \
+ --budget "blender=all,cadquery=25M,glsl=25M" --captions_per_sample 1
+```
-## Reproducing (short version)
+The study's ShareGPT data is published as
+**[`ilabai/3dcodeverse-llamafactory`](https://huggingface.co/datasets/ilabai/3dcodeverse-llamafactory)**
+(private; see the dataset card for its non-commercial terms). `llamafactory/build_hf_dataset.py`
+packages the corpus, historical preference sets, held-out tests, and mix descriptions from prepared
+local inputs. The published pair datasets are also used by the Hub recipe in `configs/lf/`.
-1. Envs (see `docs/REPORT.md` §1 for the pitfalls): `lf` = LLaMA-Factory (editable, upstream, unpatched) + torch 2.8.0+cu128 + flash-attn 2.8.3 + flash-linear-attention 0.5.2 + **triton 3.7.1** (Qwen3.5 GDN kernels on Hopper) + deepspeed; `vllm` = vLLM 0.27; `llmft` = data/eval (trimesh, cadquery, playwright, …). Blender 5.0.1 headless and an OpenSCAD AppImage are called by path.
-2. Data: only `metadata.parquet` of each 3DCodeVerse subset is needed. `scripts/build_multidialect.py` → `data/multidialect/<dialect>/{train,test}.jsonl`; `scripts/build_md_*.py` build the mixes (token-length filter ≤8192, cached by `scripts/cache_ntok.py`); `scripts/to_llamafactory.py` writes ShareGPT JSON + `dataset_info.json`.
-3. Train: `GPUS=0,1,2,3 scripts/lf_train.sh configs/lf/lora_md_xl.yaml gradient_accumulation_steps=2` (LoRA r64, cutoff 8192, packing, FA2) → `scripts/lf_export.sh configs/lf/export_md_xl.yaml` to merge.
-4. Evaluate with `../eval/llm/` (its README): from `../eval`, `python -m llm.run_eval --backend vllm --model <merged> --run <name> --suites headline heldout` (3DCodeBench: vLLM gen → Blender exec → Chamfer/F vs GT, plus the held-out dialect sets). Decode OpenSCAD/GLSL with sampling (T=0.7), not greedy.
-5. Execution-feedback DPO on top of any SFT model: `scripts/md_xl_dpo_round.sh` (sample → run in each dialect's executor → (OK, FAIL) pairs → `stage: dpo`).
+`data_specs/dataset_info.json` is a historical registry snapshot. The current local SFT recipes point
+at separate `mm_mix`, `full9b`, `pairs_mix`, or `pub_img` directories on the original workstation;
+their complete registries and data are not included in this checkout. Prepare or obtain the matching
+data and registry before launching a recipe. The remote recipe reads its registry from the Hub and
+still needs access to the dataset and its local media files.
-Paths are absolute for the machine the study ran on (`/wekafs/ict/hx_624/...`); adapt `env.sh`, the `*_dir`/`model_name_or_path` keys in `configs/lf/*.yaml`, and the tool paths at the top of `scripts/*.sh` before running elsewhere.
+## Train, export, evaluate
-## Headline results (3DCodeBench, 212 tasks, execution rate / [email protected]; details in `docs/REPORT.md`)
+Read [`docs/SETUP.md`](docs/SETUP.md) for the original environment setup and
+[`docs/LLAMA_FACTORY.md`](docs/LLAMA_FACTORY.md) for the integration. The shell launchers and data
+tools still contain `/wekafs/ict/hx_624/...` workstation paths. Adapt `env.sh`, the launchers, the
+YAML model/data/output paths, and tool paths before running elsewhere; changing `env.sh` alone does
+not update the other files.
-Qwen3.5-9B zero-shot 0% → LoRA on 5k Blender samples 75–82% → + execution-feedback DPO 93.9–96.2% → 4-dialect mixes: 25.6k 67%, 102k 84.9%, 270k (all 3DCodeVerse) 90.1%; 4 H100s: LoRA ≈17k tok/s, 270k samples / 293M tokens in 4.8 h.
+After that setup and data preparation, a current SFT recipe can be launched from `finetune/`:
+
+```bash
+GPUS=0,1,2,3 bash scripts/lf_train.sh configs/lf/sft_9b_lora64_lr1e-4_len8k_mmmix-plus.yaml
+bash scripts/export_lora.sh /absolute/path/to/run /absolute/path/to/the-same-base-model
+```
+
+The export script writes the merged model into `<run>/merged`. In the evaluation environment, from
+the repository's `eval/` directory, use the current standalone evaluator:
+
+```bash
+python -m llm.run_eval --backend vllm --model /absolute/path/to/run/merged \
+ --run my_sft --suites headline heldout
+```
+
+See [`../eval/llm/README.md`](../eval/llm/README.md) for tool/asset setup and suite decoding defaults.
+
+The stop-DPO recipes are in `configs/lf/dpo_*.yaml`; their workstation orchestration remains in
+`scripts/stop_dpo_pairs.sh` and `scripts/stop_dpo_round.sh`. Those scripts still depend on the old
+generation entrypoint, output schema, and prepared training prompts. Adapting them to `eval/llm/`
+is separate work; the folder reorganization does not make that historical pipeline portable.
+
+## Experiment records
+
+[`llm_finetune_exps.md`](llm_finetune_exps.md) and [`docs/REPORT.md`](docs/REPORT.md) document the
+results and lessons from the study. Historical run names and commands in these records may refer to
+recipes or drivers no longer present in the current checkout. Use [`configs/lf/README.md`](configs/lf/README.md)
+for the current recipe inventory.
diff --git a/finetune/docs/LLAMA_FACTORY.md b/finetune/docs/LLAMA_FACTORY.md
index 72e57ed2..7b7c40de 100644
--- a/finetune/docs/LLAMA_FACTORY.md
+++ b/finetune/docs/LLAMA_FACTORY.md
@@ -1,68 +1,92 @@
-# How this project uses LLaMA-Factory (primer)
+# How this project uses LLaMA-Factory
-[LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) is a config-driven finetuning framework on top of 🤗 transformers/peft/trl/deepspeed: you describe *model + data + stage + method + trainer args* in one YAML and run `llamafactory-cli train that.yaml`. We use it **unmodified** (editable install of upstream commit `c4e09c7`, 0.9.6.dev0; `git status` clean). Everything project-specific is a YAML in `configs/lf/` or a dataset entry in `data_specs/dataset_info.json`.
+[LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) consumes a YAML describing the model,
+data, training stage, and trainer arguments via `llamafactory-cli train <config.yaml>`. This project
+uses upstream unmodified: the training environment snapshot pins commit
+`c4e09c7cbe18844816af9e18a97fe465515edbcd` (0.9.6.dev0) in
+[`../env/requirements-lf.txt`](../env/requirements-lf.txt).
-## The pieces we touch
-| LLaMA-Factory concept | where in this repo | notes |
+Our [`../llamafactory/`](../llamafactory/) directory contains **training-data tools**, not framework
+source: conversion, validation, multimodal pair construction, token counting, mix sampling, and Hub
+packaging. Recipes remain in `configs/lf/`, environment snapshots in `env/`, and train/export
+launchers in `scripts/` (paths below are relative to `finetune/`).
+
+## The integration points
+
+| concept | where | notes |
|---|---|---|
-| **dataset registry** `dataset_info.json` | `data_specs/dataset_info.json` (copy to your `dataset_dir`) | every SFT set is ShareGPT JSON: `{"conversations":[{"from":"human","value":prompt},{"from":"gpt","value":code}],"system":...}`; DPO sets add `"chosen"/"rejected"` and `"ranking": true`. `scripts/to_llamafactory.py` writes both the JSON and the registry entry. |
-| **template** | `template: qwen3_5`, `enable_thinking: false` | Qwen3.5 is a VLM wrapper (`Qwen3_5ForConditionalGeneration`); we freeze the vision tower (`freeze_vision_tower: true`, `freeze_multi_modal_projector: true`). The template treats literal `<image>/<video>/<audio>` in text as media placeholders → escape them in data (`scripts/build_md_xl.py` does). |
-| **stage** | `stage: sft` (`lora_*.yaml`, `qwen35_9b_full_sft*.yaml`, `full_md_bal.yaml`), `stage: dpo` (`dpo_*.yaml`) | DPO: `pref_beta: 0.1`, `pref_loss: sigmoid`, lr 5e-6, cutoff 4096 (pairs are ~2× the memory of SFT). |
-| **method** | `finetuning_type: lora` (r16/r64, alpha 2r, dropout 0.05, `lora_target: all`) or `full` | full FT uses `deepspeed: …/ds_z3_config.json` (ZeRO-3; `configs/lf/ds_z3_offload.json` for 2-GPU CPU-offload). |
-| **packing** | `packing: true`, `neat_packing: true`, `cutoff_len: 8192` | neat_packing requires `per_device_eval_batch_size: 1`; keep `save_steps ≤ eval_steps`. |
-| **attention** | `flash_attn: fa2` | plus fla/causal-conv1d for the Gated-DeltaNet layers (triton 3.7.1 on Hopper). |
-| **tokenized cache** | `tokenized_path: …/tokenized_<name>_len8192_packed` | second run on the same data skips preprocessing. |
-| **launch** | `scripts/lf_train.sh <yaml> key=value…` | = `torchrun --nproc_per_node=$N llamafactory-cli train <abs yaml> key=value` with `CUDA_VISIBLE_DEVICES`, per-run Triton cache; any YAML key can be overridden on the CLI (we override `gradient_accumulation_steps`, `cutoff_len`, `num_train_epochs`). |
-| **merge LoRA** | `scripts/lf_export.sh configs/lf/export_<name>.yaml` | `llamafactory-cli export` → a plain HF checkpoint for vLLM. |
-| **checkpoints** | `save_steps`, `save_total_limit`, `save_only_model: true` | checkpoint sweeps (`scripts/*_ckpt_sweep.sh`) merge each `checkpoint-N` and evaluate it. |
+| dataset registry | each recipe's `dataset_dir` / `dataset_info.json` | `data_specs/dataset_info.json` is a historical snapshot; current local multimodal recipes require additional external registry entries and data |
+| data conversion | `llamafactory/to_llamafactory.py` and the other builders | ShareGPT uses `conversations` and `system`; preference sets add `chosen` / `rejected` and registry `ranking: true` |
+| current recipes | `configs/lf/sft_*.yaml`, `configs/lf/dpo_*.yaml` | six SFT recipes and two stop-DPO recipes; see [`../configs/lf/README.md`](../configs/lf/README.md) |
+| template | `template: qwen3_5`, `enable_thinking: false` | Qwen3.5 is a VLM wrapper; current recipes freeze the vision tower and multimodal projector |
+| LoRA | `finetuning_type: lora` | current SFT uses rank 64 / alpha 128; stop-DPO uses rank 16 / alpha 32 |
+| stop-DPO | `stage: dpo` | current recipes use `pref_beta: 0.1`, sigmoid loss, learning rate 5e-6, and cutoff 2048; historical execution/geometry-feedback studies used other recipes |
+| multimodal packing | `packing: false`, `neat_packing: false` | current SFT recipes disable packing for image-containing data |
+| attention | `flash_attn: fa2` | original Qwen3.5/Hopper environment also uses fla, causal-conv1d, and triton 3.7.1 |
+| preprocessing cache | `tokenized_path` in each local SFT recipe | reuse only for the same data and preprocessing settings |
+| training | `scripts/lf_train.sh <yaml> key=value…` | launches the installed LF CLI with selected GPUs and a local Triton cache; the wrapper still contains workstation paths |
+| merging | `scripts/export_lora.sh <run_dir> <base_model>` | builds an export YAML from `configs/lf/export_lora_template.yaml` and calls `scripts/lf_export.sh`; specify the same base model used for the adapter |
+| evaluation | `../eval/llm/` | model generation, dialect execution, geometry metrics, and reporting; see its README for the current CLI |
+
+Full-parameter and ZeRO-3 experiments are described in the historical reports. DeepSpeed templates
+remain in `configs/`; they do not turn the current LoRA recipes into full-parameter recipes.
+
+## Current SFT example
+
+The complete current recipe is
+[`sft_9b_lora64_lr1e-4_len8k_mmmix-plus.yaml`](../configs/lf/sft_9b_lora64_lr1e-4_len8k_mmmix-plus.yaml).
+Its data and training settings include:
-## Minimal recipe (what `configs/lf/lora_md_xl.yaml` says, annotated)
```yaml
-model_name_or_path: <path to Qwen3.5-9B>
-trust_remote_code: true
-flash_attn: fa2
stage: sft
-do_train: true
-finetuning_type: lora # or: full (+ deepspeed: ds_z3_config.json)
+finetuning_type: lora
lora_rank: 64
lora_alpha: 128
-lora_dropout: 0.05
lora_target: all
-freeze_vision_tower: true
-freeze_multi_modal_projector: true
-dataset_dir: <dir with dataset_info.json>
-dataset: md_xl_train
-eval_dataset: md_xl_val
+dataset_dir: /path/to/mm_mix
+dataset: 3dcv_mm_img_text_train,3dcv_mm_img_train,md_9b_plus_train
+eval_dataset: 3dcv_mm_img_text_val
template: qwen3_5
enable_thinking: false
cutoff_len: 8192
-packing: true
-neat_packing: true
-tokenized_path: <cache dir>
-output_dir: <run dir>
+packing: false
+neat_packing: false
per_device_train_batch_size: 1
-gradient_accumulation_steps: 2 # 4 GPUs × 1 × 2 × 8192 ≈ 64k tokens / step
-learning_rate: 1.0e-4
+gradient_accumulation_steps: 6
+learning_rate: 0.0001
num_train_epochs: 1.0
-lr_scheduler_type: cosine
-warmup_ratio: 0.03
bf16: true
-gradient_checkpointing: true
-per_device_eval_batch_size: 1
-eval_strategy: steps
-eval_steps: 450
-save_steps: 450
-save_total_limit: 14
-logging_steps: 5
-report_to: tensorboard
```
-Throughput on 4×H100 with this recipe: ≈17k tokens/s (LoRA r64), 270k samples / 293M tokens in 4.8 h; full-parameter ZeRO-3 on 4 GPUs ≈14k tokens/s.
-## Pitfalls we hit (details in `docs/REPORT.md` §1)
-1. torch 2.9.x + Qwen3.5 (Conv3D vision tower) is rejected by LLaMA-Factory → torch 2.8.0.
-2. fla Gated-DeltaNet backward on Hopper needs triton 3.7.1.
-3. `neat_packing` + eval batch size > 1 crashes at the first eval.
-4. `<image>` as literal text in a sample → "number of images does not match" → escape in data.
-5. Triton cache on a network FS races between concurrent runs → per-run local cache.
-6. DPO with 8k cutoff OOMs on long GLSL/OpenSCAD pairs on 80 GB → cutoff 4096.
-7. All paths handed to LLaMA-Factory must be absolute (it chdirs).
+This is an excerpt, not a standalone config. Prepare the named datasets and their registry, and
+adapt the model, cache, data, and output paths in the complete file before launching it. From
+`finetune/`, after adapting the workstation launchers:
+
+```bash
+GPUS=0,1,2,3 bash scripts/lf_train.sh configs/lf/sft_9b_lora64_lr1e-4_len8k_mmmix-plus.yaml
+bash scripts/export_lora.sh /absolute/path/to/run /absolute/path/to/the-same-base-model
+```
+
+The wrapper forwards LF CLI overrides, but it currently derives its own log and cache paths from
+the original YAML before applying those overrides. Set the intended output path in the recipe when
+using these wrappers. Changing `env.sh` alone does not relocate their hardcoded conda/LF paths.
+
+## Data and environment checks
+
+- Every `dataset` and `eval_dataset` name must resolve in the registry selected by `dataset_dir`.
+ Do not assume the historical registry snapshot contains a current multimodal mix.
+- Literal `<image>` tokens in code count as media placeholders too. Run
+ `python llamafactory/audit_pairs.py ...` against the prepared pair datasets before publishing;
+ the pair builder rejects assistant code containing that literal token.
+- Image placeholders must match the image list, media paths must resolve, and image-containing
+ rows must use the multimodal settings above.
+- The original training environment uses torch 2.8.0 and triton 3.7.1 for its Qwen3.5/Hopper stack;
+ see [`SETUP.md`](SETUP.md) and the pinned environment snapshot for the remaining dependencies.
+- Keep Triton caches on local disk for concurrent training on the original network filesystem.
+- The stop-DPO shell pipeline still expects the historical generation script and output format.
+ The current `eval/llm/` API uses different filenames/fields, so migrating that pipeline requires a
+ schema adapter and validation of its pairing behavior.
+
+Historical runs and their pitfalls remain in [`REPORT.md`](REPORT.md), [`REPORT_FULL.md`](REPORT_FULL.md),
+and [`../llm_finetune_exps.md`](../llm_finetune_exps.md). Their old paths and recipe names are records
+of those runs, rather than the current entrypoint inventory.
diff --git a/finetune/llamafactory/README.md b/finetune/llamafactory/README.md
new file mode 100644
index 00000000..fa21aec2
--- /dev/null
+++ b/finetune/llamafactory/README.md
@@ -0,0 +1,132 @@
+# llamafactory/ — 3DCodeVerse training-data tools
+
+This is the canonical home for the project's LLaMA-Factory data preparation: corpus conversion,
+quality checks, multimodal pairs, mix sampling, and publication. It contains project tooling, not
+upstream LLaMA-Factory source. General corpus curation stays in [`../../toolkits/`](../../toolkits/);
+training configs, environments, and launchers stay in the parent [`finetune/`](../README.md) component.
+
+The seven former data scripts in `../scripts/` forward to the implementation here for CLI compatibility.
+For new workflows, call `python llamafactory/<script>.py ...` from `finetune/`. Per-source shell drivers
+live in [`runners/`](runners/).
+
+## Script catalog
+
+| script | what it does |
+|---|---|
+| `convert_subdirs.py` | one `<subdir>_llamafactory/` per corpus subdirectory: ShareGPT `train.json`, registry, `qc.json`, and README. Checks empty/short/mojibake code, duplicates, captions, benchmark overlap, syntax, length, and literal media placeholders |
+| `flatten_multifile.py` | merges a Shadertoy shader's Common tab into its passes, and recovers `code.urdf` for Articraft subdirectories whose metadata has no code; writes parquet datasets |
+| `qc_repair.py` | checks parquet datasets with glslangValidator or Python `ast.parse`, attempts repairs, re-validates them, and reports duplicate groups within/across subdirectories |
+| `glsl_render_batch.py`, `glsl_render.py`, `render_verify_glsl.py` | render shaders in headless WebGL2 and check non-uniform images and animation |
+| `finalize_glsl.py` | prefer the original program, use a repair if it compiles, otherwise record `compiles=False`; writes `compiles` / `repair` columns |
+| `verify_unflattened_glsl.py` | compile-check the Shadertoy shards without a flattened variant and record the same `compiles` / `repair` fields |
+| `build_mm_dataset.py` | build `mm_img`, `mm_img_text`, and `mm_text` variants from the same source samples and split |
+| `build_image_dataset.py` | earlier single-variant image builder, retained for reproducing the first pilot |
+| `build_pairs_3dcodebench.py` | build text→code, image→code, and image+text→code pairs for one source subdirectory. Text variants retain usable captions; image-only pairs use one row per sample. Reads renders by scanning tars rather than trusting stale byte offsets |
+| `shrink_pair_images.py` | re-encode extracted PNG renders as JPEGs at a configurable maximum size (default 448 px); source tars retain the originals |
+| `audit_pairs.py` | check pair datasets for absolute/missing/non-JPEG media paths, placeholder counts across all turns, turn shape, fenced code, empty systems, and row-count/QC mismatches; exits nonzero on findings |
+| `upload_pairs.py` | publish pair datasets under `<source>/<subdir>_llamafactory_{text,img,imgtext}` and optionally renders under `<source>/pair_renders/`, then update the Hub registry |
+| `drop_old_pair_trees.py` | delete superseded top-level pair trees after checking that the Hub registry no longer points into them |
+| `build_md_max.py`, `sample_mix.py` | build the combined corpus and sample mixes with per-dialect token budgets and captions-per-sample caps |
+| `build_refine_sft.py` | turn improved harness refinement transitions into message JSONL for SFT |
+| `to_llamafactory.py` | convert message JSONL to ShareGPT JSON and registry entries |
+| `build_hf_dataset.py` | package prepared corpus data, historical preference sets, held-out tests, and mix descriptions for the Hub |
+| `cache_ntok.py`, `ntok_cache.py` | calculate/cache token counts for data builders |
+| `extract.py` | dialect-aware extraction from free-form model output for data QC |
+| `analyze_3dcodeverse.py`, `fetch_hf_files.py` | corpus inventory and selective Hub downloads |
+| `runners/` | per-source drivers for the pair-building workflows |
+
+## Formats and stage boundaries
+
+`convert_subdirs.py` writes JSON, while the flattening/pair builders and downstream QC tools use
+parquet. These are separate workflows with prepared inputs; the directory move does not provide a
+missing JSON→parquet conversion stage.
+
+```text
+<subdir>_llamafactory/ corpus conversion
+ train.json ShareGPT system + conversations[human,gpt]
+ dataset_info.json local registry entry
+ qc.json counts and quality findings
+ README.md provenance and filtering notes
+<subdir>_llamafactory_flat/ flattened shader variants
+ train.parquet code and verification fields where produced
+<subdir>_llamafactory_{text,img,imgtext}/
+ train.parquet pairs, plus images for multimodal variants
+ dataset_info.json
+ qc.json
+```
+
+For multimodal data, the total number of `<image>` placeholders across **all conversation turns**
+must equal `len(images)`. Literal placeholders inside assistant code also count; the pair builder
+rejects those samples and the audit checks both turns. Use `packing: false` and `neat_packing: false`
+for image-containing rows.
+
+The audit validates prepared outputs; it does not rewrite image paths or convert formats. Prepare
+relative JPEG media paths and the corresponding files before expecting it to pass.
+
+## Requirements and paths
+
+The original data environment uses Python 3.12 with `pandas`, `pyarrow`, `transformers` (tokenizer),
+Pillow, and `huggingface_hub` for publication. Particular scripts also need `requests`, Playwright
+and Chromium, glslangValidator, Blender, or OpenSCAD. See [`../env/`](../env/) for environment snapshots
+and [`../docs/SETUP.md`](../docs/SETUP.md) for the original setup.
+
+Many input, model, output, and binary paths still refer to `/wekafs/ict/hx_624/...` in script constants
+or defaults. Inspect and adapt them before running on another machine. Not every path has a CLI
+option, and setting `env.sh` alone does not change those constants. Hub operations require access
+to the relevant datasets; upload and deletion scripts change remote data.
+
+| dialect | verification used by the workflows |
+|---|---|
+| Blender-Python | Python syntax checks; execution workflows use headless Blender, with scipy needed by some reference scripts |
+| CadQuery | Python syntax checks; execution workflows use CadQuery |
+| OpenSCAD | OpenSCAD CLI |
+| GLSL | glslangValidator and headless WebGL2; compilation and visible rendering are distinct checks |
+| three.js / web | headless Chromium through Playwright |
+| URDF | recovery from source tars and XML checks |
+
+## Examples
+
+Run from `finetune/` after preparing inputs and adapting the script-specific paths. These are examples
+for separate stages, not a single ready-to-run pipeline:
+
+```bash
+# Corpus metadata -> ShareGPT JSON datasets
+python llamafactory/convert_subdirs.py --out /path/to/json-datasets --workers 16
+
+# Flatten shader passes / recover URDF from the prepared source corpus
+python llamafactory/flatten_multifile.py --shadertoy --out /path/to/flat-datasets
+python llamafactory/flatten_multifile.py --urdf_tars /path/to/tars --out /path/to/flat-datasets
+
+# QC expects prepared parquet datasets
+python llamafactory/qc_repair.py --pass all --root /path/to/parquet-datasets
+
+# Mix sampling expects message-format train.jsonl and val.jsonl
+python llamafactory/sample_mix.py --src /path/to/md_max --out /path/to/mix \
+ --budget "blender=all,cadquery=25M,glsl=25M" --captions_per_sample 1
+
+# Multimodal variants use the builder's prepared sources
+python llamafactory/build_mm_dataset.py --views 4 --out /path/to/mm
+
+# Audit every pair directory under the supplied root before publication
+python llamafactory/audit_pairs.py --roots /path/to/pairs --media_root /path/to/media
+```
+
+## Lessons from the data work
+
+1. **Check the checker.** Many shader failures came from validation defects: sampler types, missing
+ uniforms, transformations of minified code, or the GLSL ES profile. Use known-good controls.
+2. **Keep filtering decisions explicit.** Record QC findings and the reasons for drops. Benchmark
+ membership and runtime validity serve different purposes; an experiment must declare its split
+ and filtering policy rather than assuming conversion enforces it.
+3. **Benchmark overlap belongs to split selection.** `convert_subdirs.py` counts overlaps with the
+ 212 benchmark factories in `qc.json` and keeps those rows. It previously dropped entire instance
+ folders. Counting overlap does not itself make a later training mix leak-free.
+4. **Compare before deleting.** Same-named trees previously held different contents; the root copy
+ was sometimes newer. Compare contents and verify the published registry before retiring a copy.
+5. **Audit global invariants globally.** A full scan once found 36,964 absolute media paths that spot
+ checks missed. Also validate the audit itself: parquet sequence values can be numpy arrays,
+ which invalidated an earlier list-only check.
+
+Results and historical context remain in [`../llm_finetune_exps.md`](../llm_finetune_exps.md) and
+[`../docs/REPORT.md`](../docs/REPORT.md). Current training recipes are listed in
+[`../configs/lf/README.md`](../configs/lf/README.md).
diff --git a/toolkits/llamafactory/analyze_3dcodeverse.py b/finetune/llamafactory/analyze_3dcodeverse.py
similarity index 100%
rename from toolkits/llamafactory/analyze_3dcodeverse.py
rename to finetune/llamafactory/analyze_3dcodeverse.py
diff --git a/toolkits/llamafactory/audit_pairs.py b/finetune/llamafactory/audit_pairs.py
similarity index 93%
rename from toolkits/llamafactory/audit_pairs.py
rename to finetune/llamafactory/audit_pairs.py
index 3297f383..5d3897dc 100644
--- a/toolkits/llamafactory/audit_pairs.py
+++ b/finetune/llamafactory/audit_pairs.py
@@ -3,7 +3,7 @@
Sampling proves a problem exists; it cannot prove one does not. Checking four datasets said "paths are relative";
a full scan of all 411 found 36,964 absolute paths in three of them. This scans everything.
-usage: python scripts/audit_pairs.py [--media_root DIR] [--fix]
+usage: python llamafactory/audit_pairs.py [--media_root DIR] [--roots GLOB]
"""
import argparse
import collections
@@ -52,13 +52,13 @@ def main():
if not str(r.get("system") or "").strip():
bad["empty_system"] += 1
if kind == "text":
- if "<image>" in conv[0]["value"]:
+ if any("<image>" in t["value"] for t in conv):
bad["placeholder_in_text_pair"] += 1
continue
# `or []` on a numpy array raises rather than defaulting; the same numpy-vs-list confusion as above
raw = r.get("images")
imgs = [str(x) for x in raw] if raw is not None else []
- if conv[0]["value"].count("<image>") != len(imgs):
+ if sum(t["value"].count("<image>") for t in conv) != len(imgs):
bad["placeholder_mismatch"] += 1
for x in imgs:
tot += 1
diff --git a/toolkits/llamafactory/build_hf_dataset.py b/finetune/llamafactory/build_hf_dataset.py
similarity index 97%
rename from toolkits/llamafactory/build_hf_dataset.py
rename to finetune/llamafactory/build_hf_dataset.py
index f5a08d6f..ba656b62 100644
--- a/toolkits/llamafactory/build_hf_dataset.py
+++ b/finetune/llamafactory/build_hf_dataset.py
@@ -1,7 +1,7 @@
"""Package the finetuning data as a HuggingFace dataset in LLaMA-Factory (ShareGPT) format.
Design: publish the *corpus* + the *recipes* + the *derived* sets, not the 15 near-duplicate training mixes
-(they are all samples of the same corpus and would multiply the size for nothing — `scripts/sample_mix.py`
+(they are all samples of the same corpus and would multiply the size for nothing — `llamafactory/sample_mix.py`
regenerates any of them from a one-line budget spec).
sft/<dialect>/train-*.parquet the md_max corpus (768k prompt→code pairs), sharded by dialect
@@ -12,7 +12,7 @@ regenerates any of them from a one-line budget spec).
mixes.json the exact budget spec of every mix used in the report
README.md dataset card
-usage: python scripts/build_hf_dataset.py [--out /wekafs/ict/hx_624/hf_ds] [--push ilabai/3dcodeverse-llamafactory] [--private]
+usage: python llamafactory/build_hf_dataset.py [--out /wekafs/ict/hx_624/hf_ds] [--push ilabai/3dcodeverse-llamafactory] [--private]
"""
import argparse
import collections
@@ -193,7 +193,7 @@ llamafactory-cli train your_config.yaml # with: dataset_dir: data/3dcv, data
## Reproducing the mixes from the report
The report's training mixes are *samples* of this corpus, so they are described by a budget rather than shipped
-as duplicated files (`mixes.json` has the exact specs; `scripts/sample_mix.py` in the repo applies them):
+as duplicated files (`mixes.json` has the exact specs; `llamafactory/sample_mix.py` in the repo applies them):
| mix | pairs | tokens | captions/sample | result |
|---|---|---|---|---|
diff --git a/toolkits/llamafactory/build_image_dataset.py b/finetune/llamafactory/build_image_dataset.py
similarity index 98%
rename from toolkits/llamafactory/build_image_dataset.py
rename to finetune/llamafactory/build_image_dataset.py
index f92b014b..5803e105 100644
--- a/toolkits/llamafactory/build_image_dataset.py
+++ b/finetune/llamafactory/build_image_dataset.py
@@ -14,7 +14,7 @@ Sources (all already on disk):
threejs_distill 4 views per sample
animation2code 4 views per sample
-usage: python scripts/build_image_dataset.py [--out data/img_code] [--views 1] [--max_per_source 20000]
+usage: python llamafactory/build_image_dataset.py [--out data/img_code] [--views 1] [--max_per_source 20000]
"""
import argparse
import collections
diff --git a/toolkits/llamafactory/build_md_max.py b/finetune/llamafactory/build_md_max.py
similarity index 98%
rename from toolkits/llamafactory/build_md_max.py
rename to finetune/llamafactory/build_md_max.py
index 409192f2..d25612ec 100644
--- a/toolkits/llamafactory/build_md_max.py
+++ b/finetune/llamafactory/build_md_max.py
@@ -17,7 +17,7 @@ Deliberately excluded and why:
Each sample yields up to 3 training pairs (captions: instruction / detailed / factory), deduped by (prompt, code).
Token length is estimated as tokens(code) + tokens(prompt) + template overhead, tokenizing each unique code/prompt once.
-usage: python scripts/build_md_max.py [--max_tokens 8192] [--captions 3] [--out data/md_max]
+usage: python llamafactory/build_md_max.py [--max_tokens 8192] [--captions 3] [--out data/md_max]
"""
import argparse
import glob
@@ -26,13 +26,11 @@ import json
import os
import random
import re
-import sys
import unicodedata
from concurrent.futures import ProcessPoolExecutor
import pandas as pd
-sys.path.insert(0, "/wekafs/ict/hx_624/llm-ft/scripts")
META = "/wekafs/ict/hx_624/data/3dcodeverse_meta"
FILES = "/wekafs/ict/hx_624/data/3dcodeverse_files"
MD = "/wekafs/ict/hx_624/llm-ft/data/multidialect"
diff --git a/toolkits/llamafactory/build_mm_dataset.py b/finetune/llamafactory/build_mm_dataset.py
similarity index 99%
rename from toolkits/llamafactory/build_mm_dataset.py
rename to finetune/llamafactory/build_mm_dataset.py
index 806cf266..f7fc2ffd 100644
--- a/toolkits/llamafactory/build_mm_dataset.py
+++ b/finetune/llamafactory/build_mm_dataset.py
@@ -16,7 +16,7 @@ Format notes (checked against LLaMA-Factory's loader):
Sources: bioinspired3d (tars, 4 views), 3dcodebench factories (tars, 4 views, benchmark factories excluded),
blender_distill / threejs_distill / animation2code (file trees, 4 views).
-usage: python scripts/build_mm_dataset.py [--views 4] [--out data/mm]
+usage: python llamafactory/build_mm_dataset.py [--views 4] [--out data/mm]
"""
import argparse
import collections
diff --git a/toolkits/llamafactory/build_pairs_3dcodebench.py b/finetune/llamafactory/build_pairs_3dcodebench.py
similarity index 97%
rename from toolkits/llamafactory/build_pairs_3dcodebench.py
rename to finetune/llamafactory/build_pairs_3dcodebench.py
index 82b006f4..359c3eed 100644
--- a/toolkits/llamafactory/build_pairs_3dcodebench.py
+++ b/finetune/llamafactory/build_pairs_3dcodebench.py
@@ -5,7 +5,7 @@ separate row -- an augmentation three independent runs showed does not help. Her
pair type with one caption, and the renders come from the sample's own slice of the tar (metadata carries
`tar`/`byte_start`/`byte_len`, so a ranged read fetches one sample without downloading a 2 GB shard).
- python scripts/build_pairs_3dcodebench.py --subdir 3dcodebench/factories_geo --out DIR [--views 4] [--workers 8]
+ python llamafactory/build_pairs_3dcodebench.py --subdir 3dcodebench/factories_geo --out DIR [--views 4] [--workers 8]
[--limit N] [--local_tars DIR]
Writes <out>/<name>_{text,img,imgtext}/train.parquet + dataset_info.json + qc.json.
@@ -305,6 +305,12 @@ def main():
if len(code.strip()) < 40:
qc["short_code"] += 1
return None
+ # LLaMA-Factory counts <image> placeholders across EVERY turn, so a literal "<image>" in the answer
+ # (Shadertoy writes "// <image>" as a multipass section marker) reads as one more image than the row
+ # carries and aborts the whole run hours in. Seven such rows shipped once; drop them at the source.
+ if "<image>" in code:
+ qc["literal_image_token"] += 1
+ return None
caps = r.get("captions")
caps = dict(caps) if isinstance(caps, dict) else (json.loads(caps) if isinstance(caps, str) else {})
caps_list = captions_of(caps)
diff --git a/toolkits/llamafactory/build_refine_sft.py b/finetune/llamafactory/build_refine_sft.py
similarity index 100%
rename from toolkits/llamafactory/build_refine_sft.py
rename to finetune/llamafactory/build_refine_sft.py
diff --git a/toolkits/llamafactory/cache_ntok.py b/finetune/llamafactory/cache_ntok.py
similarity index 100%
rename from toolkits/llamafactory/cache_ntok.py
rename to finetune/llamafactory/cache_ntok.py
diff --git a/toolkits/llamafactory/convert_subdirs.py b/finetune/llamafactory/convert_subdirs.py
similarity index 98%
rename from toolkits/llamafactory/convert_subdirs.py
rename to finetune/llamafactory/convert_subdirs.py
index c63f1a7a..90edeab7 100644
--- a/toolkits/llamafactory/convert_subdirs.py
+++ b/finetune/llamafactory/convert_subdirs.py
@@ -15,7 +15,7 @@ Quality checks applied per sample (a sample can fail several):
scad-glsl, tag check for html) / too_long (>8192 tokens, still exported but flagged) / media_placeholder
(literal <image>/<video>/<audio> — breaks the qwen3_5 template unless escaped; we escape it and flag it)
-usage: python scripts/convert_subdirs.py [--out DIR] [--workers 16] [--limit_subdirs N]
+usage: python llamafactory/convert_subdirs.py [--out DIR] [--workers 16] [--limit_subdirs N]
"""
import argparse
import ast
@@ -25,13 +25,11 @@ import hashlib
import json
import os
import re
-import sys
import unicodedata
from concurrent.futures import ProcessPoolExecutor
import pandas as pd
-sys.path.insert(0, "/wekafs/ict/hx_624/llm-ft/scripts")
META = "/wekafs/ict/hx_624/data/3dcodeverse_meta"
FILES = "/wekafs/ict/hx_624/data/3dcodeverse_files"
BENCH_DIR = "/wekafs/ict/hx_624/data/3dcodebench/data"
@@ -234,7 +232,7 @@ def convert_one(args):
issues = "\n".join(f"- **{k}**: {v}" for k, v in sorted(qc.items(), key=lambda x: -x[1])) or "- none"
open(os.path.join(d, "README.md"), "w").write(f"""# {sub} — LLaMA-Factory format
-Generated by `finetune/scripts/convert_subdirs.py` from `{sub}/metadata.parquet` (or its file tree).
+Generated by `finetune/llamafactory/convert_subdirs.py` from `{sub}/metadata.parquet` (or its file tree).
| | |
|---|---|
diff --git a/toolkits/llamafactory/drop_old_pair_trees.py b/finetune/llamafactory/drop_old_pair_trees.py
similarity index 100%
rename from toolkits/llamafactory/drop_old_pair_trees.py
rename to finetune/llamafactory/drop_old_pair_trees.py
diff --git a/toolkits/llamafactory/extract.py b/finetune/llamafactory/extract.py
similarity index 100%
rename from toolkits/llamafactory/extract.py
rename to finetune/llamafactory/extract.py
diff --git a/toolkits/llamafactory/fetch_hf_files.py b/finetune/llamafactory/fetch_hf_files.py
similarity index 100%
rename from toolkits/llamafactory/fetch_hf_files.py
rename to finetune/llamafactory/fetch_hf_files.py
diff --git a/toolkits/llamafactory/finalize_glsl.py b/finetune/llamafactory/finalize_glsl.py
similarity index 97%
rename from toolkits/llamafactory/finalize_glsl.py
rename to finetune/llamafactory/finalize_glsl.py
index bbc883c1..cfde0157 100644
--- a/toolkits/llamafactory/finalize_glsl.py
+++ b/finetune/llamafactory/finalize_glsl.py
@@ -6,10 +6,9 @@ Order of preference, so the data stays as close to the author's source as possib
3. otherwise -> kept, marked compiles=False
Writes the `compiles` / `repair` columns back into every shadertoy *_llamafactory_flat parquet.
"""
-import collections, glob, hashlib, json, os, re, sys
+import collections, glob, hashlib, json, os, re
from concurrent.futures import ProcessPoolExecutor
import pandas as pd
-sys.path.insert(0, "/wekafs/ict/hx_624/llm-ft/scripts")
from qc_repair import code_of, glsl_compile
REPAIR = "/wekafs/ict/hx_624/glsl_repair"
diff --git a/toolkits/llamafactory/flatten_multifile.py b/finetune/llamafactory/flatten_multifile.py
similarity index 97%
rename from toolkits/llamafactory/flatten_multifile.py
rename to finetune/llamafactory/flatten_multifile.py
index 23e9af06..ef6bdb0c 100644
--- a/toolkits/llamafactory/flatten_multifile.py
+++ b/finetune/llamafactory/flatten_multifile.py
@@ -14,8 +14,8 @@ Two sources are multi-file in the original dataset and both can be flattened exa
Everything is then re-emitted in LLaMA-Factory ShareGPT format next to the other converted subdirs.
usage:
- python scripts/flatten_multifile.py --shadertoy [--out /wekafs/ict/hx_624/hf_subdirs]
- python scripts/flatten_multifile.py --urdf_tars /wekafs/ict/hx_624/data/3dcodeverse_tars
+ python llamafactory/flatten_multifile.py --shadertoy [--out /wekafs/ict/hx_624/hf_subdirs]
+ python llamafactory/flatten_multifile.py --urdf_tars /wekafs/ict/hx_624/data/3dcodeverse_tars
"""
import argparse
import collections
diff --git a/toolkits/llamafactory/glsl_render.py b/finetune/llamafactory/glsl_render.py
similarity index 100%
rename from toolkits/llamafactory/glsl_render.py
rename to finetune/llamafactory/glsl_render.py
diff --git a/toolkits/llamafactory/glsl_render_batch.py b/finetune/llamafactory/glsl_render_batch.py
similarity index 100%
rename from toolkits/llamafactory/glsl_render_batch.py
rename to finetune/llamafactory/glsl_render_batch.py
diff --git a/toolkits/llamafactory/ntok_cache.py b/finetune/llamafactory/ntok_cache.py
similarity index 90%
rename from toolkits/llamafactory/ntok_cache.py
rename to finetune/llamafactory/ntok_cache.py
index 7884a458..c984bef2 100644
--- a/toolkits/llamafactory/ntok_cache.py
+++ b/finetune/llamafactory/ntok_cache.py
@@ -1,4 +1,4 @@
-"""Helper for build scripts: load cached token counts for a jsonl (scripts/cache_ntok.py), falling back to tokenizing."""
+"""Helper for build scripts: load cached token counts for a jsonl (llamafactory/cache_ntok.py), falling back to tokenizing."""
import os
def load_ntok(path, tok=None):
c = path + ".ntok"
diff --git a/toolkits/llamafactory/qc_repair.py b/finetune/llamafactory/qc_repair.py
similarity index 99%
rename from toolkits/llamafactory/qc_repair.py
rename to finetune/llamafactory/qc_repair.py
index 22fe2afd..a62e58ce 100644
--- a/toolkits/llamafactory/qc_repair.py
+++ b/finetune/llamafactory/qc_repair.py
@@ -13,7 +13,7 @@ Three passes, each writing a report and (where it helps) a repaired copy of the
dedup find exact-duplicate programs *within* and *across* subdirs of the same source and write the
duplicate groups so the upstream dataset can be cleaned too.
-usage: python scripts/qc_repair.py --root /wekafs/ict/hx_624/hf_subdirs [--pass glsl|python|dedup|all] [--workers 32]
+usage: python llamafactory/qc_repair.py --root /wekafs/ict/hx_624/hf_subdirs [--pass glsl|python|dedup|all] [--workers 32]
"""
import argparse
import ast
diff --git a/toolkits/llamafactory/render_verify_glsl.py b/finetune/llamafactory/render_verify_glsl.py
similarity index 96%
rename from toolkits/llamafactory/render_verify_glsl.py
rename to finetune/llamafactory/render_verify_glsl.py
index c3773022..33a274d1 100644
--- a/toolkits/llamafactory/render_verify_glsl.py
+++ b/finetune/llamafactory/render_verify_glsl.py
@@ -4,12 +4,11 @@ Samples (or exhaustively walks) the unique programs in the flattened shadertoy d
WebGL2, and records status / distinct colours / pixel std / whether the image changes with iTime. Writes
`renders_ok`, `renders_status` back into the parquet rows when run with --write.
-usage: python scripts/render_verify_glsl.py [--sample 3000] [--workers 8] [--write]
+usage: python llamafactory/render_verify_glsl.py [--sample 3000] [--workers 8] [--write]
"""
-import argparse, collections, glob, hashlib, json, os, random, re, sys, tempfile
+import argparse, collections, glob, hashlib, json, os, random, re, tempfile
from concurrent.futures import ProcessPoolExecutor
import pandas as pd
-sys.path.insert(0, "/wekafs/ict/hx_624/llm-ft/eval")
os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/wekafs/ict/hx_624/cache/ms-playwright")
def code_of(m):
diff --git a/toolkits/llamafactory/runners/run_3dcb_pairs.sh b/finetune/llamafactory/runners/run_3dcb_pairs.sh
similarity index 84%
rename from toolkits/llamafactory/runners/run_3dcb_pairs.sh
rename to finetune/llamafactory/runners/run_3dcb_pairs.sh
index 44ff4e69..53727e01 100755
--- a/toolkits/llamafactory/runners/run_3dcb_pairs.sh
+++ b/finetune/llamafactory/runners/run_3dcb_pairs.sh
@@ -3,7 +3,11 @@
# Renders come from a sequential scan of each tar, NOT from metadata's byte offsets: for instances_geo only 577
# of 1,953 offsets point at a real tar header (the archives were repacked after the metadata was written), and
# trusting them silently cost 70% of the image pairs.
-set -uo pipefail; cd /wekafs/ict/hx_624/llm-ft
+set -uo pipefail
+FINETUNE_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)"
+PAIR_BUILDER="$FINETUNE_DIR/llamafactory/build_pairs_3dcodebench.py"
+cd "$FINETUNE_DIR" || exit 1
+mkdir -p logs
source /wekafs/ict/hx_624/anaconda3/etc/profile.d/conda.sh; conda activate llmft
OUT=/wekafs/ict/hx_624/hf_pairs
for SUB in instances_tex instances_geo_full instances_tex_full; do
@@ -27,6 +31,6 @@ os.makedirs(d, exist_ok=True); shutil.copy(p, f"{d}/metadata.parquet")
PY
fi
echo "[3dcb] $(date) building $SUB"
- python scripts/build_pairs_3dcodebench.py --subdir 3dcodebench/$SUB --out $OUT --workers 16 --views 4 2>&1 | tail -6
+ python "$PAIR_BUILDER" --subdir 3dcodebench/$SUB --out $OUT --workers 16 --views 4 2>&1 | tail -6
done
echo "[3dcb] ALL DONE"
diff --git a/toolkits/llamafactory/runners/run_other_pairs.sh b/finetune/llamafactory/runners/run_other_pairs.sh
similarity index 84%
rename from toolkits/llamafactory/runners/run_other_pairs.sh
rename to finetune/llamafactory/runners/run_other_pairs.sh
index 7e1b20cf..143b5bfd 100755
--- a/toolkits/llamafactory/runners/run_other_pairs.sh
+++ b/finetune/llamafactory/runners/run_other_pairs.sh
@@ -2,7 +2,11 @@
# The remaining parquet-backed sources. deepcad and bioinspired3d have renders, so they get all three pair types;
# articraft (URDF) and thingiverse have none, so only text->code is produced and the img/imgtext folders are
# simply not written -- an empty image dataset would be worse than an absent one.
-set -uo pipefail; cd /wekafs/ict/hx_624/llm-ft
+set -uo pipefail
+FINETUNE_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)"
+PAIR_BUILDER="$FINETUNE_DIR/llamafactory/build_pairs_3dcodebench.py"
+cd "$FINETUNE_DIR" || exit 1
+mkdir -p logs
source /wekafs/ict/hx_624/anaconda3/etc/profile.d/conda.sh; conda activate llmft
OUT=/wekafs/ict/hx_624/hf_pairs_other
mkdir -p $OUT
@@ -10,11 +14,11 @@ build () {
local SUB=$1 DIA=$2
local Q="$OUT/$(basename $SUB)_llamafactory_text/qc.json"
[ -f "$Q" ] && { echo "[oth] $SUB already built"; return; }
- python scripts/build_pairs_3dcodebench.py --subdir "$SUB" --dialect "$DIA" --out $OUT \
+ python "$PAIR_BUILDER" --subdir "$SUB" --dialect "$DIA" --out $OUT \
--workers 12 --views 4 --img_root /wekafs/ict/hx_624/data/pair_images > logs/pairs_oth_$(basename $SUB).log 2>&1
echo "[oth] $(date +%H:%M) $SUB $(grep -E '^\[pairs\] (text|img|imgtext)' logs/pairs_oth_$(basename $SUB).log | tr '\n' ' ' | tr -s ' ' | cut -c1-110)"
}
-export -f build; export OUT
+export -f build; export PAIR_BUILDER OUT
LIST=""
for d in /wekafs/ict/hx_624/data/3dcodeverse_meta/deepcad/*/metadata.parquet; do LIST="$LIST deepcad/$(basename $(dirname $d)):cadquery"; done
for d in /wekafs/ict/hx_624/data/3dcodeverse_meta/bioinspired3d/*/metadata.parquet; do LIST="$LIST bioinspired3d/$(basename $(dirname $d)):blender"; done
diff --git a/toolkits/llamafactory/runners/run_shadertoy_pairs.sh b/finetune/llamafactory/runners/run_shadertoy_pairs.sh
similarity index 81%
rename from toolkits/llamafactory/runners/run_shadertoy_pairs.sh
rename to finetune/llamafactory/runners/run_shadertoy_pairs.sh
index b24896b9..54f0205b 100755
--- a/toolkits/llamafactory/runners/run_shadertoy_pairs.sh
+++ b/finetune/llamafactory/runners/run_shadertoy_pairs.sh
@@ -3,7 +3,11 @@
# in metadata are not reliable), and the code comes from whichever verified parquet exists for that shard --
# *_llamafactory_flat for the 68 that could be flattened, the separately verified tree for the other 44 -- so the
# pairs carry a compiles column instead of being unverified GLSL.
-set -uo pipefail; cd /wekafs/ict/hx_624/llm-ft
+set -uo pipefail
+FINETUNE_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)"
+PAIR_BUILDER="$FINETUNE_DIR/llamafactory/build_pairs_3dcodebench.py"
+cd "$FINETUNE_DIR" || exit 1
+mkdir -p logs
source /wekafs/ict/hx_624/anaconda3/etc/profile.d/conda.sh; conda activate llmft
OUT=/wekafs/ict/hx_624/hf_pairs_shadertoy
VER="/wekafs/ict/hx_624/hf_subdirs/shadertoy,/wekafs/ict/hx_624/hf_subdirs_verified/shadertoy"
@@ -13,12 +17,12 @@ build () {
local SUB=$1
local Q="$OUT/${SUB}_llamafactory_img/qc.json"
if [ -f "$Q" ]; then echo "[st] $SUB already built — skipping"; return; fi
- python scripts/build_pairs_3dcodebench.py --subdir shadertoy/$SUB --dialect glsl \
+ python "$PAIR_BUILDER" --subdir shadertoy/$SUB --dialect glsl \
--out $OUT --workers 12 --views 4 --verified_from "$VER" > logs/pairs_st_$SUB.log 2>&1
local line=$(grep -E "^\[pairs\] (text|img|imgtext)" logs/pairs_st_$SUB.log | tr '\n' ' ' | tr -s ' ')
echo "[st] $(date +%H:%M) $SUB ${line:-FAILED: $(grep -iE 'error|Traceback' logs/pairs_st_$SUB.log | tail -1 | cut -c1-90)}"
}
-export -f build; export OUT VER
+export -f build; export PAIR_BUILDER OUT VER
SHARDS=$(ls -d /wekafs/ict/hx_624/data/3dcodeverse_meta/shadertoy/*/ | xargs -n1 basename)
echo "[st] $(echo "$SHARDS" | wc -l) shards, $PAR at a time"
echo "$SHARDS" | xargs -P $PAR -I{} bash -c 'build "$@"' _ {}
diff --git a/toolkits/llamafactory/sample_mix.py b/finetune/llamafactory/sample_mix.py
similarity index 98%
rename from toolkits/llamafactory/sample_mix.py
rename to finetune/llamafactory/sample_mix.py
index 548d8489..283585b4 100644
--- a/toolkits/llamafactory/sample_mix.py
+++ b/finetune/llamafactory/sample_mix.py
@@ -2,7 +2,7 @@
md_max is ~0.77M pairs / ~0.9B tokens and 97% CadQuery+GLSL; every real run wants a different slice of it.
- python scripts/sample_mix.py --out data/md_27b_max \
+ python llamafactory/sample_mix.py --out data/md_27b_max \
--budget blender=all,openscad=all,threejs=all,web=all,cadquery=25M,glsl=25M \
[--src data/md_max] [--captions_per_sample 2] [--seed 5]
diff --git a/toolkits/llamafactory/shrink_pair_images.py b/finetune/llamafactory/shrink_pair_images.py
similarity index 95%
rename from toolkits/llamafactory/shrink_pair_images.py
rename to finetune/llamafactory/shrink_pair_images.py
index 1f4b0e49..7059157f 100644
--- a/toolkits/llamafactory/shrink_pair_images.py
+++ b/finetune/llamafactory/shrink_pair_images.py
@@ -5,7 +5,7 @@ dataset. Every vision tower we train downsamples to 448x448 anyway (our own 27B
448^2), so a <=448px JPEG loses nothing a training run would have seen — and the untouched originals stay in the
corpus tars for anyone who needs them. Measured saving: 95%, so 84 GB becomes about 4 GB.
-usage: python scripts/shrink_pair_images.py [--root DIR] [--out DIR] [--max_px 448] [--quality 88] [--workers 64]
+usage: python llamafactory/shrink_pair_images.py [--root DIR] [--out DIR] [--max_px 448] [--quality 88] [--workers 64]
"""
import argparse
import glob
diff --git a/toolkits/llamafactory/to_llamafactory.py b/finetune/llamafactory/to_llamafactory.py
similarity index 100%
rename from toolkits/llamafactory/to_llamafactory.py
rename to finetune/llamafactory/to_llamafactory.py
diff --git a/toolkits/llamafactory/upload_pairs.py b/finetune/llamafactory/upload_pairs.py
similarity index 98%
rename from toolkits/llamafactory/upload_pairs.py
rename to finetune/llamafactory/upload_pairs.py
index 9af73627..c7101918 100644
--- a/toolkits/llamafactory/upload_pairs.py
+++ b/finetune/llamafactory/upload_pairs.py
@@ -5,7 +5,7 @@ rule as <source>/<subdir>_llamafactory_{text,img,imgtext}, and their renders as
top-level pairs/ or pair_images/ would read as if it were a source of its own, which is the same mistake the
five stray *_llamafactory folders made.
-usage: python scripts/upload_pairs.py [--batch 24] [--images] [--drop_old]
+usage: python llamafactory/upload_pairs.py [--batch 24] [--images] [--drop_old]
"""
import argparse
import glob
diff --git a/toolkits/llamafactory/verify_unflattened_glsl.py b/finetune/llamafactory/verify_unflattened_glsl.py
similarity index 97%
rename from toolkits/llamafactory/verify_unflattened_glsl.py
rename to finetune/llamafactory/verify_unflattened_glsl.py
index 5bb8dc33..301a0b69 100644
--- a/toolkits/llamafactory/verify_unflattened_glsl.py
+++ b/finetune/llamafactory/verify_unflattened_glsl.py
@@ -6,7 +6,7 @@ from and `flatten_multifile.py` skipped them — exactly 44 of 112, with zero mi
checker and the same minimal repairs over those, and writes the `compiles` / `repair` columns the flattened
folders already carry, so a training mix can filter the whole corpus by one rule instead of two.
-usage: python scripts/verify_unflattened_glsl.py [--workers 40] [--limit_shards N] [--out DIR]
+usage: python llamafactory/verify_unflattened_glsl.py [--workers 40] [--limit_shards N] [--out DIR]
"""
import argparse
import collections
diff --git a/finetune/scripts/README.md b/finetune/scripts/README.md
index 9d775b76..77bd6dcd 100644
--- a/finetune/scripts/README.md
+++ b/finetune/scripts/README.md
@@ -1,19 +1,34 @@
-# scripts/
+# scripts/ — launchers and compatibility entrypoints
-Everything here is generic. Per-experiment drivers were one-offs bound to a single config; they are in
-`archive/scripts/` (kept on disk, not synced).
+Training/export launchers and workstation orchestration stay here. Canonical LLaMA-Factory data
+implementations live in [`../llamafactory/`](../llamafactory/); see its README for the script catalog.
| script | what it does |
|---|---|
| `lf_train.sh <config>` | run a LLaMA-Factory config on `GPUS=<ids>` |
-| `train_when_free.sh <config> [need_mib] [log]` | wait for a card that is *still* free at launch, retry if another user takes it mid-load |
-| `free_gpus.sh [need_mib] [gpu_list]` | list cards with that much free memory (the box is shared) |
+| `train_when_free.sh <config> [need_mib] [log]` | wait for a free card and retry a training launch after a detected OOM |
+| `free_gpus.sh [need_mib] [gpu_list]` | list cards with the requested free memory |
| `lf_export.sh <config>` / `export_lora.sh <run_dir> [base]` | merge a LoRA adapter into `<run_dir>/merged` |
-| `stop_dpo_pairs.sh` | build termination preference pairs (`TAG`, `BASE_MODEL` env) |
-| `stop_dpo_round.sh <base_model> <tag>` | one full stop-DPO round: pairs → train → merge |
-| `supervisor2.sh` | keeps the queue in `queue.txt` running and warns on idle cards |
-| `sync_repo.sh` | mirror scripts/configs/eval + REPORT.md to the 3dcodeverse repo |
+| `stop_dpo_pairs.sh` | build termination preference pairs (`TAG`, `BASE_MODEL` env), using the historical generator/output schema |
+| `stop_dpo_round.sh <base_model> <tag>` | historical stop-DPO orchestration: pairs → train → merge |
+| `supervisor2.sh` | process the workstation job queue in `queue.txt` |
+| `sync_repo.sh` | legacy synchronization from the original workstation's external working tree; its scripts copy excludes Python files to preserve the compatibility forwarders |
-Data tooling: `build_pairs_3dcodebench.py` (pair builder), `convert_subdirs.py`, `shrink_pair_images.py`,
-`audit_pairs.py` (**run before publishing** — scans every row for absolute paths, missing images and
-`<image>`-count mismatches), `upload_pairs.py`, `drop_old_pair_trees.py`, `verify_unflattened_glsl.py`.
+These shell scripts retain workstation-specific paths and assumptions. The stop-DPO generation
+interface and some queue entries refer to historical scripts; inspect and adapt them before use.
+`sync_repo.sh` also predates the current directory ownership and should not be used to synchronize
+this layout without updating it.
+
+The following seven Python files are thin CLI forwarders, preserving existing command paths while
+running the implementation in `../llamafactory/`:
+
+- `audit_pairs.py`
+- `build_pairs_3dcodebench.py`
+- `convert_subdirs.py`
+- `drop_old_pair_trees.py`
+- `shrink_pair_images.py`
+- `upload_pairs.py`
+- `verify_unflattened_glsl.py`
+
+Use `python llamafactory/<script>.py ...` from `finetune/` for new workflows. Historical per-experiment
+drivers were archived outside this checkout; their absence is not filled by these compatibility paths.
diff --git a/finetune/scripts/audit_pairs.py b/finetune/scripts/audit_pairs.py
index 176d3401..a6f32161 100644
--- a/finetune/scripts/audit_pairs.py
+++ b/finetune/scripts/audit_pairs.py
@@ -1,81 +1,14 @@
-"""Audit every pair dataset for the invariants that fail silently at train time.
+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""
-Sampling proves a problem exists; it cannot prove one does not. Checking four datasets said "paths are relative";
-a full scan of all 411 found 36,964 absolute paths in three of them. This scans everything.
-
-usage: python scripts/audit_pairs.py [--media_root DIR] [--fix]
-"""
-import argparse
-import collections
-import glob
-import json
-import os
-
-import pandas as pd
+import runpy
+import sys
+from pathlib import Path
def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--media_root", default="/wekafs/ict/hx_624/data/pair_media_root")
- ap.add_argument("--roots", default="/wekafs/ict/hx_624/hf_pairs*")
- a = ap.parse_args()
- dirs = sorted(glob.glob(f"{a.roots}/*_llamafactory_*"))
- bad = collections.Counter()
- tot = rows = 0
- examples = {}
- for d in dirs:
- kind = os.path.basename(d).rsplit("_", 1)[-1]
- try:
- df = pd.read_parquet(f"{d}/train.parquet")
- except Exception as e:
- bad["unreadable_parquet"] += 1
- examples.setdefault("unreadable_parquet", f"{d}: {type(e).__name__}")
- continue
- q = json.load(open(f"{d}/qc.json"))
- if len(df) != q["rows"]:
- bad["row_count_vs_qc"] += 1
- for r in df.to_dict("records"):
- rows += 1
- # parquet hands back numpy arrays, not lists: an isinstance check against (list, tuple) alone
- # marks every single row as malformed, which is a defect in the checker, not the data
- conv = r.get("conversations")
- try:
- conv = list(conv)
- except Exception:
- conv = None
- if conv is None or len(conv) != 2:
- bad["bad_turn_shape"] += 1
- continue
- code = conv[1]["value"]
- if not code.startswith("```") or not code.rstrip().endswith("```"):
- bad["unfenced_code"] += 1
- if not str(r.get("system") or "").strip():
- bad["empty_system"] += 1
- if kind == "text":
- if any("<image>" in t["value"] for t in conv):
- bad["placeholder_in_text_pair"] += 1
- continue
- # `or []` on a numpy array raises rather than defaulting; the same numpy-vs-list confusion as above
- raw = r.get("images")
- imgs = [str(x) for x in raw] if raw is not None else []
- if sum(t["value"].count("<image>") for t in conv) != len(imgs):
- bad["placeholder_mismatch"] += 1
- for x in imgs:
- tot += 1
- if x.startswith("/"):
- bad["absolute_path"] += 1
- examples.setdefault("absolute_path", x)
- elif not x.endswith(".jpg"):
- bad["not_jpg"] += 1
- examples.setdefault("not_jpg", x)
- elif not os.path.exists(os.path.join(a.media_root, x)):
- bad["unresolvable"] += 1
- examples.setdefault("unresolvable", x)
- print(f" {len(dirs)} datasets | {rows:,} rows | {tot:,} image references")
- print(f" problems: {dict(bad) if bad else 'NONE'}")
- for k, v in examples.items():
- print(f" example {k}: {v[:100]}")
- raise SystemExit(1 if bad else 0)
+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"
+ sys.path.insert(0, str(scripts))
+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")
if __name__ == "__main__":
diff --git a/finetune/scripts/build_pairs_3dcodebench.py b/finetune/scripts/build_pairs_3dcodebench.py
index 1dc73647..a6f32161 100644
--- a/finetune/scripts/build_pairs_3dcodebench.py
+++ b/finetune/scripts/build_pairs_3dcodebench.py
@@ -1,383 +1,14 @@
-"""Build all three pair types for a 3DCodeVerse subdirectory: text->code, image->code, image+text->code.
+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""
-The corpus so far only carries text->code, and its row count is inflated ~3x because every caption variant is a
-separate row -- an augmentation three independent runs showed does not help. Here each SAMPLE appears once per
-pair type with one caption, and the renders come from the sample's own slice of the tar (metadata carries
-`tar`/`byte_start`/`byte_len`, so a ranged read fetches one sample without downloading a 2 GB shard).
-
- python scripts/build_pairs_3dcodebench.py --subdir 3dcodebench/factories_geo --out DIR [--views 4] [--workers 8]
- [--limit N] [--local_tars DIR]
-
-Writes <out>/<name>_{text,img,imgtext}/train.parquet + dataset_info.json + qc.json.
-"""
-import argparse
-import collections
-import glob
-import io
-import json
-import os
-import tarfile
-import time
-
-import pandas as pd
-import pyarrow.parquet as pq
-import requests
-
-META = "/wekafs/ict/hx_624/data/3dcodeverse_meta"
-REPO = "ilabai/3dcodeverse"
-SYS = {
- "web": "You are an expert creative web developer. Write a complete, standalone HTML page (inline CSS/JS, no local assets) that reproduces the requested animation. Output ONLY the code in one ```html block.",
- "threejs": "You are an expert three.js developer. Write a complete, standalone HTML page that renders the requested scene with three.js. Output ONLY the code in one ```html block.",
- "openscad": "You are an expert in OpenSCAD. Write a complete, standalone OpenSCAD program that builds the requested object. Output ONLY the code in one ```scad block.",
- "cadquery": "You are an expert in parametric CAD with CadQuery. Write a complete, standalone CadQuery script that builds the requested part. Output ONLY the code in one ```python block.",
- "blender": "You are an expert in procedural 3D modeling with Blender Python (bpy). Write a complete, standalone Blender 5 Python script that builds the requested object from scratch (clear the default scene first). Output ONLY the code in one ```python block.",
- "glsl": "You are an expert GLSL shader programmer. Write a complete Shadertoy-style fragment shader. Output ONLY the code in one ```glsl block.",
-}
-FENCE_RE = __import__('re').compile(r'^```[a-zA-Z]*\n|\n?```$')
-FENCE = {"web": "html", "threejs": "html", "openscad": "scad", "cadquery": "python", "blender": "python", "glsl": "glsl"}
-ASK_TEXT = {"web": "Write a single-file animated web page that reproduces: ", "threejs": "Write a single-file three.js page that renders: ", "openscad": "Write the OpenSCAD program that builds this object: ", "cadquery": "Write the CadQuery script that builds this part: ", "blender": "Write the Blender Python code that builds this object: ",
- "glsl": "Write the shader described here: "}
-ASK_IMG = {"web": "These are frames of one animation. Write the single-file animated web page that reproduces it.", "threejs": "These are reference renders of one scene. Write the single-file three.js page that renders it.", "openscad": "These are reference renders of one object. Write the OpenSCAD program that builds it.", "cadquery": "These are reference renders of one part. Write the CadQuery script that builds it.", "blender": "These are reference renders of one object. Write the Blender Python code that builds it.",
- "glsl": "These are frames rendered by one shader. Write the shader that produces them."}
-ASK_BOTH = {"web": "These are frames of one animation. Write the single-file animated web page that reproduces it.\n\nThe animation: ", "threejs": "These are reference renders of one scene. Write the single-file three.js page that renders it.\n\nThe scene: ", "openscad": "These are reference renders of one object. Write the OpenSCAD program that builds it.\n\nThe object: ", "cadquery": "These are reference renders of one part. Write the CadQuery script that builds it.\n\nThe part: ", "blender": "These are reference renders of one object. Write the Blender Python code that builds it.\n\nThe object: ",
- "glsl": "These are frames rendered by one shader. Write the shader that produces them.\n\nThe shader: "}
-IMPERATIVE = ("write", "create", "implement", "make", "build", "generate", "design", "produce")
-
-
-def _to_noun_phrase(v):
- """the ASK_* strings already supply the verb, so an imperative caption is reduced to what follows it"""
- low = v.lower()
- if low.startswith(IMPERATIVE):
- for sep in (" that builds ", " that renders ", " that reproduces ", " of ", " for "):
- if sep in low:
- return v[low.index(sep) + len(sep):].strip()[:1200]
- return None
- return v[:1200]
-
-
-def captions_of(caps):
- """-> [(caption_type, text)] for EVERY usable variant.
-
- One program often carries three descriptions (`instruction`, `detailed`, `factory`) written from different
- angles. Keeping only the best one makes a smaller corpus, and three separate runs here showed the extra
- variants do not raise execution rate — but that is a finding about one training mix, not a property of the
- data, so the dataset ships all of them and a training run decides which to use (`caption_type` selects)."""
- out = []
- for k in ("user_instruction", "instruction", "detailed", "brief", "factory"):
- v = (caps.get(k) or "").strip()
- if not v or len(v) < 15:
- continue
- n = _to_noun_phrase(v)
- if n:
- out.append((k, n))
- return out
-
-
-def fetch_sample(row, token, local_tars=None):
- """-> list of render bytes for one sample, via a local tar if we have it, else a ranged read"""
- tar_rel, start, length = row["tar"], int(row["byte_start"]), int(row["byte_len"])
- if local_tars:
- p = os.path.join(local_tars, os.path.basename(tar_rel))
- if os.path.exists(p):
- with open(p, "rb") as f:
- f.seek(start)
- blob = f.read(length)
- return read_renders(blob)
- url = f"https://huggingface.co/datasets/{REPO}/resolve/main/{tar_rel}"
- hdr = {"Authorization": f"Bearer {token}", "Range": f"bytes={start}-{start + length - 1}"}
- # under a dozen concurrent readers the Hub throttles and times out; returning [] on the first failure looks
- # exactly like "this sample has no renders", which silently cost 70% of the image pairs on the first run
- for attempt in range(4):
- try:
- r = requests.get(url, headers=hdr, timeout=300)
- if r.status_code in (200, 206):
- return read_renders(r.content)
- last = f"HTTP {r.status_code}"
- except Exception as e:
- last = type(e).__name__
- time.sleep(2 ** attempt)
- raise FetchError(last)
-
-
-class FetchError(Exception):
- """the sample could not be read at all — distinct from a sample that genuinely carries no renders"""
-
-
-def read_renders(blob):
- """A byte slice out of the middle of a tar has no end-of-archive marker, so tarfile treats it as truncated —
- and it raises in getmembers(), not in open(), which is why guarding only the open() let one bad sample kill a
- whole run. Append the two zero blocks tar expects, and scan defensively."""
- blob = blob + b"\0" * 1024
- try:
- tf = tarfile.open(fileobj=io.BytesIO(blob))
- except Exception:
- return []
- out = []
- try:
- members = sorted(tf.getmembers(), key=lambda m: m.name)
- except Exception:
- return []
- for m in members:
- if m.isfile() and "/renders/" in m.name and m.name.endswith(".png"):
- try:
- f = tf.extractfile(m)
- if f:
- out.append((os.path.basename(m.name), f.read()))
- except Exception:
- pass
- return out
-
-
-
-def renders_by_scanning(subdir, sample_ids, views, token, local_tars=None, cache_root="/wekafs/ict/hx_624/data/pair_tars"):
- """Index every sample's renders by walking each tar once.
-
- The byte offsets in metadata.parquet cannot be trusted: for 3dcodebench/instances_geo only 577 of 1,953 point
- at a real tar header (checked by reading 512 bytes at each offset and looking for the ustar magic), because
- the archives were repacked after the metadata was written. Sequential scanning ignores the offsets entirely,
- and for a whole-subdirectory conversion it is also cheaper -- one pass per tar instead of one request per
- sample.
- """
- import glob as _glob
- from huggingface_hub import hf_hub_download
- want = {sid.split("/")[-1] for sid in sample_ids}
- found = {}
- tars = sorted({r for r in _tars_of(subdir)})
- for i, tar_rel in enumerate(tars, 1):
- if local_tars:
- path = os.path.join(local_tars, os.path.basename(tar_rel))
- if not os.path.exists(path):
- path = None
- else:
- path = None
- if path is None:
- path = hf_hub_download(REPO, tar_rel, repo_type="dataset", token=token, cache_dir=cache_root)
- n_before = len(found)
- try:
- with tarfile.open(path) as tf:
- for m in tf:
- if not (m.isfile() and "/renders/" in m.name and m.name.endswith(".png")):
- continue
- key = m.name.split("/")[0]
- if key not in want:
- continue
- lst = found.setdefault(key, [])
- if len(lst) >= views:
- continue
- f = tf.extractfile(m)
- if f:
- lst.append((os.path.basename(m.name), f.read()))
- except Exception as e:
- print(f"[pairs] tar {os.path.basename(tar_rel)} unreadable: {type(e).__name__}", flush=True)
- print(f"[pairs] scanned {i}/{len(tars)} {os.path.basename(tar_rel)}: "
- f"+{len(found)-n_before} samples ({len(found):,}/{len(want):,})", flush=True)
- return found
-
-
-def _has_verify_cols(path):
- import pyarrow.parquet as _pq
- try:
- return {"compiles", "repair"} <= set(_pq.ParquetFile(path).schema_arrow.names)
- except Exception:
- return False
-
-
-def _tars_of(subdir):
- from huggingface_hub import HfApi
- api = HfApi(token=os.environ.get("HF_TOKEN", ""))
- return [f for f in api.list_repo_files(REPO, repo_type="dataset")
- if f.startswith(f"{subdir}/") and f.endswith(".tar")]
-
-
-def load_file_tree(subdir, files_root="/wekafs/ict/hx_624/data/3dcodeverse_files"):
- """Sources kept as a directory per sample rather than a parquet: animation2code, blender_distill,
- threejs_distill. Each sample directory holds the code, captions.json, meta.json and a renders/ folder, so the
- renders need no tar scan at all."""
- rows = []
- for cp in sorted(glob.glob(f"{files_root}/{subdir}/*/*/code.py") + glob.glob(f"{files_root}/{subdir}/*/*/code.html")
- + glob.glob(f"{files_root}/{subdir}/*/*/index.html")):
- d = os.path.dirname(cp)
- caps = {}
- cj = os.path.join(d, "captions.json")
- if os.path.exists(cj):
- try:
- caps = json.load(open(cj))
- except Exception:
- caps = {}
- rows.append({"id": os.path.relpath(d, files_root), "code": open(cp, encoding="utf-8", errors="replace").read(),
- "captions": caps, "_renders": sorted(glob.glob(os.path.join(d, "renders", "*.png")))})
- return rows
+import runpy
+import sys
+from pathlib import Path
def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--subdir", required=True)
- ap.add_argument("--out", required=True)
- ap.add_argument("--views", type=int, default=4)
- ap.add_argument("--workers", type=int, default=8)
- ap.add_argument("--limit", type=int, default=None)
- ap.add_argument("--local_tars", default=None)
- ap.add_argument("--img_root", default="/wekafs/ict/hx_624/data/pair_images")
- ap.add_argument("--dialect", default="blender")
- # the compile verification lives in the *_llamafactory parquets; carrying it into the pair datasets is the
- # difference between "here is some GLSL" and "here is GLSL that a compiler accepted"
- # some sources carry no code in metadata at all -- articraft's two URDF subdirs have 6,146 samples whose
- # `code` column is empty because the URDF lives in the tar; it was recovered once into the *_llamafactory
- # parquets, and this reads it back rather than recovering it a second time
- ap.add_argument("--code_from", default=None, help="dir with <name>_llamafactory/train.parquet to take code from")
- ap.add_argument("--verified_from", default=None, help="dir with <name>_llamafactory/train.parquet holding compiles/repair")
- a = ap.parse_args()
- token = os.environ.get("HF_TOKEN", "")
- mp = f"{META}/{a.subdir}/metadata.parquet"
- from_tree = not os.path.exists(mp)
- rows = load_file_tree(a.subdir) if from_tree else pq.read_table(mp).to_pylist()
- if a.limit:
- rows = rows[: a.limit]
- print(f"[pairs] {a.subdir}: {len(rows):,} source samples", flush=True)
-
- from concurrent.futures import ThreadPoolExecutor
- name = a.subdir.replace("/", "__")
- img_dir = os.path.join(a.img_root, a.subdir)
- os.makedirs(img_dir, exist_ok=True)
- qc = collections.Counter()
-
- verified = {}
- if a.verified_from:
- # the verification lives in two places depending on the shard: the 68 that could be flattened carry it in
- # <sub>_llamafactory_flat, the 44 that could not are in the separately verified tree
- base = os.path.basename(a.subdir)
- cands = []
- for root in a.verified_from.split(","):
- cands += [os.path.join(root, f"{base}_llamafactory_flat", "train.parquet"),
- os.path.join(root, f"{base}_llamafactory", "train.parquet")]
- vp = next((c for c in cands if os.path.exists(c) and _has_verify_cols(c)), None)
- if vp:
- vdf = pd.read_parquet(vp, columns=["id", "compiles", "repair", "conversations"])
- for t in vdf.itertuples():
- # one row per caption variant; the code is identical across them, so first wins
- verified.setdefault(str(t.id), (bool(t.compiles), t.repair, t.conversations[1]["value"]))
- print(f"[pairs] verification joined for {len(verified):,} ids from {vp}", flush=True)
- else:
- print(f"[pairs] WARNING no parquet with compiles/repair for {base} — pairs carry no compiles column", flush=True)
- if a.code_from and not verified:
- cp = os.path.join(a.code_from, f"{os.path.basename(a.subdir)}_llamafactory", "train.parquet")
- if os.path.exists(cp):
- cdf = pd.read_parquet(cp, columns=["id", "conversations"])
- for t in cdf.itertuples():
- verified.setdefault(str(t.id), (None, None, t.conversations[1]["value"]))
- print(f"[pairs] code recovered for {len(verified):,} ids from {cp}", flush=True)
- else:
- print(f"[pairs] WARNING --code_from given but {cp} is absent", flush=True)
- ids = [str(r["id"]) for r in rows]
- # a rebuild should not re-download 200 GB of tars just to re-emit rows; if the renders are already extracted
- # for every sample, use them
- on_disk = {}
- for r in rows:
- key = str(r["id"]).replace("/", "_")
- got = sorted(glob.glob(os.path.join(img_dir, key, "*")))[: a.views]
- if got:
- on_disk[str(r["id"]).split("/")[-1]] = [(os.path.basename(x), None) for x in got]
- # a coverage threshold cannot work here: the samples that have no renders never will, so "98% of rows" is
- # unreachable for a source where 12% legitimately lack them. A marker written after a completed scan is the
- # only honest signal that the extraction already ran to the end.
- if from_tree:
- render_index = {str(r["id"]): [(x, None) for x in r["_renders"]] for r in rows}
- print(f"[pairs] file-tree source: renders read in place for "
- f"{sum(1 for v in render_index.values() if v):,}/{len(rows):,} samples", flush=True)
- marker = os.path.join(img_dir, ".scan_complete")
- if from_tree:
- pass
- elif os.path.exists(marker) and on_disk:
- print(f"[pairs] renders already extracted ({len(on_disk):,} samples) — skipping the tar scan", flush=True)
- render_index = on_disk
- else:
- render_index = renders_by_scanning(a.subdir, ids, a.views, token, a.local_tars)
- os.makedirs(img_dir, exist_ok=True)
- open(marker, "w").write(f"{len(render_index)}\n")
-
- def one(r):
- code = r.get("code") or ""
- # the recovered/verified code has to be substituted BEFORE the length test: the URDF subdirs carry an
- # empty `code` column by design, and checking first rejected all 6,146 of them as "short_code"
- v = verified.get(str(r["id"]))
- if v and v[2]:
- code = FENCE_RE.sub("", v[2].strip()) or code
- if len(code.strip()) < 40:
- qc["short_code"] += 1
- return None
- # LLaMA-Factory counts <image> placeholders across EVERY turn, so a literal "<image>" in the answer
- # (Shadertoy writes "// <image>" as a multipass section marker) reads as one more image than the row
- # carries and aborts the whole run hours in. Seven such rows shipped once; drop them at the source.
- if "<image>" in code:
- qc["literal_image_token"] += 1
- return None
- caps = r.get("captions")
- caps = dict(caps) if isinstance(caps, dict) else (json.loads(caps) if isinstance(caps, str) else {})
- caps_list = captions_of(caps)
- renders = render_index.get(str(r["id"]) if from_tree else str(r["id"]).split("/")[-1], [])[: a.views]
- paths = []
- for fname, blob in renders:
- d = os.path.join(img_dir, str(r["id"]).replace("/", "_"))
- p = fname if os.path.isabs(fname) else os.path.join(d, fname)
- if blob is not None and not os.path.exists(p):
- os.makedirs(d, exist_ok=True)
- open(p, "wb").write(blob)
- if os.path.exists(p):
- paths.append(p)
- if not paths:
- qc["no_renders"] += 1
- if not caps_list:
- qc["no_caption"] += 1
- return {"id": str(r["id"]), "code": code, "captions": caps_list, "images": paths,
- "compiles": (v[0] if v else None), "repair": (v[1] if v else None)}
-
- with ThreadPoolExecutor(a.workers) as ex:
- got = [x for x in ex.map(one, rows) if x]
- print(f"[pairs] usable samples: {len(got):,} | qc={dict(qc)}", flush=True)
- if qc["fetch_failed"]:
- print(f"[pairs] WARNING {qc['fetch_failed']:,} samples could not be fetched after 4 attempts — their image "
- f"pairs are MISSING, not absent by nature. Re-run to fill them in.", flush=True)
-
- d, fence = a.dialect, FENCE.get(a.dialect, "python")
- out = {"text": [], "img": [], "imgtext": []}
- for g in got:
- gpt = {"from": "gpt", "value": f"```{fence}\n{g['code'].strip()}\n```"}
- base = {"id": g["id"], "system": SYS[d]}
- if g["compiles"] is not None:
- base["compiles"] = g["compiles"]; base["repair"] = g["repair"]
- for ctype, cap in g["captions"]:
- out["text"].append(dict(base, caption_type=ctype,
- conversations=[{"from": "human", "value": ASK_TEXT[d] + cap}, gpt]))
- if g["images"]:
- ph = "<image>" * len(g["images"])
- # image->code has no caption to vary, so it stays one row per sample by construction
- out["img"].append(dict(base, caption_type=None, images=list(g["images"]),
- conversations=[{"from": "human", "value": ph + "\n" + ASK_IMG[d]}, gpt]))
- for ctype, cap in g["captions"]:
- out["imgtext"].append(dict(base, caption_type=ctype, images=list(g["images"]),
- conversations=[{"from": "human", "value": ph + "\n" + ASK_BOTH[d] + cap}, gpt]))
- info = {}
- for kind, rowsk in out.items():
- if not rowsk:
- print(f"[pairs] {kind}: 0 rows — not written", flush=True)
- continue
- folder = os.path.join(a.out, f"{os.path.basename(a.subdir)}_llamafactory_{kind}")
- os.makedirs(folder, exist_ok=True)
- pd.DataFrame(rowsk).to_parquet(os.path.join(folder, "train.parquet"), index=False)
- cols = {"messages": "conversations", "system": "system"}
- if kind != "text":
- cols["images"] = "images"
- entry = {"file_name": "train.parquet", "formatting": "sharegpt", "columns": cols}
- json.dump({f"3dcv_{name}_{kind}": entry}, open(os.path.join(folder, "dataset_info.json"), "w"), indent=2)
- json.dump({"subdir": a.subdir, "pair_type": kind, "rows": len(rowsk),
- "source_samples": len(rows), "usable": len(got), **{k: v for k, v in qc.items()}},
- open(os.path.join(folder, "qc.json"), "w"), indent=2)
- info[kind] = len(rowsk)
- print(f"[pairs] {kind:8s} {len(rowsk):>7,} rows -> {folder}", flush=True)
- # the invariant LLaMA-Factory enforces on multimodal rows
- bad = sum(1 for k in ("img", "imgtext") for r in out[k]
- if r["conversations"][0]["value"].count("<image>") != len(r["images"]))
- print(f"[pairs] placeholder/image mismatches: {bad} (must be 0)", flush=True)
+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"
+ sys.path.insert(0, str(scripts))
+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")
if __name__ == "__main__":
diff --git a/finetune/scripts/convert_subdirs.py b/finetune/scripts/convert_subdirs.py
index c63f1a7a..a6f32161 100644
--- a/finetune/scripts/convert_subdirs.py
+++ b/finetune/scripts/convert_subdirs.py
@@ -1,300 +1,14 @@
-"""Convert EVERY 3DCodeVerse subdirectory into its own LLaMA-Factory dataset + a data-quality report.
+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""
-One output folder per source subdir, named `<subdir>_llamafactory/`, so it can sit next to the source on the
-Hub (e.g. `3dcodebench/factories_geo/` → `3dcodebench/factories_geo_llamafactory/`):
-
- <subdir>_llamafactory/
- train.json ShareGPT: [{system, conversations:[human, gpt]}] — one row per caption variant
- dataset_info.json drop-in LLaMA-Factory registry entry for this folder
- qc.json per-subdir quality counters + token stats
- README.md what it is, how many rows, what was dropped and why
-
-Quality checks applied per sample (a sample can fail several):
- empty_code / short_code (<40 chars) / bad_chars (control chars or U+FFFD) / no_caption /
- duplicate_code (same code hash inside the subdir) / syntax_bad (AST for Python dialects, brace balance for
- scad-glsl, tag check for html) / too_long (>8192 tokens, still exported but flagged) / media_placeholder
- (literal <image>/<video>/<audio> — breaks the qwen3_5 template unless escaped; we escape it and flag it)
-
-usage: python scripts/convert_subdirs.py [--out DIR] [--workers 16] [--limit_subdirs N]
-"""
-import argparse
-import ast
-import collections
-import glob
-import hashlib
-import json
-import os
-import re
+import runpy
import sys
-import unicodedata
-from concurrent.futures import ProcessPoolExecutor
-
-import pandas as pd
-
-sys.path.insert(0, "/wekafs/ict/hx_624/llm-ft/scripts")
-META = "/wekafs/ict/hx_624/data/3dcodeverse_meta"
-FILES = "/wekafs/ict/hx_624/data/3dcodeverse_files"
-BENCH_DIR = "/wekafs/ict/hx_624/data/3dcodebench/data"
-
-# subdir prefix -> dialect
-DIALECT_OF = {
- "3dcodebench": "blender", "bioinspired3d": "blender", "blender_distill": "blender",
- "deepcad": "cadquery", "articraft": "cadquery", "shadertoy": "glsl",
- "thingiverse": "openscad", "threejs_distill": "threejs", "animation2code": "web",
-}
-SYS = {
- "blender": "You are an expert in procedural 3D modeling with Blender Python (bpy). Given a description of an object, write a complete, standalone Blender 5 Python script that builds it from scratch (clear the default scene first). Output ONLY the code in one ```python block.",
- "cadquery": "You are an expert CAD programmer. Given a description of a part, write a complete, standalone CadQuery (Python) script that builds it and leaves the final solid in a variable named `result`. Output ONLY the code in one ```python block.",
- "openscad": "You are an expert OpenSCAD programmer. Given a request, write a complete, standalone OpenSCAD (.scad) program that produces the described model. Output ONLY the code in one ```openscad block.",
- "glsl": "You are an expert shader programmer. Given a description, write a complete Shadertoy-style GLSL ES 3.00 fragment shader defining `void mainImage(out vec4 fragColor, in vec2 fragCoord)`. Output ONLY the code in one ```glsl block.",
- "threejs": "You are an expert three.js developer. Given a description, write a complete, standalone HTML page that renders the described 3D scene with three.js. Output ONLY the code in one ```html block.",
- "web": "You are an expert creative web developer. Given a description of an animation, write a complete, standalone HTML page (inline CSS/JS, no local assets) that reproduces it. Output ONLY the code in one ```html block.",
- "urdf": "You are an expert in articulated 3D assets. Given a description, write a complete, self-contained URDF file for it. Output ONLY the code in one ```xml block.",
-}
-FENCE = {"blender": "python", "cadquery": "python", "openscad": "openscad", "glsl": "glsl",
- "threejs": "html", "web": "html", "urdf": "xml"}
-LEAD = {
- "blender": "Create the 3D object described below using Python Blender code: ",
- "cadquery": "Write a CadQuery (Python) script that builds the following CAD model: ",
- "openscad": "Write an OpenSCAD program for the following: ",
- "glsl": "Write a GLSL fragment shader for Shadertoy that renders: ",
- "threejs": "Write a standalone three.js page that renders: ",
- "web": "Write a standalone animated web page that reproduces: ",
- "urdf": "Write a URDF description of: ",
-}
-IMPERATIVE = ("write", "create", "implement", "make", "build", "generate", "design", "produce")
-MEDIA = re.compile(r"<(image|video|audio)>")
-
-
-def bad_char_count(s):
- return sum(1 for ch in s if unicodedata.category(ch) == "Cc" and ch not in "\n\t\r") + s.count("�")
-
-
-def syntax_ok(code, dialect):
- if dialect in ("blender", "cadquery"):
- try:
- ast.parse(code)
- return True
- except Exception:
- return False
- if dialect == "openscad":
- return code.count("{") == code.count("}") and code.count("(") == code.count(")")
- if dialect == "glsl":
- return code.count("{") == code.count("}") and "void" in code
- if dialect in ("threejs", "web"):
- return "<" in code and ">" in code
- if dialect == "urdf":
- return "<robot" in code or "<link" in code
- return True
-
-
-def phrase(dialect, kind, text):
- t = (text or "").strip()
- if not t:
- return None
- if kind == "instruction" and t.lower().startswith(IMPERATIVE):
- return t
- return LEAD[dialect] + t
-
-
-def normalize_caps(caps):
- """tree sources use their own caption keys"""
- caps = {k: v for k, v in caps.items() if isinstance(v, str) and v.strip()}
- if any(k in caps for k in ("instruction", "detailed", "factory")) and "user_instruction" not in caps and "brief" not in caps:
- return caps
- norm = {}
- if caps.get("user_instruction"):
- norm["instruction"] = caps["user_instruction"]
- if caps.get("instruction"):
- norm["detailed"] = caps["instruction"]
- if caps.get("brief"):
- extra = caps.get("function", "")
- norm["instruction"] = (caps["brief"] + ((" — " + extra) if extra and extra not in caps["brief"] else "")).strip()
- if caps.get("source_title") and "detailed" not in norm:
- norm["detailed"] = caps["source_title"]
- if caps.get("style") and norm.get("detailed"):
- norm["detailed"] += f" (style: {caps['style']})"
- return norm or caps
-
-
-def load_subdir(sub):
- """-> (dialect, [ {id, name, code, caps, n_images} ]) for a parquet subdir or a file tree"""
- top = sub.split("/")[0]
- dialect = DIALECT_OF.get(top, "blender")
- if top == "articraft" and "urdf" in sub:
- dialect = "urdf"
- rows = []
- p = f"{META}/{sub}/metadata.parquet"
- if os.path.exists(p):
- df = pd.read_parquet(p)
- for r in df.itertuples():
- caps = r.captions if isinstance(r.captions, dict) else (json.loads(r.captions) if isinstance(r.captions, str) else {})
- meta = {}
- try:
- meta = json.loads(r.meta_json) if isinstance(r.meta_json, str) else (r.meta_json or {})
- except Exception:
- pass
- rows.append({"id": str(r.id), "name": str(getattr(r, "name", "") or ""),
- "code": getattr(r, "code", None) or "", "caps": dict(caps),
- "n_images": len(meta.get("renders", []) or [])})
- return dialect, rows
- # file tree
- for cp in sorted(glob.glob(f"{FILES}/{sub}/*/code.py") + glob.glob(f"{FILES}/{sub}/*/code.html")
- + glob.glob(f"{FILES}/{sub}/*/index.html") + glob.glob(f"{FILES}/{sub}/*/*/code.py")
- + glob.glob(f"{FILES}/{sub}/*/*/code.html") + glob.glob(f"{FILES}/{sub}/*/*/index.html")):
- d = os.path.dirname(cp)
- try:
- code = open(cp, encoding="utf-8", errors="replace").read()
- caps = json.load(open(os.path.join(d, "captions.json"))) if os.path.exists(os.path.join(d, "captions.json")) else {}
- except Exception:
- continue
- rows.append({"id": os.path.relpath(d, FILES), "name": os.path.basename(d), "code": code,
- "caps": caps, "n_images": len(glob.glob(os.path.join(d, "renders", "*.png")))})
- return dialect, rows
-
-
-_TOK = None
-
-
-def _init():
- global _TOK
- from transformers import AutoTokenizer
- _TOK = AutoTokenizer.from_pretrained("/wekafs/ict/hx_624/models/Qwen3.5-9B")
-
-
-def convert_one(args):
- sub, out_root, bench_factories = args
- dialect, rows = load_subdir(sub)
- if not rows:
- return {"subdir": sub, "error": "no rows"}
- qc = collections.Counter()
- seen_code, out, tok_lens, img_counts = {}, [], [], []
- for r in rows:
- code = r["code"] if isinstance(r["code"], str) else ""
- img_counts.append(r.get("n_images", 0))
- if not code.strip():
- qc["empty_code"] += 1
- continue
- if len(code.strip()) < 40:
- qc["short_code"] += 1
- continue
- if bad_char_count(code) > 5:
- qc["bad_chars"] += 1
- continue
- h = hashlib.md5(code.encode()).hexdigest()
- if h in seen_code:
- qc["duplicate_code"] += 1
- continue
- seen_code[h] = 1
- if not syntax_ok(code, dialect):
- qc["syntax_bad"] += 1 # exported anyway, but counted
- if MEDIA.search(code) or any(MEDIA.search(v) for v in r["caps"].values() if isinstance(v, str)):
- qc["media_placeholder"] += 1
- code = MEDIA.sub(lambda m: f"<{m.group(1)} >", code)
- norm_name = r["name"].replace("Factory", "").replace("_geo", "").replace("_tex", "")
- if r["name"] in bench_factories or r["name"].rsplit("_", 1)[0] in bench_factories or norm_name in bench_factories:
- # counted, never dropped: which samples a benchmark forbids is a decision for whoever defines the
- # benchmark, and a converter that quietly removes 100% of a subdirectory (as this once did for both
- # 3dcodebench/instances_*) is the wrong place to make it
- qc["benchmark_overlap"] += 1
- caps = normalize_caps(r["caps"])
- n_before = len(out)
- for kind in ("instruction", "detailed", "factory"):
- pr = phrase(dialect, kind, caps.get(kind))
- if not pr or len(pr) < 20:
- continue
- pr = MEDIA.sub(lambda m: f"<{m.group(1)} >", pr)
- n = len(_TOK(pr).input_ids) + len(_TOK(code).input_ids) + 60
- tok_lens.append(n)
- if n > 8192:
- qc["too_long_gt8192"] += 1
- out.append({"id": r["id"], "caption_type": kind, "n_tokens": n,
- "system": SYS[dialect],
- "conversations": [{"from": "human", "value": pr},
- {"from": "gpt", "value": f"```{FENCE[dialect]}\n{code.strip()}\n```"}]})
- if len(out) == n_before:
- qc["no_caption"] += 1
- name = sub.replace("/", "__")
- d = os.path.join(out_root, sub + "_llamafactory")
- os.makedirs(d, exist_ok=True)
- json.dump(out, open(os.path.join(d, "train.json"), "w"), ensure_ascii=False)
- key = f"3dcv_{name}"
- json.dump({key: {"file_name": "train.json", "formatting": "sharegpt",
- "columns": {"messages": "conversations", "system": "system"}}},
- open(os.path.join(d, "dataset_info.json"), "w"), indent=2)
- tok_lens.sort()
- stat = {"subdir": sub, "dialect": dialect, "source_samples": len(rows), "exported_pairs": len(out),
- "unique_samples_exported": len(seen_code),
- "images_per_sample_median": (sorted(img_counts)[len(img_counts) // 2] if img_counts else 0),
- "tokens_p50": tok_lens[len(tok_lens) // 2] if tok_lens else 0,
- "tokens_p90": tok_lens[int(len(tok_lens) * 0.9)] if tok_lens else 0,
- "tokens_max": tok_lens[-1] if tok_lens else 0,
- **{k: v for k, v in qc.items()}}
- json.dump(stat, open(os.path.join(d, "qc.json"), "w"), indent=2)
- issues = "\n".join(f"- **{k}**: {v}" for k, v in sorted(qc.items(), key=lambda x: -x[1])) or "- none"
- open(os.path.join(d, "README.md"), "w").write(f"""# {sub} — LLaMA-Factory format
-
-Generated by `finetune/scripts/convert_subdirs.py` from `{sub}/metadata.parquet` (or its file tree).
-
-| | |
-|---|---|
-| dialect | `{dialect}` |
-| source samples | {len(rows):,} |
-| exported pairs | {len(out):,} (one row per caption variant: instruction / detailed / factory) |
-| unique code samples exported | {stat['unique_samples_exported']:,} |
-| token length p50 / p90 / max | {stat['tokens_p50']} / {stat['tokens_p90']} / {stat['tokens_max']} |
-| renders per sample (median) | {stat['images_per_sample_median']} |
-
-## Data-quality findings
-
-{issues}
-
-`syntax_bad` rows are still exported (a shader can be valid without balanced braces in our crude check); every
-other counter means the sample was dropped, EXCEPT `benchmark_overlap`, which is counted and kept: the
-sample belongs to one of the 212
-3DCodeBench evaluation factories and is excluded to avoid test leakage.
-
-## Use
-
-```yaml
-dataset_dir: <this folder>
-dataset: {key}
-```
-""")
- return stat
+from pathlib import Path
def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--out", default="/wekafs/ict/hx_624/hf_subdirs")
- ap.add_argument("--workers", type=int, default=16)
- ap.add_argument("--limit_subdirs", type=int, default=None)
- ap.add_argument("--only", default=None, help="comma-separated subdirs to convert")
- a = ap.parse_args()
- bench = frozenset(os.path.basename(p).replace("_seed0", "") for p in glob.glob(f"{BENCH_DIR}/*_seed0"))
- subs = ["/".join(p.split("/")[-3:-1]) for p in sorted(glob.glob(f"{META}/*/*/metadata.parquet"))]
- subs += ["blender_distill", "threejs_distill", "animation2code"]
- if a.only:
- want = {x.strip() for x in a.only.split(",") if x.strip()}
- subs = [x for x in subs if x in want]
- if a.limit_subdirs:
- subs = subs[: a.limit_subdirs]
- print(f"[conv] {len(subs)} subdirs -> {a.out}", flush=True)
- os.makedirs(a.out, exist_ok=True)
- stats = []
- with ProcessPoolExecutor(a.workers, initializer=_init) as ex:
- for st in ex.map(convert_one, [(s, a.out, bench) for s in subs]):
- stats.append(st)
- print(f"[conv] {st.get('subdir'):45s} pairs={st.get('exported_pairs', 0):7,} "
- f"issues={ {k: v for k, v in st.items() if k in ('empty_code','short_code','bad_chars','duplicate_code','syntax_bad','no_caption','too_long_gt8192','media_placeholder','benchmark_overlap')} }", flush=True)
- df = pd.DataFrame(stats)
- df.to_csv(os.path.join(a.out, "quality_report.csv"), index=False)
- try:
- df.to_excel(os.path.join(a.out, "quality_report.xlsx"), index=False)
- except Exception as e:
- print("[conv] xlsx skipped:", e)
- json.dump(stats, open(os.path.join(a.out, "quality_report.json"), "w"), indent=2)
- print(f"[conv] wrote quality_report.{{csv,xlsx,json}} | total exported pairs: {int(df['exported_pairs'].sum()):,}")
+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"
+ sys.path.insert(0, str(scripts))
+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")
if __name__ == "__main__":
diff --git a/finetune/scripts/drop_old_pair_trees.py b/finetune/scripts/drop_old_pair_trees.py
index 3e92b10d..a6f32161 100644
--- a/finetune/scripts/drop_old_pair_trees.py
+++ b/finetune/scripts/drop_old_pair_trees.py
@@ -1,26 +1,15 @@
-"""Remove the superseded top-level pairs/ and pair_images/ once their replacements are registered.
+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""
-Both were mine: the pair datasets were first published into a separate top-level tree, which read as if it were
-a source of its own. They now live at <source>/<subdir>_llamafactory_{text,img,imgtext} and
-<source>/pair_renders/, and dataset_info.json points only at those, so the old tree is safe to drop.
-"""
-import os, json
-from huggingface_hub import HfApi, hf_hub_download, CommitOperationDelete
-tok = os.environ["HF_TOKEN"]; REPO = "ilabai/3dcodeverse"; api = HfApi(token=tok)
-agg = json.load(open(hf_hub_download(REPO, "dataset_info.json", repo_type="dataset", token=tok)))
-still = [k for k, v in agg.items() if str(v.get("folder", "")).startswith(("pairs/", "pair_images/"))]
-if still:
- raise SystemExit(f"REFUSING: {len(still)} index entries still point into the old tree")
-for d in ("pairs", "pair_images"):
- files = [e.path for e in api.list_repo_tree(REPO, repo_type="dataset", path_in_repo=d, recursive=True)
- if e.__class__.__name__.startswith("RepoFile")]
- print(f"[drop] {d}/: {len(files):,} files", flush=True)
- B = 400
- for i in range(0, len(files), B):
- api.create_commit(repo_id=REPO, repo_type="dataset",
- operations=[CommitOperationDelete(path_in_repo=f) for f in files[i:i+B]],
- commit_message=f"remove the superseded {d}/ ({i+1}-{min(i+B, len(files))} of {len(files)})")
- if (i // B) % 20 == 0:
- print(f"[drop] {min(i+B, len(files)):,}/{len(files):,}", flush=True)
- print(f"[drop] {d}/ removed", flush=True)
-print("[drop] DONE", flush=True)
+import runpy
+import sys
+from pathlib import Path
+
+
+def main():
+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"
+ sys.path.insert(0, str(scripts))
+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/finetune/scripts/shrink_pair_images.py b/finetune/scripts/shrink_pair_images.py
index 1f4b0e49..a6f32161 100644
--- a/finetune/scripts/shrink_pair_images.py
+++ b/finetune/scripts/shrink_pair_images.py
@@ -1,57 +1,14 @@
-"""Re-encode the extracted renders to training resolution.
+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""
-The originals average 220 KB (640x360 or 1024x1024 PNG) and total 84 GB, which is impractical to ship beside a
-dataset. Every vision tower we train downsamples to 448x448 anyway (our own 27B run set image_max_pixels to
-448^2), so a <=448px JPEG loses nothing a training run would have seen — and the untouched originals stay in the
-corpus tars for anyone who needs them. Measured saving: 95%, so 84 GB becomes about 4 GB.
-
-usage: python scripts/shrink_pair_images.py [--root DIR] [--out DIR] [--max_px 448] [--quality 88] [--workers 64]
-"""
-import argparse
-import glob
-import os
-from concurrent.futures import ProcessPoolExecutor
-
-from PIL import Image
-
-
-def one(args):
- src, dst, max_px, q = args
- if os.path.exists(dst):
- return 0, 0, 0
- try:
- im = Image.open(src)
- im = im.convert("RGB")
- im.thumbnail((max_px, max_px), Image.LANCZOS)
- os.makedirs(os.path.dirname(dst), exist_ok=True)
- im.save(dst, "JPEG", quality=q, optimize=True)
- return os.path.getsize(src), os.path.getsize(dst), 1
- except Exception:
- return 0, 0, -1
+import runpy
+import sys
+from pathlib import Path
def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--root", default="/wekafs/ict/hx_624/data/pair_images")
- ap.add_argument("--out", default="/wekafs/ict/hx_624/data/pair_images_small")
- ap.add_argument("--max_px", type=int, default=448)
- ap.add_argument("--quality", type=int, default=88)
- ap.add_argument("--workers", type=int, default=64)
- a = ap.parse_args()
- srcs = glob.glob(os.path.join(a.root, "**", "*.png"), recursive=True)
- print(f"[shrink] {len(srcs):,} renders -> {a.out} at <={a.max_px}px q{a.quality}", flush=True)
- jobs = [(s, os.path.join(a.out, os.path.relpath(s, a.root)).rsplit(".", 1)[0] + ".jpg", a.max_px, a.quality)
- for s in srcs]
- big = small = done = failed = 0
- with ProcessPoolExecutor(a.workers) as ex:
- for i, (b, s, ok) in enumerate(ex.map(one, jobs, chunksize=64), 1):
- big += b; small += s
- if ok == 1: done += 1
- elif ok < 0: failed += 1
- if i % 50000 == 0:
- print(f"[shrink] {i:,}/{len(jobs):,} | {big/1e9:.1f} GB -> {small/1e9:.1f} GB | failed {failed}", flush=True)
- print(f"[shrink] DONE {done:,} written, {failed} failed | {big/1e9:.1f} GB -> {small/1e9:.1f} GB "
- f"({(1-small/max(big,1))*100:.1f}% smaller)", flush=True)
+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"
+ sys.path.insert(0, str(scripts))
+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")
if __name__ == "__main__":
diff --git a/finetune/scripts/sync_repo.sh b/finetune/scripts/sync_repo.sh
index 8f3f5a67..fc52a7eb 100755
--- a/finetune/scripts/sync_repo.sh
+++ b/finetune/scripts/sync_repo.sh
@@ -3,7 +3,9 @@
# scripts/sync_repo.sh "commit message" (default message if omitted)
#
# What is synced (code + docs only):
-# scripts/ configs/ eval/ env.sh -> finetune/
+# shell scripts/ configs/ eval/ env.sh -> finetune/
+# Python data tools are maintained in the repo's finetune/llamafactory/;
+# their compatibility launchers in scripts/ must not be overwritten by old copies.
# REPORT.md -> finetune/docs/REPORT.md
# README.md -> finetune/docs/PROJECT_README.md
# llm_finetune_exps.md -> finetune/llm_finetune_exps.md
@@ -18,7 +20,10 @@ DST=$REPO/finetune
MSG=${1:-"finetune: sync scripts, configs, eval stack and docs"}
[ -d "$DST" ] || { echo "repo clone missing at $REPO"; exit 1; }
-rsync -a --delete --exclude '__pycache__' --exclude '*.pyc' "$SRC/scripts/" "$DST/scripts/"
+# Preserve the repo-owned entrypoints/docs and this script itself, so an older
+# external sync script cannot restore the pre-consolidation mirroring rules.
+rsync -a --delete --exclude '__pycache__' --exclude '*.pyc' --exclude '*.py' \
+ --exclude '/sync_repo.sh' --exclude '/README.md' "$SRC/scripts/" "$DST/scripts/"
# --delete on configs too: without it a config retired into configs/lf/archive/ was copied to its new
# location but the old copy lingered in the repo forever, so a tidy-up never actually reached GitHub.
rsync -a --delete --exclude '__pycache__' "$SRC/configs/" "$DST/configs/"
diff --git a/finetune/scripts/upload_pairs.py b/finetune/scripts/upload_pairs.py
index 9af73627..a6f32161 100644
--- a/finetune/scripts/upload_pairs.py
+++ b/finetune/scripts/upload_pairs.py
@@ -1,96 +1,14 @@
-"""Publish the pair datasets INTO their source folders, not into a separate top-level tree.
+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""
-Every converted dataset in this repo lives at <source>/<subdir>_llamafactory; the pair datasets follow the same
-rule as <source>/<subdir>_llamafactory_{text,img,imgtext}, and their renders as <source>/pair_renders/... . A
-top-level pairs/ or pair_images/ would read as if it were a source of its own, which is the same mistake the
-five stray *_llamafactory folders made.
-
-usage: python scripts/upload_pairs.py [--batch 24] [--images] [--drop_old]
-"""
-import argparse
-import glob
-import json
-import os
-
-from huggingface_hub import HfApi, CommitOperationAdd, CommitOperationDelete, hf_hub_download
-
-REPO = "ilabai/3dcodeverse"
-ROOTS = ("/wekafs/ict/hx_624/hf_pairs_shadertoy", "/wekafs/ict/hx_624/hf_pairs", "/wekafs/ict/hx_624/hf_pairs_other")
-# the two sources the user renamed after these were built
-RENAME = {"blender_distill": "blender_coding_agent", "threejs_distill": "threejs_coding_agent"}
-
-
-def collect():
- out = []
- for root in ROOTS:
- for d in sorted(glob.glob(f"{root}/*_llamafactory_*")):
- q = json.load(open(f"{d}/qc.json"))
- if q["rows"] == 0:
- continue
- src = RENAME.get(q["subdir"].split("/")[0], q["subdir"].split("/")[0])
- out.append((d, f"{src}/{os.path.basename(d)}", q))
- return out
+import runpy
+import sys
+from pathlib import Path
def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--batch", type=int, default=24)
- ap.add_argument("--images", action="store_true")
- ap.add_argument("--drop_old", action="store_true", help="remove the earlier top-level pairs/ and pair_images/")
- a = ap.parse_args()
- api = HfApi(token=os.environ["HF_TOKEN"])
- items = collect()
- print(f"[up] {len(items)} pair datasets, {sum(q['rows'] for _, _, q in items):,} rows", flush=True)
-
- for i in range(0, len(items), a.batch):
- chunk = items[i:i + a.batch]
- ops = []
- for d, path, _ in chunk:
- for f in ("train.parquet", "dataset_info.json", "qc.json"):
- if os.path.exists(f"{d}/{f}"):
- ops.append(CommitOperationAdd(path_in_repo=f"{path}/{f}", path_or_fileobj=f"{d}/{f}"))
- api.create_commit(repo_id=REPO, repo_type="dataset", operations=ops,
- commit_message=f"pairs: publish {i+1}-{i+len(chunk)} of {len(items)} under their sources")
- print(f"[up] pushed {i+1}-{i+len(chunk)}", flush=True)
-
- p = hf_hub_download(REPO, "dataset_info.json", repo_type="dataset", token=os.environ["HF_TOKEN"])
- agg = json.load(open(p))
- before = len(agg)
- # drop the entries that pointed into the old top-level tree, then register the new locations
- agg = {k: v for k, v in agg.items() if not str(v.get("folder", "")).startswith("pairs/")}
- for d, path, q in items:
- info = json.load(open(f"{d}/dataset_info.json"))
- for name, attr in info.items():
- agg[name] = {"hf_hub_url": REPO, "folder": path, "formatting": "sharegpt", "split": "train",
- "columns": attr.get("columns", {"messages": "conversations", "system": "system"})}
- out = "/tmp/claude-1035/-wekafs-ict-hx-624/8a069962-27e5-496d-b000-d4e3dd4dac29/scratchpad/dataset_info_v2.json"
- json.dump(agg, open(out, "w"), indent=2, ensure_ascii=False, sort_keys=True)
- api.upload_file(path_or_fileobj=out, path_in_repo="dataset_info.json", repo_id=REPO, repo_type="dataset",
- commit_message=f"dataset_info: point the pair datasets at their source folders ({before} -> {len(agg)})")
- print(f"[up] index {before} -> {len(agg)} entries", flush=True)
-
- if a.images:
- for src in sorted(os.listdir("/wekafs/ict/hx_624/data/pair_images_small")):
- d = f"/wekafs/ict/hx_624/data/pair_images_small/{src}"
- if not os.path.isdir(d):
- continue
- dst = f"{RENAME.get(src, src)}/pair_renders"
- print(f"[up] renders {src} -> {dst}", flush=True)
- api.upload_folder(folder_path=d, path_in_repo=dst, repo_id=REPO, repo_type="dataset",
- commit_message=f"{dst}: renders at training resolution")
-
- if a.drop_old:
- for old in ("pairs", "pair_images"):
- try:
- es = list(api.list_repo_tree(REPO, repo_type="dataset", path_in_repo=old, recursive=True))
- files = [e.path for e in es if e.__class__.__name__.startswith("RepoFile")]
- for i in range(0, len(files), 200):
- api.create_commit(repo_id=REPO, repo_type="dataset",
- operations=[CommitOperationDelete(path_in_repo=f) for f in files[i:i+200]],
- commit_message=f"remove the superseded top-level {old}/ ({i+1}-{min(i+200,len(files))})")
- print(f"[up] removed {len(files)} files from {old}/", flush=True)
- except Exception as e:
- print(f"[up] {old}/: {type(e).__name__} {str(e)[:80]}", flush=True)
+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"
+ sys.path.insert(0, str(scripts))
+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")
if __name__ == "__main__":
diff --git a/finetune/scripts/verify_unflattened_glsl.py b/finetune/scripts/verify_unflattened_glsl.py
index 5bb8dc33..a6f32161 100644
--- a/finetune/scripts/verify_unflattened_glsl.py
+++ b/finetune/scripts/verify_unflattened_glsl.py
@@ -1,107 +1,14 @@
-"""Compile-verify the 44 shadertoy shards that the flattening pass never reached.
+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""
-Those shards' metadata carries no `shader_json`, so there was no renderpass structure to merge a Common tab
-from and `flatten_multifile.py` skipped them — exactly 44 of 112, with zero mismatches. The consequence is that
-73,281 shaders are verified (99.04% compile) while 46,680 have never been near a compiler. This runs the same
-checker and the same minimal repairs over those, and writes the `compiles` / `repair` columns the flattened
-folders already carry, so a training mix can filter the whole corpus by one rule instead of two.
-
-usage: python scripts/verify_unflattened_glsl.py [--workers 40] [--limit_shards N] [--out DIR]
-"""
-import argparse
-import collections
-import glob
-import json
-import os
-import re
+import runpy
import sys
-
-import pandas as pd
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from qc_repair import glsl_compile, repair_glsl
-
-SRC = "/wekafs/ict/hx_624/hf_subdirs/shadertoy"
-FENCE = re.compile(r"^```[a-zA-Z]*\n|\n?```$")
-
-
-def strip_fence(v):
- return FENCE.sub("", v.strip())
-
-
-def verify_one(code):
- """-> (compiles, repair_label, code_to_keep). Never returns a program that does not compile as 'repaired'."""
- ok, _ = glsl_compile(code)
- if ok:
- return True, "original", code
- try:
- fixed, recipe = repair_glsl(code)
- except Exception:
- return False, "unfixable", code
- if fixed and fixed != code:
- ok2, _ = glsl_compile(fixed)
- if ok2:
- return True, f"repaired:{recipe or 'other'}", fixed
- return False, "unfixable", code
-
-
-def do_shard(args):
- sub, out_root = args
- p = f"{SRC}/{sub}_llamafactory/train.parquet"
- df = pd.read_parquet(p)
- # one shader appears once per caption variant; compile each distinct program once
- by_code = {}
- codes = [strip_fence(c[1]["value"]) for c in df["conversations"]]
- for c in set(codes):
- by_code[c] = verify_one(c)
- okc, howc, conv = [], [], []
- for row, c in zip(df.itertuples(), codes):
- ok, how, keep = by_code[c]
- okc.append(ok); howc.append(how)
- turns = [dict(t) for t in row.conversations]
- turns[1] = dict(turns[1], value=f"```glsl\n{keep}\n```")
- conv.append(turns)
- df["conversations"] = conv
- df["compiles"] = okc
- df["repair"] = howc
- d = os.path.join(out_root, f"{sub}_llamafactory")
- os.makedirs(d, exist_ok=True)
- df.to_parquet(os.path.join(d, "train.parquet"), index=False)
- for extra in ("dataset_info.json", "qc.json", "README.md"):
- s = f"{SRC}/{sub}_llamafactory/{extra}"
- if os.path.exists(s):
- open(os.path.join(d, extra), "w").write(open(s).read())
- st = {"shard": sub, "rows": len(df), "unique_programs": len(by_code),
- "compiles": int(sum(okc)), "repaired": int(sum(1 for h in howc if h.startswith("repaired"))),
- "unfixable": int(sum(1 for h in howc if h == "unfixable"))}
- json.dump(st, open(os.path.join(d, "verify.json"), "w"), indent=2)
- return st
+from pathlib import Path
def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--workers", type=int, default=40)
- ap.add_argument("--limit_shards", type=int, default=None)
- ap.add_argument("--out", default="/wekafs/ict/hx_624/hf_subdirs_verified/shadertoy")
- a = ap.parse_args()
- flat = {os.path.basename(p)[: -len("_llamafactory_flat")] for p in glob.glob(f"{SRC}/*_llamafactory_flat")}
- allp = {os.path.basename(p)[: -len("_llamafactory")] for p in glob.glob(f"{SRC}/*_llamafactory")}
- todo = sorted(allp - flat)
- if a.limit_shards:
- todo = todo[: a.limit_shards]
- os.makedirs(a.out, exist_ok=True)
- print(f"[verify] {len(todo)} shards never verified -> {a.out}", flush=True)
- from concurrent.futures import ProcessPoolExecutor
- tot = collections.Counter()
- with ProcessPoolExecutor(a.workers) as ex:
- for st in ex.map(do_shard, [(s, a.out) for s in todo]):
- for k in ("rows", "unique_programs", "compiles", "repaired", "unfixable"):
- tot[k] += st[k]
- print(f"[verify] {st['shard']:4s} rows={st['rows']:>6,} uniq={st['unique_programs']:>5,} "
- f"ok={st['compiles']:>6,} repaired={st['repaired']:>4,} unfixable={st['unfixable']:>4,}", flush=True)
- rate = tot["compiles"] / max(tot["rows"], 1) * 100
- print(f"[verify] TOTAL rows={tot['rows']:,} unique={tot['unique_programs']:,} "
- f"compiles={tot['compiles']:,} ({rate:.2f}%) repaired={tot['repaired']:,} unfixable={tot['unfixable']:,}", flush=True)
+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"
+ sys.path.insert(0, str(scripts))
+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")
if __name__ == "__main__":
diff --git a/finetune/tests/test_llamafactory_entrypoints.py b/finetune/tests/test_llamafactory_entrypoints.py
new file mode 100644
index 00000000..e29acfc8
--- /dev/null
+++ b/finetune/tests/test_llamafactory_entrypoints.py
@@ -0,0 +1,183 @@
+"""Offline regression checks for the consolidated dataset scripts and legacy CLIs.
+
+Parquet I/O uses JSON-backed doubles; the actual CLI, audit, sample filtering,
+and forwarding code runs without data/model dependencies or network access.
+"""
+import json
+import os
+import runpy
+import shutil
+import subprocess
+import sys
+import types
+from pathlib import Path
+
+import pytest
+
+FINETUNE = Path(__file__).resolve().parents[1]
+ENTRY_DIRS = ("llamafactory", "scripts")
+LEGACY_SCRIPTS = (
+ "audit_pairs.py",
+ "build_pairs_3dcodebench.py",
+ "convert_subdirs.py",
+ "drop_old_pair_trees.py",
+ "shrink_pair_images.py",
+ "upload_pairs.py",
+ "verify_unflattened_glsl.py",
+)
+
+
+class JsonFrame:
+ def __init__(self, rows):
+ self.rows = rows
+
+ def __len__(self):
+ return len(self.rows)
+
+ def to_dict(self, orient):
+ assert orient == "records"
+ return self.rows
+
+ def to_parquet(self, path, *, index):
+ assert index is False
+ Path(path).write_text(json.dumps(self.rows))
+
+
+def offline_io(monkeypatch):
+ pandas = types.ModuleType("pandas")
+ pandas.DataFrame = JsonFrame
+ pandas.read_parquet = lambda path: JsonFrame(json.loads(Path(path).read_text()))
+ monkeypatch.setitem(sys.modules, "pandas", pandas)
+ parquet = types.ModuleType("pyarrow.parquet")
+ arrow = types.ModuleType("pyarrow")
+ arrow.parquet = parquet
+ monkeypatch.setitem(sys.modules, "pyarrow", arrow)
+ monkeypatch.setitem(sys.modules, "pyarrow.parquet", parquet)
+
+ def no_network(*args, **kwargs):
+ pytest.fail("dataset CLI attempted a network request")
+
+ requests = types.ModuleType("requests")
+ requests.get = no_network
+ hub = types.ModuleType("huggingface_hub")
+ hub.HfApi = hub.hf_hub_download = no_network
+ monkeypatch.setitem(sys.modules, "requests", requests)
+ monkeypatch.setitem(sys.modules, "huggingface_hub", hub)
+ # Legacy entry points add their canonical sibling directory to sys.path.
+ monkeypatch.setattr(sys, "path", list(sys.path))
+ return parquet
+
+
+def run_cli(monkeypatch, entry_dir, filename, *args):
+ script = FINETUNE / entry_dir / filename
+ monkeypatch.setattr(sys, "argv", [str(script), *map(str, args)])
+ try:
+ runpy.run_path(str(script), run_name="__main__")
+ except SystemExit as exc:
+ return exc.code
+ return 0
+
+
[email protected]("entry_dir", ENTRY_DIRS)
[email protected]("kind", ("text", "img", "imgtext"))
[email protected]("literal_in_answer", (False, True))
+def test_audit_counts_image_tokens_in_assistant_turn(
+ tmp_path, monkeypatch, capsys, offline_io, entry_dir, kind, literal_in_answer
+):
+ root = tmp_path / "datasets"
+ dataset = root / f"fixture_llamafactory_{kind}"
+ dataset.mkdir(parents=True)
+ media = tmp_path / "media"
+ media.mkdir()
+ (media / "view.jpg").write_bytes(b"local image fixture")
+ row = {
+ "system": "Write a shader.",
+ "conversations": [
+ {"from": "human", "value": ("<image>" if kind != "text" else "") + "Draw a sphere."},
+ {"from": "gpt", "value": "```glsl\n// " + ("<image>" if literal_in_answer else "sphere") + "\n```"},
+ ],
+ }
+ if kind != "text":
+ row["images"] = ["view.jpg"]
+ (dataset / "train.parquet").write_text(json.dumps([row]))
+ (dataset / "qc.json").write_text(json.dumps({"rows": 1}))
+
+ status = run_cli(monkeypatch, entry_dir, "audit_pairs.py", "--roots", root, "--media_root", media)
+
+ assert status == int(literal_in_answer)
+ output = capsys.readouterr().out
+ expected = "placeholder_in_text_pair" if kind == "text" else "placeholder_mismatch"
+ assert (expected in output) == literal_in_answer
+ if not literal_in_answer:
+ assert "problems: NONE" in output
+
+
[email protected]("entry_dir", ENTRY_DIRS)
+def test_builder_drops_literal_image_token_without_dropping_valid_sample(
+ tmp_path, monkeypatch, offline_io, entry_dir
+):
+ code = "void mainImage(out vec4 color, in vec2 xy) { color = vec4(1.0); }"
+ rows = [
+ {"id": "valid", "code": code, "captions": {"instruction": "A bright sphere against a dark background."}},
+ {"id": "literal", "code": code + "\n// <image>", "captions": {"instruction": "A bright sphere against a dark background."}},
+ ]
+ offline_io.read_table = lambda path: types.SimpleNamespace(to_pylist=lambda: rows)
+ exists = os.path.exists
+ monkeypatch.setattr(
+ os.path,
+ "exists",
+ lambda path: str(path).endswith("/review_fixture/metadata.parquet") or exists(path),
+ )
+ images = tmp_path / "images"
+ source_images = images / "review_fixture"
+ for row in rows:
+ sample_dir = source_images / row["id"]
+ sample_dir.mkdir(parents=True)
+ (sample_dir / "view.png").write_bytes(b"cached local render")
+ (source_images / ".scan_complete").write_text("2\n")
+ output = tmp_path / "output"
+
+ status = run_cli(
+ monkeypatch, entry_dir, "build_pairs_3dcodebench.py",
+ "--subdir", "review_fixture", "--dialect", "glsl",
+ "--out", output, "--img_root", images, "--workers", "1",
+ )
+
+ assert status == 0
+ for kind in ("text", "img", "imgtext"):
+ dataset = output / f"review_fixture_llamafactory_{kind}"
+ built = json.loads((dataset / "train.parquet").read_text())
+ assert [row["id"] for row in built] == ["valid"]
+ assert built[0]["conversations"][1]["value"] == f"```glsl\n{code}\n```"
+ assert sum(t["value"].count("<image>") for t in built[0]["conversations"]) == len(built[0].get("images", []))
+ qc = json.loads((dataset / "qc.json").read_text())
+ assert qc["literal_image_token"] == 1
+ assert qc["source_samples"] == 2
+ assert qc["rows"] == qc["usable"] == 1
+
+
[email protected]("filename", LEGACY_SCRIPTS)
+def test_legacy_cli_forwards_arguments_exit_status_and_sibling_imports(tmp_path, filename):
+ component = tmp_path / "checkout with spaces" / "finetune"
+ legacy = component / "scripts"
+ canonical = component / "llamafactory"
+ legacy.mkdir(parents=True)
+ canonical.mkdir()
+ shutil.copyfile(FINETUNE / "scripts" / filename, legacy / filename)
+ (canonical / "forwarder_sibling.py").write_text("VALUE = 'canonical sibling'\n")
+ (canonical / filename).write_text(
+ "import json, sys\n"
+ "from forwarder_sibling import VALUE\n"
+ "print(json.dumps({'args': sys.argv[1:], 'sibling': VALUE, 'name': __name__}))\n"
+ "raise SystemExit(7)\n"
+ )
+ args = ["--out", "folder with spaces", "--workers", "3"]
+
+ result = subprocess.run(
+ [sys.executable, str(legacy / filename), *args],
+ cwd=tmp_path, capture_output=True, text=True, timeout=10, check=False,
+ )
+
+ assert result.returncode == 7, result.stderr
+ assert json.loads(result.stdout) == {"args": args, "sibling": "canonical sibling", "name": "__main__"}
diff --git a/harness/codeverse3d/addons/dataset/refine.py b/harness/codeverse3d/addons/dataset/refine.py
index c2ef87f9..b085de45 100644
--- a/harness/codeverse3d/addons/dataset/refine.py
+++ b/harness/codeverse3d/addons/dataset/refine.py
@@ -13,7 +13,7 @@ commit. ``instructions`` is ``RoundRecord.instructions``, the task list
``build_refine_instructions`` compiled — not the per-group prompt text the sessions were
handed, which is a rendering of it.
-The rows are data, not a training format: ``toolkits/llamafactory/build_refine_sft.py``
+The rows are data, not a training format: ``finetune/llamafactory/build_refine_sft.py``
turns them into the message shape the finetune pipeline consumes. ``--with-code`` inlines
the changed files so that conversion needs no access to the runs.
"""
diff --git a/harness/docs/ARCHITECTURE.md b/harness/docs/ARCHITECTURE.md
index eca03d65..5b89c639 100644
--- a/harness/docs/ARCHITECTURE.md
+++ b/harness/docs/ARCHITECTURE.md
@@ -670,7 +670,7 @@ instructions written in response, both code snapshots and whether the score move
(the row is `addons/dataset/refine.RefineTransition`; INTERFACES has the fields and the
drop reasons). That is the supervision the harness produces that a
one-shot corpus cannot: what a failing artefact looked like, what was wrong with it
-in the harness's own words, and what the fix changed. `toolkits/llamafactory/`
+in the harness's own words, and what the fix changed. `finetune/llamafactory/`
turns those rows into training files; the harness writes the measurement, not the
trainer's format. `flywheel caption` adds
{detailed, instruction, factory} captions (image-grounded, brand-free, `--out` for
diff --git a/harness/docs/DECISIONS.md b/harness/docs/DECISIONS.md
index de746361..7830f071 100644
--- a/harness/docs/DECISIONS.md
+++ b/harness/docs/DECISIONS.md
@@ -478,7 +478,7 @@ Pointers: EVAL, PAPER_WRITING = `eval/docs/*.md`; COST, RUNBOOK, ARCHITECTURE, I
`cat-file --batch`, never `git archive` (a planted `filter.<name>.smudge` executes and no flag
turns it off); pinned by a test in `tests/flywheel_cli` (`record/_git.py`).
* **The refine transitions are the corpus this harness is for** (`3dcode flywheel refine`: one
- row per refined round); format conversion lives in `toolkits/llamafactory/`, not in the harness
+ row per refined round); format conversion lives in `finetune/llamafactory/`, not in the harness
(owner's boundary, 2026-09-03).
* **`<mimic>` changes the sweep, which is what it was for**: on the 8 of 14 prompts that declare
a coupling the sampler drives a median 3 joints per pose instead of 6; the `joint_sweep`
diff --git a/harness/docs/RUNBOOK.md b/harness/docs/RUNBOOK.md
index 1dba7000..365947f3 100644
--- a/harness/docs/RUNBOOK.md
+++ b/harness/docs/RUNBOOK.md
@@ -196,7 +196,7 @@ unless you type `--host` yourself; it never serves a path outside the declared r
# one row per round the harness asked to change; the row
# schema is codeverse3d/addons/dataset/refine.RefineTransition and
# INTERFACES has the call signatures. Training formats live
- # in toolkits/llamafactory/, not here.
+ # in finetune/llamafactory/, not here.
3dcode flywheel caption <slug> [--model …] [--out caps/] # --out = side-car mode, run untouched
3dcode flywheel index runs/ runs_index.sqlite
python -m codeverse3d.addons.calibration runs/<slug> [runs/<slug2> …] --model gemini:gemini-3.1-pro-preview --n 3 --out out/
diff --git a/toolkits/README.md b/toolkits/README.md
index 410b8d3b..8ed536b9 100644
--- a/toolkits/README.md
+++ b/toolkits/README.md
@@ -1,32 +1,29 @@
-# toolkits/ — the tools that turn raw 3D projects into trainable data
+# toolkits/ — general data CLI and corpus curation
-Everything here operates on **data**, not on models. The pipeline they cover, end to end:
-
-```
-contributor's project ──▶ curation ──▶ 3DCodeVerse on the Hub ──▶ LLaMA-Factory datasets ──▶ finetune/
- code + renders validate, per-source subdirs per-subdir ShareGPT, training runs
- + meta.json dedupe, (metadata.parquet + tars) verified & flagged and evaluation
- render, pack
-```
+These tools turn contributed or source projects into the 3DCodeVerse corpus: validate, execute,
+render, deduplicate, attach metadata, and package the results for publication.
| folder | stage | what it does |
|---|---|---|
| [`3dcode_cli/`](3dcode_cli/) | contributor → corpus | the installable `3dcode-data` CLI: validate, fingerprint/dedupe, execute, render and push a project to the staging bucket. Per-dialect modules for Blender-Python, CadQuery, build123d, FreeCAD and OpenSCAD |
| [`curation/`](curation/) | raw → corpus | per-source curation: convert a source's native format to runnable projects, execute/render them, compute ground-truth meshes, dedupe, pack to tar+parquet, upload to the Hub |
-| [`llamafactory/`](llamafactory/) | corpus → training data | turn every corpus subdirectory into a LLaMA-Factory dataset, verify it with the real compilers/runtimes, flatten multi-file samples, build multimodal (render + instruction) variants, and sample training mixes |
-`3dcode_cli/` is a complete copy of **[gaoypeng/3dcode_toolkit](https://github.com/gaoypeng/3dcode_toolkit)**
-(74 files, byte-for-byte apart from `.git`), kept here so the whole data toolchain lives in one place. It is still
-`pipx install`-able from either location:
+LLaMA-Factory conversion, training-data QC, multimodal pairs, and training mixes live in
+[`../finetune/llamafactory/`](../finetune/llamafactory/). Its README contains the training-data script
+catalog and usage examples. Training configs, environments, and launchers remain under `finetune/`.
+
+## 3dcode_cli/
+
+`3dcode_cli/` is a repository copy of
+**[gaoypeng/3dcode_toolkit](https://github.com/gaoypeng/3dcode_toolkit)**. It can be installed from
+either location:
```bash
-pipx install "git+https://github.com/gaoypeng/3dcode_toolkit" # upstream
-pipx install ./toolkits/3dcode_cli # this copy
+pipx install "git+https://github.com/gaoypeng/3dcode_toolkit" # upstream
+pipx install ./toolkits/3dcode_cli # from this repository's root
```
-If both are edited they will drift — pick one as canonical and mirror the other.
-
----
+If both copies are edited they can drift; coordinate changes with the upstream repository.
## curation/ — per source
@@ -35,68 +32,7 @@ If both are edited they will drift — pick one as canonical and mirror the othe
| `deepcad/` | `convert_vec_to_cq.py`, `batch_convert.py`, `cq_check{,_batch}.py`, `gen_gt_stl.py`, `iou_compute.py`, `render_all.py`, `dedup_check.py`, `attach_text.py`, `pack_tar_parquet.py`, `finalize.py`, `upload_to_hf.py` | DeepCAD vector sequences → CadQuery programs, executed and checked, GT meshes + IoU, captions attached, packed and uploaded |
| `articraft/` | `articraft_cq.py`, `assemble_shim.py`, … | articulated assets → CadQuery / URDF projects |
-The shape of a curation pass, which any new source should follow: **convert → execute → render → compute a
-ground truth → dedupe → attach text → pack → upload**. Every step writes files, so a failed step can be rerun
-without redoing the pipeline.
-
-## llamafactory/ — corpus → trainable datasets
-
-| script | what it does |
-|---|---|
-| `convert_subdirs.py` | one `<subdir>_llamafactory/` per corpus subdirectory: `train.parquet` (ShareGPT) + `dataset_info.json` + `qc.json` + README. Runs the per-sample quality checks (empty/short/mojibake code, in-subdir duplicates, missing captions, **benchmark overlap**, syntax, over-length, literal media placeholders) |
-| `flatten_multifile.py` | makes multi-file samples single-file: merges a Shadertoy shader's **Common tab** into each pass exactly as Shadertoy does, and recovers `code.urdf` for the articraft URDF subdirs (whose `code` column is empty in the metadata) |
-| `qc_repair.py` | validates with the **real** compiler (glslangValidator for GLSL, `ast.parse` for Python dialects), attempts minimal semantics-preserving repairs, re-validates each one, and reports exact-duplicate groups within and across subdirs |
-| `glsl_render_batch.py`, `glsl_render.py`, `render_verify_glsl.py` | compiling is not drawing: render every shader in a real headless WebGL2 context, sample two frames, and record whether the image is non-uniform and whether it animates |
-| `finalize_glsl.py` | consolidates: prefer the author's untouched program, fall back to a repair only if it compiles, otherwise mark `compiles=False`. Writes `compiles` / `repair` columns |
-| `build_mm_dataset.py` | the multimodal datasets — `mm_img` (renders only), `mm_img_text` (renders + instruction), `mm_text` (control), from the same samples with the same split so they can be compared |
-| `build_image_dataset.py` | earlier single-variant image builder (kept for reproducing the first pilot) |
-| `build_md_max.py`, `sample_mix.py` | build the full corpus (every source, every caption variant) and sample training mixes from it with a per-dialect token budget and a captions-per-sample cap |
-| `to_llamafactory.py` | jsonl → ShareGPT JSON + a `dataset_info.json` entry |
-| `build_hf_dataset.py` | package corpus + preference sets + held-out tests as a Hub dataset |
-| `cache_ntok.py`, `ntok_cache.py` | cache per-sample token counts so re-mixing is seconds, not half an hour |
-| `extract.py` | dialect-aware code extraction from free-form model output (used by evaluation, included here because dataset QC uses the same rules) |
-| `analyze_3dcodeverse.py`, `fetch_hf_files.py` | corpus inventory; selective file download from the Hub |
-
-### Two rules these tools encode
-
-1. **Suspect the checker before the data.** Of 4,192 shaders that "failed", only 588 actually needed a code
- change; the rest exposed defects in the validation harness (guessing sampler types from syntax, a missing
- `iFrameRate` uniform, a regex that deleted real code from minified lines, the wrong GLSL ES profile).
- Every checker in here has a known-good control that must pass.
-2. **Mark, don't silently drop.** Rows that cannot be used for a particular purpose — `benchmark_factory`
- (3DCodeBench contamination), `compiles=False`, over-length — stay in the dataset with a column that lets a
- training mix filter them out. The one exception is exact duplicates, which are removed and listed in
- `duplicates_<source>.json` so the upstream corpus can be cleaned too.
-
-### Typical use
-
-```bash
-# 1. corpus -> one dataset per subdirectory, with QC
-python convert_subdirs.py --out /path/to/out --workers 16
-
-# 2. make multi-file samples single-file, recover code that lives in tars
-python flatten_multifile.py --shadertoy --out /path/to/out
-python flatten_multifile.py --urdf_tars /path/to/tars --out /path/to/out
-
-# 3. verify with the real toolchain, repair minimally, consolidate
-python qc_repair.py --pass all --root /path/to/out
-python render_verify_glsl.py --workers 10 --write
-python finalize_glsl.py
-
-# 4. build a training mix (per-dialect token budget, one caption per sample)
-python sample_mix.py --out mix/ --budget "blender=all,cadquery=25M,glsl=25M" --captions_per_sample 1
-
-# 5. multimodal variants (renders + instruction)
-python build_mm_dataset.py --views 4 --out mm/
-```
-
-Results produced with these tools: [`finetune/llm_finetune_exps.md`](../finetune/llm_finetune_exps.md) and
-[`finetune/docs/REPORT.md`](../finetune/docs/REPORT.md).
-
----
-
-## Layout note
-
-Nothing outside this folder hardcodes the name `toolkits/`: the scripts take their input and output paths as
-arguments, and the only references are the links in this file, `../README.md` and `../finetune/README.md`.
-Renaming the folder is therefore a `git mv` plus those three links.
+The shape of a curation pass is **convert → execute → render → compute ground truth → dedupe →
+attach text → pack → upload**. Scripts retain intermediate files; inspect their completion checks
+before rerunning a partially completed pass. Some scripts contain paths from the original workstation;
+adapt their input, output, and tool paths before running them elsewhere.
diff --git a/toolkits/llamafactory/README.md b/toolkits/llamafactory/README.md
deleted file mode 100644
index 9710236f..00000000
--- a/toolkits/llamafactory/README.md
+++ /dev/null
@@ -1,71 +0,0 @@
-# llamafactory/ — 3DCodeVerse → LLaMA-Factory datasets
-
-Scripts that turn the corpus on the Hub into datasets a training run can consume directly, and that verify the
-data with the same runtimes the evaluation uses. See [`../README.md`](../README.md) for the table of every script
-and the two rules they encode; the sections below are the details you need to run them.
-
-## Output layout
-
-```
-<subdir>_llamafactory/ one per corpus subdirectory, next to the source
- train.parquet ShareGPT: system + conversations[human,gpt], plus id / caption_type / n_tokens
- dataset_info.json drop-in registry entry -> dataset: 3dcv_<subdir>
- qc.json per-subdir counters (drops, syntax, token stats)
- README.md what it is and what was dropped, with reasons
-<subdir>_llamafactory_flat/ flattened multi-file variants (Shadertoy passes with Common merged)
- train.parquet + compiles (bool) and repair (original / repaired:<cat> / unfixable)
-```
-
-Multimodal datasets (`build_mm_dataset.py`) add an `images` column; the number of `<image>` placeholders in the
-user turn **must** equal `len(images)` — LLaMA-Factory's `mm_plugin._validate_input` raises otherwise. Multimodal
-rows also cannot be packed: set `packing: false` and `neat_packing: false`.
-
-## Dialects covered
-
-| dialect | runtime used for verification | notes |
-|---|---|---|
-| Blender-Python | Blender 5.0.1 headless (needs `scipy` in its bundled Python) | 15 of the 212 3DCodeBench reference scripts import scipy |
-| CadQuery | CadQuery 2.8 | the runner captures `show_object`/exports and prefers `result`/`solid` variables |
-| OpenSCAD | OpenSCAD AppImage (`--export-format binstl`) | decode with sampling: greedy collapses into the Customizer parameter block |
-| GLSL | glslangValidator **and** headless WebGL2 | try ES 3.00 and 3.10 × the channel-type assignments a shader actually uses |
-| three.js / web | headless Chromium (Playwright) | judge blankness from the composited screenshot, not `gl.readPixels` |
-| URDF | XML parse + link/joint counts | code recovered from the source tars |
-
-## Requirements
-
-Python 3.12 with `pandas`, `pyarrow`, `transformers` (tokenizer only), `playwright` (+ `playwright install chromium`),
-and `huggingface_hub` for the upload paths; `glslangValidator` on PATH (conda-forge `glslang`); Blender / OpenSCAD
-as system binaries. Paths to the binaries are constants at the top of each script — change them there.
-| `build_pairs_3dcodebench.py` | the three pair types for one source subdirectory — text→code, image→code, image+text→code — with one caption per sample rather than one row per caption variant. Renders are read by **scanning each tar**, because the byte offsets in `metadata.parquet` cannot be trusted (for `3dcodebench/instances_geo` only 577 of 1,953 point at a real tar header; the archives were repacked after the metadata was written, and following them silently cost 70% of the image pairs) |
-| `verify_unflattened_glsl.py` | compile-verifies the 44 Shadertoy shards `flatten_multifile.py` never reached — their metadata carries no `shader_json`, so there was no renderpass structure to merge a Common tab from. Writes the same `compiles` / `repair` columns the flattened shards have, so one filter covers the whole GLSL corpus |
-| `runners/` | the per-source drivers that call the above over every subdirectory |
-
-### A third rule these tools now encode
-
-3. **A conversion must never decide what a benchmark forbids.** `convert_subdirs.py` used to drop any sample
- overlapping the 212 3DCodeBench factories, which silently emptied both `3dcodebench/instances_*` folders —
- 100% of their rows were overlaps. The overlap is now *counted* in `qc.json` and the rows are kept, so the
- split is a decision the training run makes, not one the converter already made.
-| `shrink_pair_images.py` | re-encodes the extracted renders to training resolution (≤448px JPEG). Measured 90.8% smaller — 113.7 GB becomes 10.5 GB — and loses nothing a training run would have seen, because every vision tower downsamples to 448² anyway; the originals stay in the corpus tars |
-| `upload_pairs.py` | publishes the pair datasets INTO their source folders (`<source>/<subdir>_llamafactory_{text,img,imgtext}`) and their renders as `<source>/pair_renders/`, then registers everything in the root `dataset_info.json` |
-| `drop_old_pair_trees.py` | removes a superseded tree only after verifying the index no longer points into it — it refuses to run otherwise |
-
-### A fourth rule, learned the expensive way
-
-4. **Same name does not mean same content — compare before deleting.** Twice a folder appeared both at the
- repository root and inside its source, and both times the root copy was the NEWER one (1,008 vs 949 samples,
- and earlier 949 vs 318). Deleting the apparent duplicate would have silently discarded 1,239 samples. The
- safe order is always: compare, promote the newer copy, then delete — in a single commit, so the repository is
- never seen with the data in two places or in neither.
-| `audit_pairs.py` | checks every pair dataset for the invariants that fail only once training has started: absolute paths, wrong extension, image files that do not exist, `<image>` count vs `len(images)`, turn shape, fenced code, empty system, row count vs `qc.json`. Exits non-zero on any finding, so it can gate a publish |
-
-### A fifth rule
-
-5. **Sampling proves a problem exists; it cannot prove one does not.** Checking four datasets said "every path is
- relative". Scanning all 411 found **36,964 absolute paths** in three of them — introduced by a later rebuild
- that reused already-extracted renders and took a code path which skipped the relativisation. For a global
- invariant, scan globally: `audit_pairs.py` exists for exactly this.
-
- The same audit script then reported all 1,948,765 rows as malformed, which was a defect in the checker
- (parquet returns numpy arrays, and `isinstance(x, list)` is False for them) — rule 1 applies to your own
- checkers too.