Official Codebase of 3DCodeVerse
Python
refactor: consolidate LLaMAFactory tooling under finetune
Yipeng Gao0bea329baca2
diff --git a/CLAUDE.md b/CLAUDE.mdindex d25786a5..fe964661 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -10,4 +10,7 @@ The harness lives in **`harness/`** — read `harness/CLAUDE.md` +A/B rigs, reports; its tests: `cd eval && python -m pytest`), `eval/llm` evaluates a bare LLM/VLM(was `finetune/3dcodeverse_eval`). Read `eval/README.md` first.-Other top-level folders are separate components (`toolkits/` data tooling, `finetune/` training).+Other top-level folders are separate components: `toolkits/` owns the general data CLI and source curation;+`finetune/` owns training and its data preparation. The canonical LLaMA-Factory data scripts live in+`finetune/llamafactory/`; `finetune/configs/`, `finetune/env/`, and the training launchers in+`finetune/scripts/` retain their existing locations. Read `finetune/CLAUDE.md` before changing them.diff --git a/README.md b/README.mdindex 46ff10ce..803dbc27 100644--- a/README.md+++ b/README.md@@ -73,8 +73,8 @@ on 805 fixed tasks:| Directory | Description | Documentation || :-- | :-- | :-- || [`harness/`](harness/) | Agent-based 3D content generation, execution, rendering, and verification. | [Harness guide](harness/README.md) |-| [`toolkits/`](toolkits/) | Dataset curation, format conversion, validation, and training-data preparation. | [Data toolkit guide](toolkits/README.md) |-| [`finetune/`](finetune/) | Supervised fine-tuning and preference optimization for open models. | [Training guide](finetune/README.md) |+| [`toolkits/`](toolkits/) | General data CLI and per-source corpus curation. | [Data toolkit guide](toolkits/README.md) |+| [`finetune/`](finetune/) | LLaMA-Factory training-data preparation, supervised fine-tuning, and preference optimization for open models. | [Training guide](finetune/README.md) || [`eval/`](eval/) | Benchmarks and evaluation pipelines for the harness and standalone models. | [Evaluation guide](eval/README.md) |## Citationdiff --git a/eval/docs/PAPER_WRITING.md b/eval/docs/PAPER_WRITING.mdindex 59dbb1fb..fff40486 100644--- a/eval/docs/PAPER_WRITING.md+++ b/eval/docs/PAPER_WRITING.md@@ -19,7 +19,7 @@ that owns it — §5.2 was exactly that until 2026-09-03 (review) and is now a pA/B with an A/A mode), `bench/compare_backends.py` (harness vs one-shot),`bench/paired_compare.py` (paired statistics), `bench/plan_stage_bench.py` +`bench/plan_stage_report.py` (the loss-event channel, data in `bench/data/plan_stage/`),-`3dcode flywheel refine` (the transition corpus) and `toolkits/llamafactory/` (its training+`3dcode flywheel refine` (the transition corpus) and `finetune/llamafactory/` (its trainingformat).## 0. The system, in one paragraph@@ -53,7 +53,7 @@ full battery were `joint_sweep`.count the loss event, run n in the hundreds (§5.1).6. **Coupled-mechanism support for URDF articulation** with a mechanism-level readout (§5.3).7. **A refine corpus**: (judge issues + gate findings + instructions) → code diff → score- delta; `toolkits/llamafactory/` converts those rows into the finetune pipeline's format,+ delta; `finetune/llamafactory/` converts those rows into the finetune pipeline's format,which the harness itself never writes (§7).## 0.2 Evaluation-protocol work that predates the measurements@@ -424,7 +424,7 @@ round +0.197, 119 rows carrying gate ERRORs. Both sides come from the round's ocommit, not from `HEAD`. (The same export read 205 rows on 2026-09-02; the corpus growswith every battery, so the selector is the date.)-`toolkits/llamafactory/build_refine_sft.py` turns the improved transitions into the message+`finetune/llamafactory/build_refine_sft.py` turns the improved transitions into the messageshape the finetune pipeline consumes: **112 samples** (92 articulated, 20 static; mediananswer 25.4 kB, median brief 28.2 kB), answered with the files the round produced in the`=== FILE: path ===` envelope. This is the one thing the harness produces that a one-shot corpus cannot: *given@@ -511,7 +511,7 @@ data the repo does not carry; they are marked in the caveat column.| base64 amplification | 261 k tokens vs 516 for one 275 kB sheet | measured on a recorded blob + vendor bundle | arithmetic + matching recorded prompt jumps; **no repo script** || token growth in turns | input(t) = 12 364 + 1 822·t; total ∝ n^1.60; dollars ∝ n^1.19–1.34 | 1 063 in-process sessions, 30 401 turns | **no repo script** — the in-process agent, since gemini-cli emits no per-turn usage || coupled mechanisms declared | 8 of 14 prompts; driven joints per pose 6 → 3 (1–2 on the one-input mechanisms) | `bench/coupling_stats.py bench/out/wave2_lean --per-prompt` | recomputed from the recorded URDFs |-| refine corpus | 254 transitions, 112 SFT samples (2026-09-03) | `3dcode flywheel refine`, `toolkits/llamafactory/build_refine_sft.py` over `bench/out` | corpus grows with every battery |+| refine corpus | 254 transitions, 112 SFT samples (2026-09-03) | `3dcode flywheel refine`, `finetune/llamafactory/build_refine_sft.py` over `bench/out` | corpus grows with every battery |## 9.2 Open questions@@ -606,7 +606,7 @@ python bench/plan_stage_bench.py --tree ../harness --label restart_on --reps 20python bench/plan_stage_report.py bench/data/plan_stage/*.jsonl# the corpus the loop produces3dcode flywheel refine bench/out refine.jsonl --with-code-python ../toolkits/llamafactory/build_refine_sft.py refine.jsonl --out refine_sft.jsonl # toolkits/ is at the repo root+python ../finetune/llamafactory/build_refine_sft.py refine.jsonl --out refine_sft.jsonl # finetune/ is at the repo root```Recorded runs referenced above (not committed): `bench/out/{compare_v4_calm, compare_art_v2,diff --git a/finetune/CLAUDE.md b/finetune/CLAUDE.mdindex 49c099e3..40d8954f 100644--- a/finetune/CLAUDE.md+++ b/finetune/CLAUDE.md@@ -1,6 +1,19 @@# finetune/ (component notes for agents)-LLM finetuning + execution-based eval for 3DCodeVerse. Read `README.md` here and `docs/REPORT.md` (conclusions + pitfalls) first.-- Training is **unpatched LLaMA-Factory** driven by `configs/lf/*.yaml`; do not fork the framework — add a YAML + a dataset entry.-- Evaluate by *executing* the code (`eval/`): 3DCodeBench (Blender → Chamfer/F vs GT), per-dialect held-out sets, pass@N. Greedy decoding collapses on boilerplate-heavy dialects (OpenSCAD/GLSL) — use sampling there.-- Absolute paths point at the original machine; adapt `env.sh` + YAML path keys + tool paths in `eval/*runner*.py` before running.+Training-data preparation and LLM fine-tuning for 3DCodeVerse. Read `README.md` here and+`docs/REPORT.md` (conclusions and pitfalls) first.++- Training is **unpatched LLaMA-Factory** driven by `configs/lf/*.yaml`; keep the upstream framework+ external and add a recipe plus the corresponding dataset registry entries.+- `llamafactory/` is the canonical home for this project's LF data conversion, QC, pair builders,+ mix sampling, and publication scripts. It is not vendored framework code. Edit implementations there;+ the seven Python entries in `scripts/` are compatibility forwarders.+- Configs stay in `configs/`, environment snapshots in `env/`, and train/export/orchestration launchers+ in `scripts/`. General source curation and the contributor CLI belong in `../toolkits/`.+- Evaluate generated code with `../eval/llm/`: 3DCodeBench, held-out dialect suites, and pass@N. Follow+ its protocol and decoding defaults. The old stop-DPO scripts still need adaptation to its generation+ schema; do not assume replacing a path is sufficient.+- Workstation paths remain in `env.sh`, launchers, configs, and data tools. Inspect them before running;+ editing `env.sh` alone is insufficient. Current local training recipes also require external data+ registries not fully represented by `data_specs/dataset_info.json`.+- Keep historical reports and experiment records intact when reorganizing code.diff --git a/finetune/README.md b/finetune/README.mdindex d062b5cf..e985561a 100644--- a/finetune/README.md+++ b/finetune/README.md@@ -1,48 +1,84 @@-# finetune/ — LLM finetuning recipes + execution-based evaluation for 3DCodeVerse+# finetune/ — training data and LLaMA-Factory recipes for 3DCodeVerse-Everything needed to reproduce the Qwen3.5-9B finetuning study on 3DCodeVerse-(text → Blender-Python / CadQuery / OpenSCAD / GLSL / three.js code), and the-conclusions in `docs/REPORT.md`. The harness (`../harness`) generates data with-API models; this folder trains and evaluates open models on that data.+This component prepares training data and fine-tunes open models to generate Blender-Python,+CadQuery, OpenSCAD, GLSL, and three.js code. The harness in [`../harness/`](../harness/) generates+data with API models; standalone model evaluation lives in [`../eval/llm/`](../eval/llm/).-**Nothing in LLaMA-Factory is patched** — all training is plain-`llamafactory-cli` YAML (`configs/lf/`) + ShareGPT JSON registered in-`data_specs/dataset_info.json`. Pinned environments are in `env/`.+Training uses **unmodified upstream LLaMA-Factory**, pinned in `env/requirements-lf.txt`, through+`llamafactory-cli` and ordinary YAML recipes. `llamafactory/` contains this project's data-preparation+tools; it is not a copy or fork of the LLaMA-Factory framework.## Layout| path | what ||---|---|-| `scripts/` | data builders (`prep_data.py`, `build_multidialect*.py`, `build_md_big.py`, `build_md_xl.py`, `build_md_bal.py`, `build_boot_data.py`, `cache_ntok.py`), thin launchers around LLaMA-Factory (`lf_train.sh`, `lf_export.sh`, `make_lora_cfg.py`), experiment pipelines train→merge→eval (`run_exp.sh`, `run_md_exp.sh`, `run_full_ft*.sh`, `*_ckpt_sweep.sh`), execution-feedback / geometry-feedback DPO rounds (`run_dpo_exec.sh`, `dpo_round2.sh`, `md_dpo_round.sh`, `md_big_dpo_round.sh`, `md_xl_dpo_round.sh`, `geo_dpo_build.sh`), bootstrapping (`self_train_round*.sh`), report tooling (`build_report_html.py`, `contact_sheet_3dcodebench.py`) |-| `configs/lf/` | one LLaMA-Factory YAML per experiment: LoRA SFT (`lora_*.yaml`), full-parameter ZeRO-3 SFT (`qwen35_9b_full_sft*.yaml`, `full_md_bal.yaml`, `ds_z3_offload.json`), DPO (`dpo_*.yaml`), LoRA merge/export (`export_*.yaml`) |-| [`../eval/llm/`](../eval/llm/) | the execution-based evaluation (was `eval/` here): vLLM / API generation, 3DCodeBench execution in Blender, geometry metrics Chamfer / F-score vs GT, pass@k, per-dialect executors (CadQuery, OpenSCAD, GLSL, three.js), held-out dialect sets — see its README |-| `data_specs/dataset_info.json` | LLaMA-Factory dataset registry for every SFT/DPO set used (ShareGPT format; the JSON files themselves are rebuilt by `scripts/`) |-| `env/requirements-{llmft,lf,vllm}.txt` | pip freezes of the three conda envs (data/eval, LLaMA-Factory training, vLLM inference) |-| `env.sh` | shell env (caches, secrets file) used by all scripts |-| `docs/REPORT.md` | full experiment report (setup, data audit, ablations, DPO, multi-dialect, scale-up, training-amount curves, qualitative checks); `docs/PROJECT_README.md` is the original working README; `docs/assets/` renders |+| [`llamafactory/`](llamafactory/) | canonical training-data scripts: ShareGPT conversion, quality checks, multimodal pairs, token counts, mix sampling, and Hub packaging; includes the per-source `runners/` |+| [`scripts/`](scripts/) | training/export launchers and workstation orchestration; seven old Python data-script paths are compatibility forwarders to `llamafactory/` |+| [`configs/lf/`](configs/lf/) | current SFT and stop-DPO YAML recipes, the LoRA export template, and a DeepSpeed offload template; see its README for the current recipe list |+| `data_specs/` | recorded dataset registry and data statistics; each run must also have the registry and datasets named by its own `dataset_dir` |+| `env/requirements-{llmft,lf,vllm}.txt` | snapshots of the original data/eval, LLaMA-Factory training, and vLLM inference environments |+| `env.sh` | original workstation shell setup for environments, caches, and a local secrets file |+| [`docs/REPORT.md`](docs/REPORT.md) | experiment conclusions; `REPORT_FULL.md`, `PROJECT_README.md`, and `llm_finetune_exps.md` preserve the historical study and run log |-## Data on the Hub+Configs, environments, and training launchers keep their existing locations. General corpus curation+and the contributor CLI remain in [`../toolkits/`](../toolkits/).-The training data in LLaMA-Factory (ShareGPT) format is published as **[`ilabai/3dcodeverse-llamafactory`](https://huggingface.co/datasets/ilabai/3dcodeverse-llamafactory)** (private; same CC-BY-NC-SA / non-commercial terms as the parent `ilabai/3dcodeverse`): the corpus sharded by dialect (`sft/<dialect>/`), the 15.7k execution-verified bootstrapped Blender samples, seven preference-pair sets (execution feedback and geometry feedback), the held-out test sets, a drop-in `dataset_info.json`, and `mixes.json` with the exact budget spec of every training mix in the report. Rebuild/refresh it with `scripts/build_hf_dataset.py`; regenerate any mix with `scripts/sample_mix.py`.+## Data preparation-## Results-* **[`llm_finetune_exps.md`](llm_finetune_exps.md)** — the full experiment log: every training run with its config and wall time, every 3DCodeBench and per-dialect evaluation, and a summary of which levers moved the numbers (auto-generated tables from `scripts/build_exp_log.py` + curated commentary).+Use the tools in [`llamafactory/`](llamafactory/README.md) for corpus conversion, validation,+multimodal datasets, and mix sampling. For example, from this directory, after preparing the source+`train.jsonl` and `val.jsonl`:-## Install & background-* **`docs/SETUP.md`** — step-by-step environment install (three conda envs with exact pins, LLaMA-Factory commit, flash-attn / fla / triton, vLLM, Blender / OpenSCAD / glslang / Playwright, caches on a network FS, smoke tests).-* **`docs/LLAMA_FACTORY.md`** — how this project uses LLaMA-Factory (unpatched): dataset registry, template, stages (SFT / DPO), LoRA vs full + ZeRO-3, packing, launch/merge scripts, an annotated minimal YAML, and the pitfalls.-* **`docs/REPORT.md`** — the experiment report (results + conclusions).+```bash+python llamafactory/sample_mix.py --src /path/to/md_max --out /path/to/mix \+ --budget "blender=all,cadquery=25M,glsl=25M" --captions_per_sample 1+```-## Reproducing (short version)+The study's ShareGPT data is published as+**[`ilabai/3dcodeverse-llamafactory`](https://huggingface.co/datasets/ilabai/3dcodeverse-llamafactory)**+(private; see the dataset card for its non-commercial terms). `llamafactory/build_hf_dataset.py`+packages the corpus, historical preference sets, held-out tests, and mix descriptions from prepared+local inputs. The published pair datasets are also used by the Hub recipe in `configs/lf/`.-1. Envs (see `docs/REPORT.md` §1 for the pitfalls): `lf` = LLaMA-Factory (editable, upstream, unpatched) + torch 2.8.0+cu128 + flash-attn 2.8.3 + flash-linear-attention 0.5.2 + **triton 3.7.1** (Qwen3.5 GDN kernels on Hopper) + deepspeed; `vllm` = vLLM 0.27; `llmft` = data/eval (trimesh, cadquery, playwright, …). Blender 5.0.1 headless and an OpenSCAD AppImage are called by path.-2. Data: only `metadata.parquet` of each 3DCodeVerse subset is needed. `scripts/build_multidialect.py` → `data/multidialect/<dialect>/{train,test}.jsonl`; `scripts/build_md_*.py` build the mixes (token-length filter ≤8192, cached by `scripts/cache_ntok.py`); `scripts/to_llamafactory.py` writes ShareGPT JSON + `dataset_info.json`.-3. Train: `GPUS=0,1,2,3 scripts/lf_train.sh configs/lf/lora_md_xl.yaml gradient_accumulation_steps=2` (LoRA r64, cutoff 8192, packing, FA2) → `scripts/lf_export.sh configs/lf/export_md_xl.yaml` to merge.-4. Evaluate with `../eval/llm/` (its README): from `../eval`, `python -m llm.run_eval --backend vllm --model <merged> --run <name> --suites headline heldout` (3DCodeBench: vLLM gen → Blender exec → Chamfer/F vs GT, plus the held-out dialect sets). Decode OpenSCAD/GLSL with sampling (T=0.7), not greedy.-5. Execution-feedback DPO on top of any SFT model: `scripts/md_xl_dpo_round.sh` (sample → run in each dialect's executor → (OK, FAIL) pairs → `stage: dpo`).+`data_specs/dataset_info.json` is a historical registry snapshot. The current local SFT recipes point+at separate `mm_mix`, `full9b`, `pairs_mix`, or `pub_img` directories on the original workstation;+their complete registries and data are not included in this checkout. Prepare or obtain the matching+data and registry before launching a recipe. The remote recipe reads its registry from the Hub and+still needs access to the dataset and its local media files.-Paths are absolute for the machine the study ran on (`/wekafs/ict/hx_624/...`); adapt `env.sh`, the `*_dir`/`model_name_or_path` keys in `configs/lf/*.yaml`, and the tool paths at the top of `scripts/*.sh` before running elsewhere.+## Train, export, evaluate-## Headline results (3DCodeBench, 212 tasks, execution rate / [email protected]; details in `docs/REPORT.md`)+Read [`docs/SETUP.md`](docs/SETUP.md) for the original environment setup and+[`docs/LLAMA_FACTORY.md`](docs/LLAMA_FACTORY.md) for the integration. The shell launchers and data+tools still contain `/wekafs/ict/hx_624/...` workstation paths. Adapt `env.sh`, the launchers, the+YAML model/data/output paths, and tool paths before running elsewhere; changing `env.sh` alone does+not update the other files.-Qwen3.5-9B zero-shot 0% → LoRA on 5k Blender samples 75–82% → + execution-feedback DPO 93.9–96.2% → 4-dialect mixes: 25.6k 67%, 102k 84.9%, 270k (all 3DCodeVerse) 90.1%; 4 H100s: LoRA ≈17k tok/s, 270k samples / 293M tokens in 4.8 h.+After that setup and data preparation, a current SFT recipe can be launched from `finetune/`:++```bash+GPUS=0,1,2,3 bash scripts/lf_train.sh configs/lf/sft_9b_lora64_lr1e-4_len8k_mmmix-plus.yaml+bash scripts/export_lora.sh /absolute/path/to/run /absolute/path/to/the-same-base-model+```++The export script writes the merged model into `<run>/merged`. In the evaluation environment, from+the repository's `eval/` directory, use the current standalone evaluator:++```bash+python -m llm.run_eval --backend vllm --model /absolute/path/to/run/merged \+ --run my_sft --suites headline heldout+```++See [`../eval/llm/README.md`](../eval/llm/README.md) for tool/asset setup and suite decoding defaults.++The stop-DPO recipes are in `configs/lf/dpo_*.yaml`; their workstation orchestration remains in+`scripts/stop_dpo_pairs.sh` and `scripts/stop_dpo_round.sh`. Those scripts still depend on the old+generation entrypoint, output schema, and prepared training prompts. Adapting them to `eval/llm/`+is separate work; the folder reorganization does not make that historical pipeline portable.++## Experiment records++[`llm_finetune_exps.md`](llm_finetune_exps.md) and [`docs/REPORT.md`](docs/REPORT.md) document the+results and lessons from the study. Historical run names and commands in these records may refer to+recipes or drivers no longer present in the current checkout. Use [`configs/lf/README.md`](configs/lf/README.md)+for the current recipe inventory.diff --git a/finetune/docs/LLAMA_FACTORY.md b/finetune/docs/LLAMA_FACTORY.mdindex 72e57ed2..7b7c40de 100644--- a/finetune/docs/LLAMA_FACTORY.md+++ b/finetune/docs/LLAMA_FACTORY.md@@ -1,68 +1,92 @@-# How this project uses LLaMA-Factory (primer)+# How this project uses LLaMA-Factory-[LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) is a config-driven finetuning framework on top of 🤗 transformers/peft/trl/deepspeed: you describe *model + data + stage + method + trainer args* in one YAML and run `llamafactory-cli train that.yaml`. We use it **unmodified** (editable install of upstream commit `c4e09c7`, 0.9.6.dev0; `git status` clean). Everything project-specific is a YAML in `configs/lf/` or a dataset entry in `data_specs/dataset_info.json`.+[LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) consumes a YAML describing the model,+data, training stage, and trainer arguments via `llamafactory-cli train <config.yaml>`. This project+uses upstream unmodified: the training environment snapshot pins commit+`c4e09c7cbe18844816af9e18a97fe465515edbcd` (0.9.6.dev0) in+[`../env/requirements-lf.txt`](../env/requirements-lf.txt).-## The pieces we touch-| LLaMA-Factory concept | where in this repo | notes |+Our [`../llamafactory/`](../llamafactory/) directory contains **training-data tools**, not framework+source: conversion, validation, multimodal pair construction, token counting, mix sampling, and Hub+packaging. Recipes remain in `configs/lf/`, environment snapshots in `env/`, and train/export+launchers in `scripts/` (paths below are relative to `finetune/`).++## The integration points++| concept | where | notes ||---|---|---|-| **dataset registry** `dataset_info.json` | `data_specs/dataset_info.json` (copy to your `dataset_dir`) | every SFT set is ShareGPT JSON: `{"conversations":[{"from":"human","value":prompt},{"from":"gpt","value":code}],"system":...}`; DPO sets add `"chosen"/"rejected"` and `"ranking": true`. `scripts/to_llamafactory.py` writes both the JSON and the registry entry. |-| **template** | `template: qwen3_5`, `enable_thinking: false` | Qwen3.5 is a VLM wrapper (`Qwen3_5ForConditionalGeneration`); we freeze the vision tower (`freeze_vision_tower: true`, `freeze_multi_modal_projector: true`). The template treats literal `<image>/<video>/<audio>` in text as media placeholders → escape them in data (`scripts/build_md_xl.py` does). |-| **stage** | `stage: sft` (`lora_*.yaml`, `qwen35_9b_full_sft*.yaml`, `full_md_bal.yaml`), `stage: dpo` (`dpo_*.yaml`) | DPO: `pref_beta: 0.1`, `pref_loss: sigmoid`, lr 5e-6, cutoff 4096 (pairs are ~2× the memory of SFT). |-| **method** | `finetuning_type: lora` (r16/r64, alpha 2r, dropout 0.05, `lora_target: all`) or `full` | full FT uses `deepspeed: …/ds_z3_config.json` (ZeRO-3; `configs/lf/ds_z3_offload.json` for 2-GPU CPU-offload). |-| **packing** | `packing: true`, `neat_packing: true`, `cutoff_len: 8192` | neat_packing requires `per_device_eval_batch_size: 1`; keep `save_steps ≤ eval_steps`. |-| **attention** | `flash_attn: fa2` | plus fla/causal-conv1d for the Gated-DeltaNet layers (triton 3.7.1 on Hopper). |-| **tokenized cache** | `tokenized_path: …/tokenized_<name>_len8192_packed` | second run on the same data skips preprocessing. |-| **launch** | `scripts/lf_train.sh <yaml> key=value…` | = `torchrun --nproc_per_node=$N llamafactory-cli train <abs yaml> key=value` with `CUDA_VISIBLE_DEVICES`, per-run Triton cache; any YAML key can be overridden on the CLI (we override `gradient_accumulation_steps`, `cutoff_len`, `num_train_epochs`). |-| **merge LoRA** | `scripts/lf_export.sh configs/lf/export_<name>.yaml` | `llamafactory-cli export` → a plain HF checkpoint for vLLM. |-| **checkpoints** | `save_steps`, `save_total_limit`, `save_only_model: true` | checkpoint sweeps (`scripts/*_ckpt_sweep.sh`) merge each `checkpoint-N` and evaluate it. |+| dataset registry | each recipe's `dataset_dir` / `dataset_info.json` | `data_specs/dataset_info.json` is a historical snapshot; current local multimodal recipes require additional external registry entries and data |+| data conversion | `llamafactory/to_llamafactory.py` and the other builders | ShareGPT uses `conversations` and `system`; preference sets add `chosen` / `rejected` and registry `ranking: true` |+| current recipes | `configs/lf/sft_*.yaml`, `configs/lf/dpo_*.yaml` | six SFT recipes and two stop-DPO recipes; see [`../configs/lf/README.md`](../configs/lf/README.md) |+| template | `template: qwen3_5`, `enable_thinking: false` | Qwen3.5 is a VLM wrapper; current recipes freeze the vision tower and multimodal projector |+| LoRA | `finetuning_type: lora` | current SFT uses rank 64 / alpha 128; stop-DPO uses rank 16 / alpha 32 |+| stop-DPO | `stage: dpo` | current recipes use `pref_beta: 0.1`, sigmoid loss, learning rate 5e-6, and cutoff 2048; historical execution/geometry-feedback studies used other recipes |+| multimodal packing | `packing: false`, `neat_packing: false` | current SFT recipes disable packing for image-containing data |+| attention | `flash_attn: fa2` | original Qwen3.5/Hopper environment also uses fla, causal-conv1d, and triton 3.7.1 |+| preprocessing cache | `tokenized_path` in each local SFT recipe | reuse only for the same data and preprocessing settings |+| training | `scripts/lf_train.sh <yaml> key=value…` | launches the installed LF CLI with selected GPUs and a local Triton cache; the wrapper still contains workstation paths |+| merging | `scripts/export_lora.sh <run_dir> <base_model>` | builds an export YAML from `configs/lf/export_lora_template.yaml` and calls `scripts/lf_export.sh`; specify the same base model used for the adapter |+| evaluation | `../eval/llm/` | model generation, dialect execution, geometry metrics, and reporting; see its README for the current CLI |++Full-parameter and ZeRO-3 experiments are described in the historical reports. DeepSpeed templates+remain in `configs/`; they do not turn the current LoRA recipes into full-parameter recipes.++## Current SFT example++The complete current recipe is+[`sft_9b_lora64_lr1e-4_len8k_mmmix-plus.yaml`](../configs/lf/sft_9b_lora64_lr1e-4_len8k_mmmix-plus.yaml).+Its data and training settings include:-## Minimal recipe (what `configs/lf/lora_md_xl.yaml` says, annotated)```yaml-model_name_or_path: <path to Qwen3.5-9B>-trust_remote_code: true-flash_attn: fa2stage: sft-do_train: true-finetuning_type: lora # or: full (+ deepspeed: ds_z3_config.json)+finetuning_type: loralora_rank: 64lora_alpha: 128-lora_dropout: 0.05lora_target: all-freeze_vision_tower: true-freeze_multi_modal_projector: true-dataset_dir: <dir with dataset_info.json>-dataset: md_xl_train-eval_dataset: md_xl_val+dataset_dir: /path/to/mm_mix+dataset: 3dcv_mm_img_text_train,3dcv_mm_img_train,md_9b_plus_train+eval_dataset: 3dcv_mm_img_text_valtemplate: qwen3_5enable_thinking: falsecutoff_len: 8192-packing: true-neat_packing: true-tokenized_path: <cache dir>-output_dir: <run dir>+packing: false+neat_packing: falseper_device_train_batch_size: 1-gradient_accumulation_steps: 2 # 4 GPUs × 1 × 2 × 8192 ≈ 64k tokens / step-learning_rate: 1.0e-4+gradient_accumulation_steps: 6+learning_rate: 0.0001num_train_epochs: 1.0-lr_scheduler_type: cosine-warmup_ratio: 0.03bf16: true-gradient_checkpointing: true-per_device_eval_batch_size: 1-eval_strategy: steps-eval_steps: 450-save_steps: 450-save_total_limit: 14-logging_steps: 5-report_to: tensorboard```-Throughput on 4×H100 with this recipe: ≈17k tokens/s (LoRA r64), 270k samples / 293M tokens in 4.8 h; full-parameter ZeRO-3 on 4 GPUs ≈14k tokens/s.-## Pitfalls we hit (details in `docs/REPORT.md` §1)-1. torch 2.9.x + Qwen3.5 (Conv3D vision tower) is rejected by LLaMA-Factory → torch 2.8.0.-2. fla Gated-DeltaNet backward on Hopper needs triton 3.7.1.-3. `neat_packing` + eval batch size > 1 crashes at the first eval.-4. `<image>` as literal text in a sample → "number of images does not match" → escape in data.-5. Triton cache on a network FS races between concurrent runs → per-run local cache.-6. DPO with 8k cutoff OOMs on long GLSL/OpenSCAD pairs on 80 GB → cutoff 4096.-7. All paths handed to LLaMA-Factory must be absolute (it chdirs).+This is an excerpt, not a standalone config. Prepare the named datasets and their registry, and+adapt the model, cache, data, and output paths in the complete file before launching it. From+`finetune/`, after adapting the workstation launchers:++```bash+GPUS=0,1,2,3 bash scripts/lf_train.sh configs/lf/sft_9b_lora64_lr1e-4_len8k_mmmix-plus.yaml+bash scripts/export_lora.sh /absolute/path/to/run /absolute/path/to/the-same-base-model+```++The wrapper forwards LF CLI overrides, but it currently derives its own log and cache paths from+the original YAML before applying those overrides. Set the intended output path in the recipe when+using these wrappers. Changing `env.sh` alone does not relocate their hardcoded conda/LF paths.++## Data and environment checks++- Every `dataset` and `eval_dataset` name must resolve in the registry selected by `dataset_dir`.+ Do not assume the historical registry snapshot contains a current multimodal mix.+- Literal `<image>` tokens in code count as media placeholders too. Run+ `python llamafactory/audit_pairs.py ...` against the prepared pair datasets before publishing;+ the pair builder rejects assistant code containing that literal token.+- Image placeholders must match the image list, media paths must resolve, and image-containing+ rows must use the multimodal settings above.+- The original training environment uses torch 2.8.0 and triton 3.7.1 for its Qwen3.5/Hopper stack;+ see [`SETUP.md`](SETUP.md) and the pinned environment snapshot for the remaining dependencies.+- Keep Triton caches on local disk for concurrent training on the original network filesystem.+- The stop-DPO shell pipeline still expects the historical generation script and output format.+ The current `eval/llm/` API uses different filenames/fields, so migrating that pipeline requires a+ schema adapter and validation of its pairing behavior.++Historical runs and their pitfalls remain in [`REPORT.md`](REPORT.md), [`REPORT_FULL.md`](REPORT_FULL.md),+and [`../llm_finetune_exps.md`](../llm_finetune_exps.md). Their old paths and recipe names are records+of those runs, rather than the current entrypoint inventory.diff --git a/finetune/llamafactory/README.md b/finetune/llamafactory/README.mdnew file mode 100644index 00000000..fa21aec2--- /dev/null+++ b/finetune/llamafactory/README.md@@ -0,0 +1,132 @@+# llamafactory/ — 3DCodeVerse training-data tools++This is the canonical home for the project's LLaMA-Factory data preparation: corpus conversion,+quality checks, multimodal pairs, mix sampling, and publication. It contains project tooling, not+upstream LLaMA-Factory source. General corpus curation stays in [`../../toolkits/`](../../toolkits/);+training configs, environments, and launchers stay in the parent [`finetune/`](../README.md) component.++The seven former data scripts in `../scripts/` forward to the implementation here for CLI compatibility.+For new workflows, call `python llamafactory/<script>.py ...` from `finetune/`. Per-source shell drivers+live in [`runners/`](runners/).++## Script catalog++| script | what it does |+|---|---|+| `convert_subdirs.py` | one `<subdir>_llamafactory/` per corpus subdirectory: ShareGPT `train.json`, registry, `qc.json`, and README. Checks empty/short/mojibake code, duplicates, captions, benchmark overlap, syntax, length, and literal media placeholders |+| `flatten_multifile.py` | merges a Shadertoy shader's Common tab into its passes, and recovers `code.urdf` for Articraft subdirectories whose metadata has no code; writes parquet datasets |+| `qc_repair.py` | checks parquet datasets with glslangValidator or Python `ast.parse`, attempts repairs, re-validates them, and reports duplicate groups within/across subdirectories |+| `glsl_render_batch.py`, `glsl_render.py`, `render_verify_glsl.py` | render shaders in headless WebGL2 and check non-uniform images and animation |+| `finalize_glsl.py` | prefer the original program, use a repair if it compiles, otherwise record `compiles=False`; writes `compiles` / `repair` columns |+| `verify_unflattened_glsl.py` | compile-check the Shadertoy shards without a flattened variant and record the same `compiles` / `repair` fields |+| `build_mm_dataset.py` | build `mm_img`, `mm_img_text`, and `mm_text` variants from the same source samples and split |+| `build_image_dataset.py` | earlier single-variant image builder, retained for reproducing the first pilot |+| `build_pairs_3dcodebench.py` | build text→code, image→code, and image+text→code pairs for one source subdirectory. Text variants retain usable captions; image-only pairs use one row per sample. Reads renders by scanning tars rather than trusting stale byte offsets |+| `shrink_pair_images.py` | re-encode extracted PNG renders as JPEGs at a configurable maximum size (default 448 px); source tars retain the originals |+| `audit_pairs.py` | check pair datasets for absolute/missing/non-JPEG media paths, placeholder counts across all turns, turn shape, fenced code, empty systems, and row-count/QC mismatches; exits nonzero on findings |+| `upload_pairs.py` | publish pair datasets under `<source>/<subdir>_llamafactory_{text,img,imgtext}` and optionally renders under `<source>/pair_renders/`, then update the Hub registry |+| `drop_old_pair_trees.py` | delete superseded top-level pair trees after checking that the Hub registry no longer points into them |+| `build_md_max.py`, `sample_mix.py` | build the combined corpus and sample mixes with per-dialect token budgets and captions-per-sample caps |+| `build_refine_sft.py` | turn improved harness refinement transitions into message JSONL for SFT |+| `to_llamafactory.py` | convert message JSONL to ShareGPT JSON and registry entries |+| `build_hf_dataset.py` | package prepared corpus data, historical preference sets, held-out tests, and mix descriptions for the Hub |+| `cache_ntok.py`, `ntok_cache.py` | calculate/cache token counts for data builders |+| `extract.py` | dialect-aware extraction from free-form model output for data QC |+| `analyze_3dcodeverse.py`, `fetch_hf_files.py` | corpus inventory and selective Hub downloads |+| `runners/` | per-source drivers for the pair-building workflows |++## Formats and stage boundaries++`convert_subdirs.py` writes JSON, while the flattening/pair builders and downstream QC tools use+parquet. These are separate workflows with prepared inputs; the directory move does not provide a+missing JSON→parquet conversion stage.++```text+<subdir>_llamafactory/ corpus conversion+ train.json ShareGPT system + conversations[human,gpt]+ dataset_info.json local registry entry+ qc.json counts and quality findings+ README.md provenance and filtering notes+<subdir>_llamafactory_flat/ flattened shader variants+ train.parquet code and verification fields where produced+<subdir>_llamafactory_{text,img,imgtext}/+ train.parquet pairs, plus images for multimodal variants+ dataset_info.json+ qc.json+```++For multimodal data, the total number of `<image>` placeholders across **all conversation turns**+must equal `len(images)`. Literal placeholders inside assistant code also count; the pair builder+rejects those samples and the audit checks both turns. Use `packing: false` and `neat_packing: false`+for image-containing rows.++The audit validates prepared outputs; it does not rewrite image paths or convert formats. Prepare+relative JPEG media paths and the corresponding files before expecting it to pass.++## Requirements and paths++The original data environment uses Python 3.12 with `pandas`, `pyarrow`, `transformers` (tokenizer),+Pillow, and `huggingface_hub` for publication. Particular scripts also need `requests`, Playwright+and Chromium, glslangValidator, Blender, or OpenSCAD. See [`../env/`](../env/) for environment snapshots+and [`../docs/SETUP.md`](../docs/SETUP.md) for the original setup.++Many input, model, output, and binary paths still refer to `/wekafs/ict/hx_624/...` in script constants+or defaults. Inspect and adapt them before running on another machine. Not every path has a CLI+option, and setting `env.sh` alone does not change those constants. Hub operations require access+to the relevant datasets; upload and deletion scripts change remote data.++| dialect | verification used by the workflows |+|---|---|+| Blender-Python | Python syntax checks; execution workflows use headless Blender, with scipy needed by some reference scripts |+| CadQuery | Python syntax checks; execution workflows use CadQuery |+| OpenSCAD | OpenSCAD CLI |+| GLSL | glslangValidator and headless WebGL2; compilation and visible rendering are distinct checks |+| three.js / web | headless Chromium through Playwright |+| URDF | recovery from source tars and XML checks |++## Examples++Run from `finetune/` after preparing inputs and adapting the script-specific paths. These are examples+for separate stages, not a single ready-to-run pipeline:++```bash+# Corpus metadata -> ShareGPT JSON datasets+python llamafactory/convert_subdirs.py --out /path/to/json-datasets --workers 16++# Flatten shader passes / recover URDF from the prepared source corpus+python llamafactory/flatten_multifile.py --shadertoy --out /path/to/flat-datasets+python llamafactory/flatten_multifile.py --urdf_tars /path/to/tars --out /path/to/flat-datasets++# QC expects prepared parquet datasets+python llamafactory/qc_repair.py --pass all --root /path/to/parquet-datasets++# Mix sampling expects message-format train.jsonl and val.jsonl+python llamafactory/sample_mix.py --src /path/to/md_max --out /path/to/mix \+ --budget "blender=all,cadquery=25M,glsl=25M" --captions_per_sample 1++# Multimodal variants use the builder's prepared sources+python llamafactory/build_mm_dataset.py --views 4 --out /path/to/mm++# Audit every pair directory under the supplied root before publication+python llamafactory/audit_pairs.py --roots /path/to/pairs --media_root /path/to/media+```++## Lessons from the data work++1. **Check the checker.** Many shader failures came from validation defects: sampler types, missing+ uniforms, transformations of minified code, or the GLSL ES profile. Use known-good controls.+2. **Keep filtering decisions explicit.** Record QC findings and the reasons for drops. Benchmark+ membership and runtime validity serve different purposes; an experiment must declare its split+ and filtering policy rather than assuming conversion enforces it.+3. **Benchmark overlap belongs to split selection.** `convert_subdirs.py` counts overlaps with the+ 212 benchmark factories in `qc.json` and keeps those rows. It previously dropped entire instance+ folders. Counting overlap does not itself make a later training mix leak-free.+4. **Compare before deleting.** Same-named trees previously held different contents; the root copy+ was sometimes newer. Compare contents and verify the published registry before retiring a copy.+5. **Audit global invariants globally.** A full scan once found 36,964 absolute media paths that spot+ checks missed. Also validate the audit itself: parquet sequence values can be numpy arrays,+ which invalidated an earlier list-only check.++Results and historical context remain in [`../llm_finetune_exps.md`](../llm_finetune_exps.md) and+[`../docs/REPORT.md`](../docs/REPORT.md). Current training recipes are listed in+[`../configs/lf/README.md`](../configs/lf/README.md).diff --git a/toolkits/llamafactory/analyze_3dcodeverse.py b/finetune/llamafactory/analyze_3dcodeverse.pysimilarity index 100%rename from toolkits/llamafactory/analyze_3dcodeverse.pyrename to finetune/llamafactory/analyze_3dcodeverse.pydiff --git a/toolkits/llamafactory/audit_pairs.py b/finetune/llamafactory/audit_pairs.pysimilarity index 93%rename from toolkits/llamafactory/audit_pairs.pyrename to finetune/llamafactory/audit_pairs.pyindex 3297f383..5d3897dc 100644--- a/toolkits/llamafactory/audit_pairs.py+++ b/finetune/llamafactory/audit_pairs.py@@ -3,7 +3,7 @@Sampling proves a problem exists; it cannot prove one does not. Checking four datasets said "paths are relative";a full scan of all 411 found 36,964 absolute paths in three of them. This scans everything.-usage: python scripts/audit_pairs.py [--media_root DIR] [--fix]+usage: python llamafactory/audit_pairs.py [--media_root DIR] [--roots GLOB]"""import argparseimport collections@@ -52,13 +52,13 @@ def main():if not str(r.get("system") or "").strip():bad["empty_system"] += 1if kind == "text":- if "<image>" in conv[0]["value"]:+ if any("<image>" in t["value"] for t in conv):bad["placeholder_in_text_pair"] += 1continue# `or []` on a numpy array raises rather than defaulting; the same numpy-vs-list confusion as aboveraw = r.get("images")imgs = [str(x) for x in raw] if raw is not None else []- if conv[0]["value"].count("<image>") != len(imgs):+ if sum(t["value"].count("<image>") for t in conv) != len(imgs):bad["placeholder_mismatch"] += 1for x in imgs:tot += 1diff --git a/toolkits/llamafactory/build_hf_dataset.py b/finetune/llamafactory/build_hf_dataset.pysimilarity index 97%rename from toolkits/llamafactory/build_hf_dataset.pyrename to finetune/llamafactory/build_hf_dataset.pyindex f5a08d6f..ba656b62 100644--- a/toolkits/llamafactory/build_hf_dataset.py+++ b/finetune/llamafactory/build_hf_dataset.py@@ -1,7 +1,7 @@"""Package the finetuning data as a HuggingFace dataset in LLaMA-Factory (ShareGPT) format.Design: publish the *corpus* + the *recipes* + the *derived* sets, not the 15 near-duplicate training mixes-(they are all samples of the same corpus and would multiply the size for nothing — `scripts/sample_mix.py`+(they are all samples of the same corpus and would multiply the size for nothing — `llamafactory/sample_mix.py`regenerates any of them from a one-line budget spec).sft/<dialect>/train-*.parquet the md_max corpus (768k prompt→code pairs), sharded by dialect@@ -12,7 +12,7 @@ regenerates any of them from a one-line budget spec).mixes.json the exact budget spec of every mix used in the reportREADME.md dataset card-usage: python scripts/build_hf_dataset.py [--out /wekafs/ict/hx_624/hf_ds] [--push ilabai/3dcodeverse-llamafactory] [--private]+usage: python llamafactory/build_hf_dataset.py [--out /wekafs/ict/hx_624/hf_ds] [--push ilabai/3dcodeverse-llamafactory] [--private]"""import argparseimport collections@@ -193,7 +193,7 @@ llamafactory-cli train your_config.yaml # with: dataset_dir: data/3dcv, data## Reproducing the mixes from the reportThe report's training mixes are *samples* of this corpus, so they are described by a budget rather than shipped-as duplicated files (`mixes.json` has the exact specs; `scripts/sample_mix.py` in the repo applies them):+as duplicated files (`mixes.json` has the exact specs; `llamafactory/sample_mix.py` in the repo applies them):| mix | pairs | tokens | captions/sample | result ||---|---|---|---|---|diff --git a/toolkits/llamafactory/build_image_dataset.py b/finetune/llamafactory/build_image_dataset.pysimilarity index 98%rename from toolkits/llamafactory/build_image_dataset.pyrename to finetune/llamafactory/build_image_dataset.pyindex f92b014b..5803e105 100644--- a/toolkits/llamafactory/build_image_dataset.py+++ b/finetune/llamafactory/build_image_dataset.py@@ -14,7 +14,7 @@ Sources (all already on disk):threejs_distill 4 views per sampleanimation2code 4 views per sample-usage: python scripts/build_image_dataset.py [--out data/img_code] [--views 1] [--max_per_source 20000]+usage: python llamafactory/build_image_dataset.py [--out data/img_code] [--views 1] [--max_per_source 20000]"""import argparseimport collectionsdiff --git a/toolkits/llamafactory/build_md_max.py b/finetune/llamafactory/build_md_max.pysimilarity index 98%rename from toolkits/llamafactory/build_md_max.pyrename to finetune/llamafactory/build_md_max.pyindex 409192f2..d25612ec 100644--- a/toolkits/llamafactory/build_md_max.py+++ b/finetune/llamafactory/build_md_max.py@@ -17,7 +17,7 @@ Deliberately excluded and why:Each sample yields up to 3 training pairs (captions: instruction / detailed / factory), deduped by (prompt, code).Token length is estimated as tokens(code) + tokens(prompt) + template overhead, tokenizing each unique code/prompt once.-usage: python scripts/build_md_max.py [--max_tokens 8192] [--captions 3] [--out data/md_max]+usage: python llamafactory/build_md_max.py [--max_tokens 8192] [--captions 3] [--out data/md_max]"""import argparseimport glob@@ -26,13 +26,11 @@ import jsonimport osimport randomimport re-import sysimport unicodedatafrom concurrent.futures import ProcessPoolExecutorimport pandas as pd-sys.path.insert(0, "/wekafs/ict/hx_624/llm-ft/scripts")META = "/wekafs/ict/hx_624/data/3dcodeverse_meta"FILES = "/wekafs/ict/hx_624/data/3dcodeverse_files"MD = "/wekafs/ict/hx_624/llm-ft/data/multidialect"diff --git a/toolkits/llamafactory/build_mm_dataset.py b/finetune/llamafactory/build_mm_dataset.pysimilarity index 99%rename from toolkits/llamafactory/build_mm_dataset.pyrename to finetune/llamafactory/build_mm_dataset.pyindex 806cf266..f7fc2ffd 100644--- a/toolkits/llamafactory/build_mm_dataset.py+++ b/finetune/llamafactory/build_mm_dataset.py@@ -16,7 +16,7 @@ Format notes (checked against LLaMA-Factory's loader):Sources: bioinspired3d (tars, 4 views), 3dcodebench factories (tars, 4 views, benchmark factories excluded),blender_distill / threejs_distill / animation2code (file trees, 4 views).-usage: python scripts/build_mm_dataset.py [--views 4] [--out data/mm]+usage: python llamafactory/build_mm_dataset.py [--views 4] [--out data/mm]"""import argparseimport collectionsdiff --git a/toolkits/llamafactory/build_pairs_3dcodebench.py b/finetune/llamafactory/build_pairs_3dcodebench.pysimilarity index 97%rename from toolkits/llamafactory/build_pairs_3dcodebench.pyrename to finetune/llamafactory/build_pairs_3dcodebench.pyindex 82b006f4..359c3eed 100644--- a/toolkits/llamafactory/build_pairs_3dcodebench.py+++ b/finetune/llamafactory/build_pairs_3dcodebench.py@@ -5,7 +5,7 @@ separate row -- an augmentation three independent runs showed does not help. Herpair type with one caption, and the renders come from the sample's own slice of the tar (metadata carries`tar`/`byte_start`/`byte_len`, so a ranged read fetches one sample without downloading a 2 GB shard).- python scripts/build_pairs_3dcodebench.py --subdir 3dcodebench/factories_geo --out DIR [--views 4] [--workers 8]+ python llamafactory/build_pairs_3dcodebench.py --subdir 3dcodebench/factories_geo --out DIR [--views 4] [--workers 8][--limit N] [--local_tars DIR]Writes <out>/<name>_{text,img,imgtext}/train.parquet + dataset_info.json + qc.json.@@ -305,6 +305,12 @@ def main():if len(code.strip()) < 40:qc["short_code"] += 1return None+ # LLaMA-Factory counts <image> placeholders across EVERY turn, so a literal "<image>" in the answer+ # (Shadertoy writes "// <image>" as a multipass section marker) reads as one more image than the row+ # carries and aborts the whole run hours in. Seven such rows shipped once; drop them at the source.+ if "<image>" in code:+ qc["literal_image_token"] += 1+ return Nonecaps = r.get("captions")caps = dict(caps) if isinstance(caps, dict) else (json.loads(caps) if isinstance(caps, str) else {})caps_list = captions_of(caps)diff --git a/toolkits/llamafactory/build_refine_sft.py b/finetune/llamafactory/build_refine_sft.pysimilarity index 100%rename from toolkits/llamafactory/build_refine_sft.pyrename to finetune/llamafactory/build_refine_sft.pydiff --git a/toolkits/llamafactory/cache_ntok.py b/finetune/llamafactory/cache_ntok.pysimilarity index 100%rename from toolkits/llamafactory/cache_ntok.pyrename to finetune/llamafactory/cache_ntok.pydiff --git a/toolkits/llamafactory/convert_subdirs.py b/finetune/llamafactory/convert_subdirs.pysimilarity index 98%rename from toolkits/llamafactory/convert_subdirs.pyrename to finetune/llamafactory/convert_subdirs.pyindex c63f1a7a..90edeab7 100644--- a/toolkits/llamafactory/convert_subdirs.py+++ b/finetune/llamafactory/convert_subdirs.py@@ -15,7 +15,7 @@ Quality checks applied per sample (a sample can fail several):scad-glsl, tag check for html) / too_long (>8192 tokens, still exported but flagged) / media_placeholder(literal <image>/<video>/<audio> — breaks the qwen3_5 template unless escaped; we escape it and flag it)-usage: python scripts/convert_subdirs.py [--out DIR] [--workers 16] [--limit_subdirs N]+usage: python llamafactory/convert_subdirs.py [--out DIR] [--workers 16] [--limit_subdirs N]"""import argparseimport ast@@ -25,13 +25,11 @@ import hashlibimport jsonimport osimport re-import sysimport unicodedatafrom concurrent.futures import ProcessPoolExecutorimport pandas as pd-sys.path.insert(0, "/wekafs/ict/hx_624/llm-ft/scripts")META = "/wekafs/ict/hx_624/data/3dcodeverse_meta"FILES = "/wekafs/ict/hx_624/data/3dcodeverse_files"BENCH_DIR = "/wekafs/ict/hx_624/data/3dcodebench/data"@@ -234,7 +232,7 @@ def convert_one(args):issues = "\n".join(f"- **{k}**: {v}" for k, v in sorted(qc.items(), key=lambda x: -x[1])) or "- none"open(os.path.join(d, "README.md"), "w").write(f"""# {sub} — LLaMA-Factory format-Generated by `finetune/scripts/convert_subdirs.py` from `{sub}/metadata.parquet` (or its file tree).+Generated by `finetune/llamafactory/convert_subdirs.py` from `{sub}/metadata.parquet` (or its file tree).| | ||---|---|diff --git a/toolkits/llamafactory/drop_old_pair_trees.py b/finetune/llamafactory/drop_old_pair_trees.pysimilarity index 100%rename from toolkits/llamafactory/drop_old_pair_trees.pyrename to finetune/llamafactory/drop_old_pair_trees.pydiff --git a/toolkits/llamafactory/extract.py b/finetune/llamafactory/extract.pysimilarity index 100%rename from toolkits/llamafactory/extract.pyrename to finetune/llamafactory/extract.pydiff --git a/toolkits/llamafactory/fetch_hf_files.py b/finetune/llamafactory/fetch_hf_files.pysimilarity index 100%rename from toolkits/llamafactory/fetch_hf_files.pyrename to finetune/llamafactory/fetch_hf_files.pydiff --git a/toolkits/llamafactory/finalize_glsl.py b/finetune/llamafactory/finalize_glsl.pysimilarity index 97%rename from toolkits/llamafactory/finalize_glsl.pyrename to finetune/llamafactory/finalize_glsl.pyindex bbc883c1..cfde0157 100644--- a/toolkits/llamafactory/finalize_glsl.py+++ b/finetune/llamafactory/finalize_glsl.py@@ -6,10 +6,9 @@ Order of preference, so the data stays as close to the author's source as possib3. otherwise -> kept, marked compiles=FalseWrites the `compiles` / `repair` columns back into every shadertoy *_llamafactory_flat parquet."""-import collections, glob, hashlib, json, os, re, sys+import collections, glob, hashlib, json, os, refrom concurrent.futures import ProcessPoolExecutorimport pandas as pd-sys.path.insert(0, "/wekafs/ict/hx_624/llm-ft/scripts")from qc_repair import code_of, glsl_compileREPAIR = "/wekafs/ict/hx_624/glsl_repair"diff --git a/toolkits/llamafactory/flatten_multifile.py b/finetune/llamafactory/flatten_multifile.pysimilarity index 97%rename from toolkits/llamafactory/flatten_multifile.pyrename to finetune/llamafactory/flatten_multifile.pyindex 23e9af06..ef6bdb0c 100644--- a/toolkits/llamafactory/flatten_multifile.py+++ b/finetune/llamafactory/flatten_multifile.py@@ -14,8 +14,8 @@ Two sources are multi-file in the original dataset and both can be flattened exaEverything is then re-emitted in LLaMA-Factory ShareGPT format next to the other converted subdirs.usage:- python scripts/flatten_multifile.py --shadertoy [--out /wekafs/ict/hx_624/hf_subdirs]- python scripts/flatten_multifile.py --urdf_tars /wekafs/ict/hx_624/data/3dcodeverse_tars+ python llamafactory/flatten_multifile.py --shadertoy [--out /wekafs/ict/hx_624/hf_subdirs]+ python llamafactory/flatten_multifile.py --urdf_tars /wekafs/ict/hx_624/data/3dcodeverse_tars"""import argparseimport collectionsdiff --git a/toolkits/llamafactory/glsl_render.py b/finetune/llamafactory/glsl_render.pysimilarity index 100%rename from toolkits/llamafactory/glsl_render.pyrename to finetune/llamafactory/glsl_render.pydiff --git a/toolkits/llamafactory/glsl_render_batch.py b/finetune/llamafactory/glsl_render_batch.pysimilarity index 100%rename from toolkits/llamafactory/glsl_render_batch.pyrename to finetune/llamafactory/glsl_render_batch.pydiff --git a/toolkits/llamafactory/ntok_cache.py b/finetune/llamafactory/ntok_cache.pysimilarity index 90%rename from toolkits/llamafactory/ntok_cache.pyrename to finetune/llamafactory/ntok_cache.pyindex 7884a458..c984bef2 100644--- a/toolkits/llamafactory/ntok_cache.py+++ b/finetune/llamafactory/ntok_cache.py@@ -1,4 +1,4 @@-"""Helper for build scripts: load cached token counts for a jsonl (scripts/cache_ntok.py), falling back to tokenizing."""+"""Helper for build scripts: load cached token counts for a jsonl (llamafactory/cache_ntok.py), falling back to tokenizing."""import osdef load_ntok(path, tok=None):c = path + ".ntok"diff --git a/toolkits/llamafactory/qc_repair.py b/finetune/llamafactory/qc_repair.pysimilarity index 99%rename from toolkits/llamafactory/qc_repair.pyrename to finetune/llamafactory/qc_repair.pyindex 22fe2afd..a62e58ce 100644--- a/toolkits/llamafactory/qc_repair.py+++ b/finetune/llamafactory/qc_repair.py@@ -13,7 +13,7 @@ Three passes, each writing a report and (where it helps) a repaired copy of thededup find exact-duplicate programs *within* and *across* subdirs of the same source and write theduplicate groups so the upstream dataset can be cleaned too.-usage: python scripts/qc_repair.py --root /wekafs/ict/hx_624/hf_subdirs [--pass glsl|python|dedup|all] [--workers 32]+usage: python llamafactory/qc_repair.py --root /wekafs/ict/hx_624/hf_subdirs [--pass glsl|python|dedup|all] [--workers 32]"""import argparseimport astdiff --git a/toolkits/llamafactory/render_verify_glsl.py b/finetune/llamafactory/render_verify_glsl.pysimilarity index 96%rename from toolkits/llamafactory/render_verify_glsl.pyrename to finetune/llamafactory/render_verify_glsl.pyindex c3773022..33a274d1 100644--- a/toolkits/llamafactory/render_verify_glsl.py+++ b/finetune/llamafactory/render_verify_glsl.py@@ -4,12 +4,11 @@ Samples (or exhaustively walks) the unique programs in the flattened shadertoy dWebGL2, and records status / distinct colours / pixel std / whether the image changes with iTime. Writes`renders_ok`, `renders_status` back into the parquet rows when run with --write.-usage: python scripts/render_verify_glsl.py [--sample 3000] [--workers 8] [--write]+usage: python llamafactory/render_verify_glsl.py [--sample 3000] [--workers 8] [--write]"""-import argparse, collections, glob, hashlib, json, os, random, re, sys, tempfile+import argparse, collections, glob, hashlib, json, os, random, re, tempfilefrom concurrent.futures import ProcessPoolExecutorimport pandas as pd-sys.path.insert(0, "/wekafs/ict/hx_624/llm-ft/eval")os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/wekafs/ict/hx_624/cache/ms-playwright")def code_of(m):diff --git a/toolkits/llamafactory/runners/run_3dcb_pairs.sh b/finetune/llamafactory/runners/run_3dcb_pairs.shsimilarity index 84%rename from toolkits/llamafactory/runners/run_3dcb_pairs.shrename to finetune/llamafactory/runners/run_3dcb_pairs.shindex 44ff4e69..53727e01 100755--- a/toolkits/llamafactory/runners/run_3dcb_pairs.sh+++ b/finetune/llamafactory/runners/run_3dcb_pairs.sh@@ -3,7 +3,11 @@# Renders come from a sequential scan of each tar, NOT from metadata's byte offsets: for instances_geo only 577# of 1,953 offsets point at a real tar header (the archives were repacked after the metadata was written), and# trusting them silently cost 70% of the image pairs.-set -uo pipefail; cd /wekafs/ict/hx_624/llm-ft+set -uo pipefail+FINETUNE_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)"+PAIR_BUILDER="$FINETUNE_DIR/llamafactory/build_pairs_3dcodebench.py"+cd "$FINETUNE_DIR" || exit 1+mkdir -p logssource /wekafs/ict/hx_624/anaconda3/etc/profile.d/conda.sh; conda activate llmftOUT=/wekafs/ict/hx_624/hf_pairsfor SUB in instances_tex instances_geo_full instances_tex_full; do@@ -27,6 +31,6 @@ os.makedirs(d, exist_ok=True); shutil.copy(p, f"{d}/metadata.parquet")PYfiecho "[3dcb] $(date) building $SUB"- python scripts/build_pairs_3dcodebench.py --subdir 3dcodebench/$SUB --out $OUT --workers 16 --views 4 2>&1 | tail -6+ python "$PAIR_BUILDER" --subdir 3dcodebench/$SUB --out $OUT --workers 16 --views 4 2>&1 | tail -6doneecho "[3dcb] ALL DONE"diff --git a/toolkits/llamafactory/runners/run_other_pairs.sh b/finetune/llamafactory/runners/run_other_pairs.shsimilarity index 84%rename from toolkits/llamafactory/runners/run_other_pairs.shrename to finetune/llamafactory/runners/run_other_pairs.shindex 7e1b20cf..143b5bfd 100755--- a/toolkits/llamafactory/runners/run_other_pairs.sh+++ b/finetune/llamafactory/runners/run_other_pairs.sh@@ -2,7 +2,11 @@# The remaining parquet-backed sources. deepcad and bioinspired3d have renders, so they get all three pair types;# articraft (URDF) and thingiverse have none, so only text->code is produced and the img/imgtext folders are# simply not written -- an empty image dataset would be worse than an absent one.-set -uo pipefail; cd /wekafs/ict/hx_624/llm-ft+set -uo pipefail+FINETUNE_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)"+PAIR_BUILDER="$FINETUNE_DIR/llamafactory/build_pairs_3dcodebench.py"+cd "$FINETUNE_DIR" || exit 1+mkdir -p logssource /wekafs/ict/hx_624/anaconda3/etc/profile.d/conda.sh; conda activate llmftOUT=/wekafs/ict/hx_624/hf_pairs_othermkdir -p $OUT@@ -10,11 +14,11 @@ build () {local SUB=$1 DIA=$2local Q="$OUT/$(basename $SUB)_llamafactory_text/qc.json"[ -f "$Q" ] && { echo "[oth] $SUB already built"; return; }- python scripts/build_pairs_3dcodebench.py --subdir "$SUB" --dialect "$DIA" --out $OUT \+ python "$PAIR_BUILDER" --subdir "$SUB" --dialect "$DIA" --out $OUT \--workers 12 --views 4 --img_root /wekafs/ict/hx_624/data/pair_images > logs/pairs_oth_$(basename $SUB).log 2>&1echo "[oth] $(date +%H:%M) $SUB $(grep -E '^\[pairs\] (text|img|imgtext)' logs/pairs_oth_$(basename $SUB).log | tr '\n' ' ' | tr -s ' ' | cut -c1-110)"}-export -f build; export OUT+export -f build; export PAIR_BUILDER OUTLIST=""for d in /wekafs/ict/hx_624/data/3dcodeverse_meta/deepcad/*/metadata.parquet; do LIST="$LIST deepcad/$(basename $(dirname $d)):cadquery"; donefor d in /wekafs/ict/hx_624/data/3dcodeverse_meta/bioinspired3d/*/metadata.parquet; do LIST="$LIST bioinspired3d/$(basename $(dirname $d)):blender"; donediff --git a/toolkits/llamafactory/runners/run_shadertoy_pairs.sh b/finetune/llamafactory/runners/run_shadertoy_pairs.shsimilarity index 81%rename from toolkits/llamafactory/runners/run_shadertoy_pairs.shrename to finetune/llamafactory/runners/run_shadertoy_pairs.shindex b24896b9..54f0205b 100755--- a/toolkits/llamafactory/runners/run_shadertoy_pairs.sh+++ b/finetune/llamafactory/runners/run_shadertoy_pairs.sh@@ -3,7 +3,11 @@# in metadata are not reliable), and the code comes from whichever verified parquet exists for that shard --# *_llamafactory_flat for the 68 that could be flattened, the separately verified tree for the other 44 -- so the# pairs carry a compiles column instead of being unverified GLSL.-set -uo pipefail; cd /wekafs/ict/hx_624/llm-ft+set -uo pipefail+FINETUNE_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)"+PAIR_BUILDER="$FINETUNE_DIR/llamafactory/build_pairs_3dcodebench.py"+cd "$FINETUNE_DIR" || exit 1+mkdir -p logssource /wekafs/ict/hx_624/anaconda3/etc/profile.d/conda.sh; conda activate llmftOUT=/wekafs/ict/hx_624/hf_pairs_shadertoyVER="/wekafs/ict/hx_624/hf_subdirs/shadertoy,/wekafs/ict/hx_624/hf_subdirs_verified/shadertoy"@@ -13,12 +17,12 @@ build () {local SUB=$1local Q="$OUT/${SUB}_llamafactory_img/qc.json"if [ -f "$Q" ]; then echo "[st] $SUB already built — skipping"; return; fi- python scripts/build_pairs_3dcodebench.py --subdir shadertoy/$SUB --dialect glsl \+ python "$PAIR_BUILDER" --subdir shadertoy/$SUB --dialect glsl \--out $OUT --workers 12 --views 4 --verified_from "$VER" > logs/pairs_st_$SUB.log 2>&1local line=$(grep -E "^\[pairs\] (text|img|imgtext)" logs/pairs_st_$SUB.log | tr '\n' ' ' | tr -s ' ')echo "[st] $(date +%H:%M) $SUB ${line:-FAILED: $(grep -iE 'error|Traceback' logs/pairs_st_$SUB.log | tail -1 | cut -c1-90)}"}-export -f build; export OUT VER+export -f build; export PAIR_BUILDER OUT VERSHARDS=$(ls -d /wekafs/ict/hx_624/data/3dcodeverse_meta/shadertoy/*/ | xargs -n1 basename)echo "[st] $(echo "$SHARDS" | wc -l) shards, $PAR at a time"echo "$SHARDS" | xargs -P $PAR -I{} bash -c 'build "$@"' _ {}diff --git a/toolkits/llamafactory/sample_mix.py b/finetune/llamafactory/sample_mix.pysimilarity index 98%rename from toolkits/llamafactory/sample_mix.pyrename to finetune/llamafactory/sample_mix.pyindex 548d8489..283585b4 100644--- a/toolkits/llamafactory/sample_mix.py+++ b/finetune/llamafactory/sample_mix.py@@ -2,7 +2,7 @@md_max is ~0.77M pairs / ~0.9B tokens and 97% CadQuery+GLSL; every real run wants a different slice of it.- python scripts/sample_mix.py --out data/md_27b_max \+ python llamafactory/sample_mix.py --out data/md_27b_max \--budget blender=all,openscad=all,threejs=all,web=all,cadquery=25M,glsl=25M \[--src data/md_max] [--captions_per_sample 2] [--seed 5]diff --git a/toolkits/llamafactory/shrink_pair_images.py b/finetune/llamafactory/shrink_pair_images.pysimilarity index 95%rename from toolkits/llamafactory/shrink_pair_images.pyrename to finetune/llamafactory/shrink_pair_images.pyindex 1f4b0e49..7059157f 100644--- a/toolkits/llamafactory/shrink_pair_images.py+++ b/finetune/llamafactory/shrink_pair_images.py@@ -5,7 +5,7 @@ dataset. Every vision tower we train downsamples to 448x448 anyway (our own 27B448^2), so a <=448px JPEG loses nothing a training run would have seen — and the untouched originals stay in thecorpus tars for anyone who needs them. Measured saving: 95%, so 84 GB becomes about 4 GB.-usage: python scripts/shrink_pair_images.py [--root DIR] [--out DIR] [--max_px 448] [--quality 88] [--workers 64]+usage: python llamafactory/shrink_pair_images.py [--root DIR] [--out DIR] [--max_px 448] [--quality 88] [--workers 64]"""import argparseimport globdiff --git a/toolkits/llamafactory/to_llamafactory.py b/finetune/llamafactory/to_llamafactory.pysimilarity index 100%rename from toolkits/llamafactory/to_llamafactory.pyrename to finetune/llamafactory/to_llamafactory.pydiff --git a/toolkits/llamafactory/upload_pairs.py b/finetune/llamafactory/upload_pairs.pysimilarity index 98%rename from toolkits/llamafactory/upload_pairs.pyrename to finetune/llamafactory/upload_pairs.pyindex 9af73627..c7101918 100644--- a/toolkits/llamafactory/upload_pairs.py+++ b/finetune/llamafactory/upload_pairs.py@@ -5,7 +5,7 @@ rule as <source>/<subdir>_llamafactory_{text,img,imgtext}, and their renders astop-level pairs/ or pair_images/ would read as if it were a source of its own, which is the same mistake thefive stray *_llamafactory folders made.-usage: python scripts/upload_pairs.py [--batch 24] [--images] [--drop_old]+usage: python llamafactory/upload_pairs.py [--batch 24] [--images] [--drop_old]"""import argparseimport globdiff --git a/toolkits/llamafactory/verify_unflattened_glsl.py b/finetune/llamafactory/verify_unflattened_glsl.pysimilarity index 97%rename from toolkits/llamafactory/verify_unflattened_glsl.pyrename to finetune/llamafactory/verify_unflattened_glsl.pyindex 5bb8dc33..301a0b69 100644--- a/toolkits/llamafactory/verify_unflattened_glsl.py+++ b/finetune/llamafactory/verify_unflattened_glsl.py@@ -6,7 +6,7 @@ from and `flatten_multifile.py` skipped them — exactly 44 of 112, with zero michecker and the same minimal repairs over those, and writes the `compiles` / `repair` columns the flattenedfolders already carry, so a training mix can filter the whole corpus by one rule instead of two.-usage: python scripts/verify_unflattened_glsl.py [--workers 40] [--limit_shards N] [--out DIR]+usage: python llamafactory/verify_unflattened_glsl.py [--workers 40] [--limit_shards N] [--out DIR]"""import argparseimport collectionsdiff --git a/finetune/scripts/README.md b/finetune/scripts/README.mdindex 9d775b76..77bd6dcd 100644--- a/finetune/scripts/README.md+++ b/finetune/scripts/README.md@@ -1,19 +1,34 @@-# scripts/+# scripts/ — launchers and compatibility entrypoints-Everything here is generic. Per-experiment drivers were one-offs bound to a single config; they are in-`archive/scripts/` (kept on disk, not synced).+Training/export launchers and workstation orchestration stay here. Canonical LLaMA-Factory data+implementations live in [`../llamafactory/`](../llamafactory/); see its README for the script catalog.| script | what it does ||---|---|| `lf_train.sh <config>` | run a LLaMA-Factory config on `GPUS=<ids>` |-| `train_when_free.sh <config> [need_mib] [log]` | wait for a card that is *still* free at launch, retry if another user takes it mid-load |-| `free_gpus.sh [need_mib] [gpu_list]` | list cards with that much free memory (the box is shared) |+| `train_when_free.sh <config> [need_mib] [log]` | wait for a free card and retry a training launch after a detected OOM |+| `free_gpus.sh [need_mib] [gpu_list]` | list cards with the requested free memory || `lf_export.sh <config>` / `export_lora.sh <run_dir> [base]` | merge a LoRA adapter into `<run_dir>/merged` |-| `stop_dpo_pairs.sh` | build termination preference pairs (`TAG`, `BASE_MODEL` env) |-| `stop_dpo_round.sh <base_model> <tag>` | one full stop-DPO round: pairs → train → merge |-| `supervisor2.sh` | keeps the queue in `queue.txt` running and warns on idle cards |-| `sync_repo.sh` | mirror scripts/configs/eval + REPORT.md to the 3dcodeverse repo |+| `stop_dpo_pairs.sh` | build termination preference pairs (`TAG`, `BASE_MODEL` env), using the historical generator/output schema |+| `stop_dpo_round.sh <base_model> <tag>` | historical stop-DPO orchestration: pairs → train → merge |+| `supervisor2.sh` | process the workstation job queue in `queue.txt` |+| `sync_repo.sh` | legacy synchronization from the original workstation's external working tree; its scripts copy excludes Python files to preserve the compatibility forwarders |-Data tooling: `build_pairs_3dcodebench.py` (pair builder), `convert_subdirs.py`, `shrink_pair_images.py`,-`audit_pairs.py` (**run before publishing** — scans every row for absolute paths, missing images and-`<image>`-count mismatches), `upload_pairs.py`, `drop_old_pair_trees.py`, `verify_unflattened_glsl.py`.+These shell scripts retain workstation-specific paths and assumptions. The stop-DPO generation+interface and some queue entries refer to historical scripts; inspect and adapt them before use.+`sync_repo.sh` also predates the current directory ownership and should not be used to synchronize+this layout without updating it.++The following seven Python files are thin CLI forwarders, preserving existing command paths while+running the implementation in `../llamafactory/`:++- `audit_pairs.py`+- `build_pairs_3dcodebench.py`+- `convert_subdirs.py`+- `drop_old_pair_trees.py`+- `shrink_pair_images.py`+- `upload_pairs.py`+- `verify_unflattened_glsl.py`++Use `python llamafactory/<script>.py ...` from `finetune/` for new workflows. Historical per-experiment+drivers were archived outside this checkout; their absence is not filled by these compatibility paths.diff --git a/finetune/scripts/audit_pairs.py b/finetune/scripts/audit_pairs.pyindex 176d3401..a6f32161 100644--- a/finetune/scripts/audit_pairs.py+++ b/finetune/scripts/audit_pairs.py@@ -1,81 +1,14 @@-"""Audit every pair dataset for the invariants that fail silently at train time.+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""-Sampling proves a problem exists; it cannot prove one does not. Checking four datasets said "paths are relative";-a full scan of all 411 found 36,964 absolute paths in three of them. This scans everything.--usage: python scripts/audit_pairs.py [--media_root DIR] [--fix]-"""-import argparse-import collections-import glob-import json-import os--import pandas as pd+import runpy+import sys+from pathlib import Pathdef main():- ap = argparse.ArgumentParser()- ap.add_argument("--media_root", default="/wekafs/ict/hx_624/data/pair_media_root")- ap.add_argument("--roots", default="/wekafs/ict/hx_624/hf_pairs*")- a = ap.parse_args()- dirs = sorted(glob.glob(f"{a.roots}/*_llamafactory_*"))- bad = collections.Counter()- tot = rows = 0- examples = {}- for d in dirs:- kind = os.path.basename(d).rsplit("_", 1)[-1]- try:- df = pd.read_parquet(f"{d}/train.parquet")- except Exception as e:- bad["unreadable_parquet"] += 1- examples.setdefault("unreadable_parquet", f"{d}: {type(e).__name__}")- continue- q = json.load(open(f"{d}/qc.json"))- if len(df) != q["rows"]:- bad["row_count_vs_qc"] += 1- for r in df.to_dict("records"):- rows += 1- # parquet hands back numpy arrays, not lists: an isinstance check against (list, tuple) alone- # marks every single row as malformed, which is a defect in the checker, not the data- conv = r.get("conversations")- try:- conv = list(conv)- except Exception:- conv = None- if conv is None or len(conv) != 2:- bad["bad_turn_shape"] += 1- continue- code = conv[1]["value"]- if not code.startswith("```") or not code.rstrip().endswith("```"):- bad["unfenced_code"] += 1- if not str(r.get("system") or "").strip():- bad["empty_system"] += 1- if kind == "text":- if any("<image>" in t["value"] for t in conv):- bad["placeholder_in_text_pair"] += 1- continue- # `or []` on a numpy array raises rather than defaulting; the same numpy-vs-list confusion as above- raw = r.get("images")- imgs = [str(x) for x in raw] if raw is not None else []- if sum(t["value"].count("<image>") for t in conv) != len(imgs):- bad["placeholder_mismatch"] += 1- for x in imgs:- tot += 1- if x.startswith("/"):- bad["absolute_path"] += 1- examples.setdefault("absolute_path", x)- elif not x.endswith(".jpg"):- bad["not_jpg"] += 1- examples.setdefault("not_jpg", x)- elif not os.path.exists(os.path.join(a.media_root, x)):- bad["unresolvable"] += 1- examples.setdefault("unresolvable", x)- print(f" {len(dirs)} datasets | {rows:,} rows | {tot:,} image references")- print(f" problems: {dict(bad) if bad else 'NONE'}")- for k, v in examples.items():- print(f" example {k}: {v[:100]}")- raise SystemExit(1 if bad else 0)+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"+ sys.path.insert(0, str(scripts))+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")if __name__ == "__main__":diff --git a/finetune/scripts/build_pairs_3dcodebench.py b/finetune/scripts/build_pairs_3dcodebench.pyindex 1dc73647..a6f32161 100644--- a/finetune/scripts/build_pairs_3dcodebench.py+++ b/finetune/scripts/build_pairs_3dcodebench.py@@ -1,383 +1,14 @@-"""Build all three pair types for a 3DCodeVerse subdirectory: text->code, image->code, image+text->code.+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""-The corpus so far only carries text->code, and its row count is inflated ~3x because every caption variant is a-separate row -- an augmentation three independent runs showed does not help. Here each SAMPLE appears once per-pair type with one caption, and the renders come from the sample's own slice of the tar (metadata carries-`tar`/`byte_start`/`byte_len`, so a ranged read fetches one sample without downloading a 2 GB shard).-- python scripts/build_pairs_3dcodebench.py --subdir 3dcodebench/factories_geo --out DIR [--views 4] [--workers 8]- [--limit N] [--local_tars DIR]--Writes <out>/<name>_{text,img,imgtext}/train.parquet + dataset_info.json + qc.json.-"""-import argparse-import collections-import glob-import io-import json-import os-import tarfile-import time--import pandas as pd-import pyarrow.parquet as pq-import requests--META = "/wekafs/ict/hx_624/data/3dcodeverse_meta"-REPO = "ilabai/3dcodeverse"-SYS = {- "web": "You are an expert creative web developer. Write a complete, standalone HTML page (inline CSS/JS, no local assets) that reproduces the requested animation. Output ONLY the code in one ```html block.",- "threejs": "You are an expert three.js developer. Write a complete, standalone HTML page that renders the requested scene with three.js. Output ONLY the code in one ```html block.",- "openscad": "You are an expert in OpenSCAD. Write a complete, standalone OpenSCAD program that builds the requested object. Output ONLY the code in one ```scad block.",- "cadquery": "You are an expert in parametric CAD with CadQuery. Write a complete, standalone CadQuery script that builds the requested part. Output ONLY the code in one ```python block.",- "blender": "You are an expert in procedural 3D modeling with Blender Python (bpy). Write a complete, standalone Blender 5 Python script that builds the requested object from scratch (clear the default scene first). Output ONLY the code in one ```python block.",- "glsl": "You are an expert GLSL shader programmer. Write a complete Shadertoy-style fragment shader. Output ONLY the code in one ```glsl block.",-}-FENCE_RE = __import__('re').compile(r'^```[a-zA-Z]*\n|\n?```$')-FENCE = {"web": "html", "threejs": "html", "openscad": "scad", "cadquery": "python", "blender": "python", "glsl": "glsl"}-ASK_TEXT = {"web": "Write a single-file animated web page that reproduces: ", "threejs": "Write a single-file three.js page that renders: ", "openscad": "Write the OpenSCAD program that builds this object: ", "cadquery": "Write the CadQuery script that builds this part: ", "blender": "Write the Blender Python code that builds this object: ",- "glsl": "Write the shader described here: "}-ASK_IMG = {"web": "These are frames of one animation. Write the single-file animated web page that reproduces it.", "threejs": "These are reference renders of one scene. Write the single-file three.js page that renders it.", "openscad": "These are reference renders of one object. Write the OpenSCAD program that builds it.", "cadquery": "These are reference renders of one part. Write the CadQuery script that builds it.", "blender": "These are reference renders of one object. Write the Blender Python code that builds it.",- "glsl": "These are frames rendered by one shader. Write the shader that produces them."}-ASK_BOTH = {"web": "These are frames of one animation. Write the single-file animated web page that reproduces it.\n\nThe animation: ", "threejs": "These are reference renders of one scene. Write the single-file three.js page that renders it.\n\nThe scene: ", "openscad": "These are reference renders of one object. Write the OpenSCAD program that builds it.\n\nThe object: ", "cadquery": "These are reference renders of one part. Write the CadQuery script that builds it.\n\nThe part: ", "blender": "These are reference renders of one object. Write the Blender Python code that builds it.\n\nThe object: ",- "glsl": "These are frames rendered by one shader. Write the shader that produces them.\n\nThe shader: "}-IMPERATIVE = ("write", "create", "implement", "make", "build", "generate", "design", "produce")---def _to_noun_phrase(v):- """the ASK_* strings already supply the verb, so an imperative caption is reduced to what follows it"""- low = v.lower()- if low.startswith(IMPERATIVE):- for sep in (" that builds ", " that renders ", " that reproduces ", " of ", " for "):- if sep in low:- return v[low.index(sep) + len(sep):].strip()[:1200]- return None- return v[:1200]---def captions_of(caps):- """-> [(caption_type, text)] for EVERY usable variant.-- One program often carries three descriptions (`instruction`, `detailed`, `factory`) written from different- angles. Keeping only the best one makes a smaller corpus, and three separate runs here showed the extra- variants do not raise execution rate — but that is a finding about one training mix, not a property of the- data, so the dataset ships all of them and a training run decides which to use (`caption_type` selects)."""- out = []- for k in ("user_instruction", "instruction", "detailed", "brief", "factory"):- v = (caps.get(k) or "").strip()- if not v or len(v) < 15:- continue- n = _to_noun_phrase(v)- if n:- out.append((k, n))- return out---def fetch_sample(row, token, local_tars=None):- """-> list of render bytes for one sample, via a local tar if we have it, else a ranged read"""- tar_rel, start, length = row["tar"], int(row["byte_start"]), int(row["byte_len"])- if local_tars:- p = os.path.join(local_tars, os.path.basename(tar_rel))- if os.path.exists(p):- with open(p, "rb") as f:- f.seek(start)- blob = f.read(length)- return read_renders(blob)- url = f"https://huggingface.co/datasets/{REPO}/resolve/main/{tar_rel}"- hdr = {"Authorization": f"Bearer {token}", "Range": f"bytes={start}-{start + length - 1}"}- # under a dozen concurrent readers the Hub throttles and times out; returning [] on the first failure looks- # exactly like "this sample has no renders", which silently cost 70% of the image pairs on the first run- for attempt in range(4):- try:- r = requests.get(url, headers=hdr, timeout=300)- if r.status_code in (200, 206):- return read_renders(r.content)- last = f"HTTP {r.status_code}"- except Exception as e:- last = type(e).__name__- time.sleep(2 ** attempt)- raise FetchError(last)---class FetchError(Exception):- """the sample could not be read at all — distinct from a sample that genuinely carries no renders"""---def read_renders(blob):- """A byte slice out of the middle of a tar has no end-of-archive marker, so tarfile treats it as truncated —- and it raises in getmembers(), not in open(), which is why guarding only the open() let one bad sample kill a- whole run. Append the two zero blocks tar expects, and scan defensively."""- blob = blob + b"\0" * 1024- try:- tf = tarfile.open(fileobj=io.BytesIO(blob))- except Exception:- return []- out = []- try:- members = sorted(tf.getmembers(), key=lambda m: m.name)- except Exception:- return []- for m in members:- if m.isfile() and "/renders/" in m.name and m.name.endswith(".png"):- try:- f = tf.extractfile(m)- if f:- out.append((os.path.basename(m.name), f.read()))- except Exception:- pass- return out----def renders_by_scanning(subdir, sample_ids, views, token, local_tars=None, cache_root="/wekafs/ict/hx_624/data/pair_tars"):- """Index every sample's renders by walking each tar once.-- The byte offsets in metadata.parquet cannot be trusted: for 3dcodebench/instances_geo only 577 of 1,953 point- at a real tar header (checked by reading 512 bytes at each offset and looking for the ustar magic), because- the archives were repacked after the metadata was written. Sequential scanning ignores the offsets entirely,- and for a whole-subdirectory conversion it is also cheaper -- one pass per tar instead of one request per- sample.- """- import glob as _glob- from huggingface_hub import hf_hub_download- want = {sid.split("/")[-1] for sid in sample_ids}- found = {}- tars = sorted({r for r in _tars_of(subdir)})- for i, tar_rel in enumerate(tars, 1):- if local_tars:- path = os.path.join(local_tars, os.path.basename(tar_rel))- if not os.path.exists(path):- path = None- else:- path = None- if path is None:- path = hf_hub_download(REPO, tar_rel, repo_type="dataset", token=token, cache_dir=cache_root)- n_before = len(found)- try:- with tarfile.open(path) as tf:- for m in tf:- if not (m.isfile() and "/renders/" in m.name and m.name.endswith(".png")):- continue- key = m.name.split("/")[0]- if key not in want:- continue- lst = found.setdefault(key, [])- if len(lst) >= views:- continue- f = tf.extractfile(m)- if f:- lst.append((os.path.basename(m.name), f.read()))- except Exception as e:- print(f"[pairs] tar {os.path.basename(tar_rel)} unreadable: {type(e).__name__}", flush=True)- print(f"[pairs] scanned {i}/{len(tars)} {os.path.basename(tar_rel)}: "- f"+{len(found)-n_before} samples ({len(found):,}/{len(want):,})", flush=True)- return found---def _has_verify_cols(path):- import pyarrow.parquet as _pq- try:- return {"compiles", "repair"} <= set(_pq.ParquetFile(path).schema_arrow.names)- except Exception:- return False---def _tars_of(subdir):- from huggingface_hub import HfApi- api = HfApi(token=os.environ.get("HF_TOKEN", ""))- return [f for f in api.list_repo_files(REPO, repo_type="dataset")- if f.startswith(f"{subdir}/") and f.endswith(".tar")]---def load_file_tree(subdir, files_root="/wekafs/ict/hx_624/data/3dcodeverse_files"):- """Sources kept as a directory per sample rather than a parquet: animation2code, blender_distill,- threejs_distill. Each sample directory holds the code, captions.json, meta.json and a renders/ folder, so the- renders need no tar scan at all."""- rows = []- for cp in sorted(glob.glob(f"{files_root}/{subdir}/*/*/code.py") + glob.glob(f"{files_root}/{subdir}/*/*/code.html")- + glob.glob(f"{files_root}/{subdir}/*/*/index.html")):- d = os.path.dirname(cp)- caps = {}- cj = os.path.join(d, "captions.json")- if os.path.exists(cj):- try:- caps = json.load(open(cj))- except Exception:- caps = {}- rows.append({"id": os.path.relpath(d, files_root), "code": open(cp, encoding="utf-8", errors="replace").read(),- "captions": caps, "_renders": sorted(glob.glob(os.path.join(d, "renders", "*.png")))})- return rows+import runpy+import sys+from pathlib import Pathdef main():- ap = argparse.ArgumentParser()- ap.add_argument("--subdir", required=True)- ap.add_argument("--out", required=True)- ap.add_argument("--views", type=int, default=4)- ap.add_argument("--workers", type=int, default=8)- ap.add_argument("--limit", type=int, default=None)- ap.add_argument("--local_tars", default=None)- ap.add_argument("--img_root", default="/wekafs/ict/hx_624/data/pair_images")- ap.add_argument("--dialect", default="blender")- # the compile verification lives in the *_llamafactory parquets; carrying it into the pair datasets is the- # difference between "here is some GLSL" and "here is GLSL that a compiler accepted"- # some sources carry no code in metadata at all -- articraft's two URDF subdirs have 6,146 samples whose- # `code` column is empty because the URDF lives in the tar; it was recovered once into the *_llamafactory- # parquets, and this reads it back rather than recovering it a second time- ap.add_argument("--code_from", default=None, help="dir with <name>_llamafactory/train.parquet to take code from")- ap.add_argument("--verified_from", default=None, help="dir with <name>_llamafactory/train.parquet holding compiles/repair")- a = ap.parse_args()- token = os.environ.get("HF_TOKEN", "")- mp = f"{META}/{a.subdir}/metadata.parquet"- from_tree = not os.path.exists(mp)- rows = load_file_tree(a.subdir) if from_tree else pq.read_table(mp).to_pylist()- if a.limit:- rows = rows[: a.limit]- print(f"[pairs] {a.subdir}: {len(rows):,} source samples", flush=True)-- from concurrent.futures import ThreadPoolExecutor- name = a.subdir.replace("/", "__")- img_dir = os.path.join(a.img_root, a.subdir)- os.makedirs(img_dir, exist_ok=True)- qc = collections.Counter()-- verified = {}- if a.verified_from:- # the verification lives in two places depending on the shard: the 68 that could be flattened carry it in- # <sub>_llamafactory_flat, the 44 that could not are in the separately verified tree- base = os.path.basename(a.subdir)- cands = []- for root in a.verified_from.split(","):- cands += [os.path.join(root, f"{base}_llamafactory_flat", "train.parquet"),- os.path.join(root, f"{base}_llamafactory", "train.parquet")]- vp = next((c for c in cands if os.path.exists(c) and _has_verify_cols(c)), None)- if vp:- vdf = pd.read_parquet(vp, columns=["id", "compiles", "repair", "conversations"])- for t in vdf.itertuples():- # one row per caption variant; the code is identical across them, so first wins- verified.setdefault(str(t.id), (bool(t.compiles), t.repair, t.conversations[1]["value"]))- print(f"[pairs] verification joined for {len(verified):,} ids from {vp}", flush=True)- else:- print(f"[pairs] WARNING no parquet with compiles/repair for {base} — pairs carry no compiles column", flush=True)- if a.code_from and not verified:- cp = os.path.join(a.code_from, f"{os.path.basename(a.subdir)}_llamafactory", "train.parquet")- if os.path.exists(cp):- cdf = pd.read_parquet(cp, columns=["id", "conversations"])- for t in cdf.itertuples():- verified.setdefault(str(t.id), (None, None, t.conversations[1]["value"]))- print(f"[pairs] code recovered for {len(verified):,} ids from {cp}", flush=True)- else:- print(f"[pairs] WARNING --code_from given but {cp} is absent", flush=True)- ids = [str(r["id"]) for r in rows]- # a rebuild should not re-download 200 GB of tars just to re-emit rows; if the renders are already extracted- # for every sample, use them- on_disk = {}- for r in rows:- key = str(r["id"]).replace("/", "_")- got = sorted(glob.glob(os.path.join(img_dir, key, "*")))[: a.views]- if got:- on_disk[str(r["id"]).split("/")[-1]] = [(os.path.basename(x), None) for x in got]- # a coverage threshold cannot work here: the samples that have no renders never will, so "98% of rows" is- # unreachable for a source where 12% legitimately lack them. A marker written after a completed scan is the- # only honest signal that the extraction already ran to the end.- if from_tree:- render_index = {str(r["id"]): [(x, None) for x in r["_renders"]] for r in rows}- print(f"[pairs] file-tree source: renders read in place for "- f"{sum(1 for v in render_index.values() if v):,}/{len(rows):,} samples", flush=True)- marker = os.path.join(img_dir, ".scan_complete")- if from_tree:- pass- elif os.path.exists(marker) and on_disk:- print(f"[pairs] renders already extracted ({len(on_disk):,} samples) — skipping the tar scan", flush=True)- render_index = on_disk- else:- render_index = renders_by_scanning(a.subdir, ids, a.views, token, a.local_tars)- os.makedirs(img_dir, exist_ok=True)- open(marker, "w").write(f"{len(render_index)}\n")-- def one(r):- code = r.get("code") or ""- # the recovered/verified code has to be substituted BEFORE the length test: the URDF subdirs carry an- # empty `code` column by design, and checking first rejected all 6,146 of them as "short_code"- v = verified.get(str(r["id"]))- if v and v[2]:- code = FENCE_RE.sub("", v[2].strip()) or code- if len(code.strip()) < 40:- qc["short_code"] += 1- return None- # LLaMA-Factory counts <image> placeholders across EVERY turn, so a literal "<image>" in the answer- # (Shadertoy writes "// <image>" as a multipass section marker) reads as one more image than the row- # carries and aborts the whole run hours in. Seven such rows shipped once; drop them at the source.- if "<image>" in code:- qc["literal_image_token"] += 1- return None- caps = r.get("captions")- caps = dict(caps) if isinstance(caps, dict) else (json.loads(caps) if isinstance(caps, str) else {})- caps_list = captions_of(caps)- renders = render_index.get(str(r["id"]) if from_tree else str(r["id"]).split("/")[-1], [])[: a.views]- paths = []- for fname, blob in renders:- d = os.path.join(img_dir, str(r["id"]).replace("/", "_"))- p = fname if os.path.isabs(fname) else os.path.join(d, fname)- if blob is not None and not os.path.exists(p):- os.makedirs(d, exist_ok=True)- open(p, "wb").write(blob)- if os.path.exists(p):- paths.append(p)- if not paths:- qc["no_renders"] += 1- if not caps_list:- qc["no_caption"] += 1- return {"id": str(r["id"]), "code": code, "captions": caps_list, "images": paths,- "compiles": (v[0] if v else None), "repair": (v[1] if v else None)}-- with ThreadPoolExecutor(a.workers) as ex:- got = [x for x in ex.map(one, rows) if x]- print(f"[pairs] usable samples: {len(got):,} | qc={dict(qc)}", flush=True)- if qc["fetch_failed"]:- print(f"[pairs] WARNING {qc['fetch_failed']:,} samples could not be fetched after 4 attempts — their image "- f"pairs are MISSING, not absent by nature. Re-run to fill them in.", flush=True)-- d, fence = a.dialect, FENCE.get(a.dialect, "python")- out = {"text": [], "img": [], "imgtext": []}- for g in got:- gpt = {"from": "gpt", "value": f"```{fence}\n{g['code'].strip()}\n```"}- base = {"id": g["id"], "system": SYS[d]}- if g["compiles"] is not None:- base["compiles"] = g["compiles"]; base["repair"] = g["repair"]- for ctype, cap in g["captions"]:- out["text"].append(dict(base, caption_type=ctype,- conversations=[{"from": "human", "value": ASK_TEXT[d] + cap}, gpt]))- if g["images"]:- ph = "<image>" * len(g["images"])- # image->code has no caption to vary, so it stays one row per sample by construction- out["img"].append(dict(base, caption_type=None, images=list(g["images"]),- conversations=[{"from": "human", "value": ph + "\n" + ASK_IMG[d]}, gpt]))- for ctype, cap in g["captions"]:- out["imgtext"].append(dict(base, caption_type=ctype, images=list(g["images"]),- conversations=[{"from": "human", "value": ph + "\n" + ASK_BOTH[d] + cap}, gpt]))- info = {}- for kind, rowsk in out.items():- if not rowsk:- print(f"[pairs] {kind}: 0 rows — not written", flush=True)- continue- folder = os.path.join(a.out, f"{os.path.basename(a.subdir)}_llamafactory_{kind}")- os.makedirs(folder, exist_ok=True)- pd.DataFrame(rowsk).to_parquet(os.path.join(folder, "train.parquet"), index=False)- cols = {"messages": "conversations", "system": "system"}- if kind != "text":- cols["images"] = "images"- entry = {"file_name": "train.parquet", "formatting": "sharegpt", "columns": cols}- json.dump({f"3dcv_{name}_{kind}": entry}, open(os.path.join(folder, "dataset_info.json"), "w"), indent=2)- json.dump({"subdir": a.subdir, "pair_type": kind, "rows": len(rowsk),- "source_samples": len(rows), "usable": len(got), **{k: v for k, v in qc.items()}},- open(os.path.join(folder, "qc.json"), "w"), indent=2)- info[kind] = len(rowsk)- print(f"[pairs] {kind:8s} {len(rowsk):>7,} rows -> {folder}", flush=True)- # the invariant LLaMA-Factory enforces on multimodal rows- bad = sum(1 for k in ("img", "imgtext") for r in out[k]- if r["conversations"][0]["value"].count("<image>") != len(r["images"]))- print(f"[pairs] placeholder/image mismatches: {bad} (must be 0)", flush=True)+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"+ sys.path.insert(0, str(scripts))+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")if __name__ == "__main__":diff --git a/finetune/scripts/convert_subdirs.py b/finetune/scripts/convert_subdirs.pyindex c63f1a7a..a6f32161 100644--- a/finetune/scripts/convert_subdirs.py+++ b/finetune/scripts/convert_subdirs.py@@ -1,300 +1,14 @@-"""Convert EVERY 3DCodeVerse subdirectory into its own LLaMA-Factory dataset + a data-quality report.+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""-One output folder per source subdir, named `<subdir>_llamafactory/`, so it can sit next to the source on the-Hub (e.g. `3dcodebench/factories_geo/` → `3dcodebench/factories_geo_llamafactory/`):-- <subdir>_llamafactory/- train.json ShareGPT: [{system, conversations:[human, gpt]}] — one row per caption variant- dataset_info.json drop-in LLaMA-Factory registry entry for this folder- qc.json per-subdir quality counters + token stats- README.md what it is, how many rows, what was dropped and why--Quality checks applied per sample (a sample can fail several):- empty_code / short_code (<40 chars) / bad_chars (control chars or U+FFFD) / no_caption /- duplicate_code (same code hash inside the subdir) / syntax_bad (AST for Python dialects, brace balance for- scad-glsl, tag check for html) / too_long (>8192 tokens, still exported but flagged) / media_placeholder- (literal <image>/<video>/<audio> — breaks the qwen3_5 template unless escaped; we escape it and flag it)--usage: python scripts/convert_subdirs.py [--out DIR] [--workers 16] [--limit_subdirs N]-"""-import argparse-import ast-import collections-import glob-import hashlib-import json-import os-import re+import runpyimport sys-import unicodedata-from concurrent.futures import ProcessPoolExecutor--import pandas as pd--sys.path.insert(0, "/wekafs/ict/hx_624/llm-ft/scripts")-META = "/wekafs/ict/hx_624/data/3dcodeverse_meta"-FILES = "/wekafs/ict/hx_624/data/3dcodeverse_files"-BENCH_DIR = "/wekafs/ict/hx_624/data/3dcodebench/data"--# subdir prefix -> dialect-DIALECT_OF = {- "3dcodebench": "blender", "bioinspired3d": "blender", "blender_distill": "blender",- "deepcad": "cadquery", "articraft": "cadquery", "shadertoy": "glsl",- "thingiverse": "openscad", "threejs_distill": "threejs", "animation2code": "web",-}-SYS = {- "blender": "You are an expert in procedural 3D modeling with Blender Python (bpy). Given a description of an object, write a complete, standalone Blender 5 Python script that builds it from scratch (clear the default scene first). Output ONLY the code in one ```python block.",- "cadquery": "You are an expert CAD programmer. Given a description of a part, write a complete, standalone CadQuery (Python) script that builds it and leaves the final solid in a variable named `result`. Output ONLY the code in one ```python block.",- "openscad": "You are an expert OpenSCAD programmer. Given a request, write a complete, standalone OpenSCAD (.scad) program that produces the described model. Output ONLY the code in one ```openscad block.",- "glsl": "You are an expert shader programmer. Given a description, write a complete Shadertoy-style GLSL ES 3.00 fragment shader defining `void mainImage(out vec4 fragColor, in vec2 fragCoord)`. Output ONLY the code in one ```glsl block.",- "threejs": "You are an expert three.js developer. Given a description, write a complete, standalone HTML page that renders the described 3D scene with three.js. Output ONLY the code in one ```html block.",- "web": "You are an expert creative web developer. Given a description of an animation, write a complete, standalone HTML page (inline CSS/JS, no local assets) that reproduces it. Output ONLY the code in one ```html block.",- "urdf": "You are an expert in articulated 3D assets. Given a description, write a complete, self-contained URDF file for it. Output ONLY the code in one ```xml block.",-}-FENCE = {"blender": "python", "cadquery": "python", "openscad": "openscad", "glsl": "glsl",- "threejs": "html", "web": "html", "urdf": "xml"}-LEAD = {- "blender": "Create the 3D object described below using Python Blender code: ",- "cadquery": "Write a CadQuery (Python) script that builds the following CAD model: ",- "openscad": "Write an OpenSCAD program for the following: ",- "glsl": "Write a GLSL fragment shader for Shadertoy that renders: ",- "threejs": "Write a standalone three.js page that renders: ",- "web": "Write a standalone animated web page that reproduces: ",- "urdf": "Write a URDF description of: ",-}-IMPERATIVE = ("write", "create", "implement", "make", "build", "generate", "design", "produce")-MEDIA = re.compile(r"<(image|video|audio)>")---def bad_char_count(s):- return sum(1 for ch in s if unicodedata.category(ch) == "Cc" and ch not in "\n\t\r") + s.count("�")---def syntax_ok(code, dialect):- if dialect in ("blender", "cadquery"):- try:- ast.parse(code)- return True- except Exception:- return False- if dialect == "openscad":- return code.count("{") == code.count("}") and code.count("(") == code.count(")")- if dialect == "glsl":- return code.count("{") == code.count("}") and "void" in code- if dialect in ("threejs", "web"):- return "<" in code and ">" in code- if dialect == "urdf":- return "<robot" in code or "<link" in code- return True---def phrase(dialect, kind, text):- t = (text or "").strip()- if not t:- return None- if kind == "instruction" and t.lower().startswith(IMPERATIVE):- return t- return LEAD[dialect] + t---def normalize_caps(caps):- """tree sources use their own caption keys"""- caps = {k: v for k, v in caps.items() if isinstance(v, str) and v.strip()}- if any(k in caps for k in ("instruction", "detailed", "factory")) and "user_instruction" not in caps and "brief" not in caps:- return caps- norm = {}- if caps.get("user_instruction"):- norm["instruction"] = caps["user_instruction"]- if caps.get("instruction"):- norm["detailed"] = caps["instruction"]- if caps.get("brief"):- extra = caps.get("function", "")- norm["instruction"] = (caps["brief"] + ((" — " + extra) if extra and extra not in caps["brief"] else "")).strip()- if caps.get("source_title") and "detailed" not in norm:- norm["detailed"] = caps["source_title"]- if caps.get("style") and norm.get("detailed"):- norm["detailed"] += f" (style: {caps['style']})"- return norm or caps---def load_subdir(sub):- """-> (dialect, [ {id, name, code, caps, n_images} ]) for a parquet subdir or a file tree"""- top = sub.split("/")[0]- dialect = DIALECT_OF.get(top, "blender")- if top == "articraft" and "urdf" in sub:- dialect = "urdf"- rows = []- p = f"{META}/{sub}/metadata.parquet"- if os.path.exists(p):- df = pd.read_parquet(p)- for r in df.itertuples():- caps = r.captions if isinstance(r.captions, dict) else (json.loads(r.captions) if isinstance(r.captions, str) else {})- meta = {}- try:- meta = json.loads(r.meta_json) if isinstance(r.meta_json, str) else (r.meta_json or {})- except Exception:- pass- rows.append({"id": str(r.id), "name": str(getattr(r, "name", "") or ""),- "code": getattr(r, "code", None) or "", "caps": dict(caps),- "n_images": len(meta.get("renders", []) or [])})- return dialect, rows- # file tree- for cp in sorted(glob.glob(f"{FILES}/{sub}/*/code.py") + glob.glob(f"{FILES}/{sub}/*/code.html")- + glob.glob(f"{FILES}/{sub}/*/index.html") + glob.glob(f"{FILES}/{sub}/*/*/code.py")- + glob.glob(f"{FILES}/{sub}/*/*/code.html") + glob.glob(f"{FILES}/{sub}/*/*/index.html")):- d = os.path.dirname(cp)- try:- code = open(cp, encoding="utf-8", errors="replace").read()- caps = json.load(open(os.path.join(d, "captions.json"))) if os.path.exists(os.path.join(d, "captions.json")) else {}- except Exception:- continue- rows.append({"id": os.path.relpath(d, FILES), "name": os.path.basename(d), "code": code,- "caps": caps, "n_images": len(glob.glob(os.path.join(d, "renders", "*.png")))})- return dialect, rows---_TOK = None---def _init():- global _TOK- from transformers import AutoTokenizer- _TOK = AutoTokenizer.from_pretrained("/wekafs/ict/hx_624/models/Qwen3.5-9B")---def convert_one(args):- sub, out_root, bench_factories = args- dialect, rows = load_subdir(sub)- if not rows:- return {"subdir": sub, "error": "no rows"}- qc = collections.Counter()- seen_code, out, tok_lens, img_counts = {}, [], [], []- for r in rows:- code = r["code"] if isinstance(r["code"], str) else ""- img_counts.append(r.get("n_images", 0))- if not code.strip():- qc["empty_code"] += 1- continue- if len(code.strip()) < 40:- qc["short_code"] += 1- continue- if bad_char_count(code) > 5:- qc["bad_chars"] += 1- continue- h = hashlib.md5(code.encode()).hexdigest()- if h in seen_code:- qc["duplicate_code"] += 1- continue- seen_code[h] = 1- if not syntax_ok(code, dialect):- qc["syntax_bad"] += 1 # exported anyway, but counted- if MEDIA.search(code) or any(MEDIA.search(v) for v in r["caps"].values() if isinstance(v, str)):- qc["media_placeholder"] += 1- code = MEDIA.sub(lambda m: f"<{m.group(1)} >", code)- norm_name = r["name"].replace("Factory", "").replace("_geo", "").replace("_tex", "")- if r["name"] in bench_factories or r["name"].rsplit("_", 1)[0] in bench_factories or norm_name in bench_factories:- # counted, never dropped: which samples a benchmark forbids is a decision for whoever defines the- # benchmark, and a converter that quietly removes 100% of a subdirectory (as this once did for both- # 3dcodebench/instances_*) is the wrong place to make it- qc["benchmark_overlap"] += 1- caps = normalize_caps(r["caps"])- n_before = len(out)- for kind in ("instruction", "detailed", "factory"):- pr = phrase(dialect, kind, caps.get(kind))- if not pr or len(pr) < 20:- continue- pr = MEDIA.sub(lambda m: f"<{m.group(1)} >", pr)- n = len(_TOK(pr).input_ids) + len(_TOK(code).input_ids) + 60- tok_lens.append(n)- if n > 8192:- qc["too_long_gt8192"] += 1- out.append({"id": r["id"], "caption_type": kind, "n_tokens": n,- "system": SYS[dialect],- "conversations": [{"from": "human", "value": pr},- {"from": "gpt", "value": f"```{FENCE[dialect]}\n{code.strip()}\n```"}]})- if len(out) == n_before:- qc["no_caption"] += 1- name = sub.replace("/", "__")- d = os.path.join(out_root, sub + "_llamafactory")- os.makedirs(d, exist_ok=True)- json.dump(out, open(os.path.join(d, "train.json"), "w"), ensure_ascii=False)- key = f"3dcv_{name}"- json.dump({key: {"file_name": "train.json", "formatting": "sharegpt",- "columns": {"messages": "conversations", "system": "system"}}},- open(os.path.join(d, "dataset_info.json"), "w"), indent=2)- tok_lens.sort()- stat = {"subdir": sub, "dialect": dialect, "source_samples": len(rows), "exported_pairs": len(out),- "unique_samples_exported": len(seen_code),- "images_per_sample_median": (sorted(img_counts)[len(img_counts) // 2] if img_counts else 0),- "tokens_p50": tok_lens[len(tok_lens) // 2] if tok_lens else 0,- "tokens_p90": tok_lens[int(len(tok_lens) * 0.9)] if tok_lens else 0,- "tokens_max": tok_lens[-1] if tok_lens else 0,- **{k: v for k, v in qc.items()}}- json.dump(stat, open(os.path.join(d, "qc.json"), "w"), indent=2)- issues = "\n".join(f"- **{k}**: {v}" for k, v in sorted(qc.items(), key=lambda x: -x[1])) or "- none"- open(os.path.join(d, "README.md"), "w").write(f"""# {sub} — LLaMA-Factory format--Generated by `finetune/scripts/convert_subdirs.py` from `{sub}/metadata.parquet` (or its file tree).--| | |-|---|---|-| dialect | `{dialect}` |-| source samples | {len(rows):,} |-| exported pairs | {len(out):,} (one row per caption variant: instruction / detailed / factory) |-| unique code samples exported | {stat['unique_samples_exported']:,} |-| token length p50 / p90 / max | {stat['tokens_p50']} / {stat['tokens_p90']} / {stat['tokens_max']} |-| renders per sample (median) | {stat['images_per_sample_median']} |--## Data-quality findings--{issues}--`syntax_bad` rows are still exported (a shader can be valid without balanced braces in our crude check); every-other counter means the sample was dropped, EXCEPT `benchmark_overlap`, which is counted and kept: the-sample belongs to one of the 212-3DCodeBench evaluation factories and is excluded to avoid test leakage.--## Use--```yaml-dataset_dir: <this folder>-dataset: {key}-```-""")- return stat+from pathlib import Pathdef main():- ap = argparse.ArgumentParser()- ap.add_argument("--out", default="/wekafs/ict/hx_624/hf_subdirs")- ap.add_argument("--workers", type=int, default=16)- ap.add_argument("--limit_subdirs", type=int, default=None)- ap.add_argument("--only", default=None, help="comma-separated subdirs to convert")- a = ap.parse_args()- bench = frozenset(os.path.basename(p).replace("_seed0", "") for p in glob.glob(f"{BENCH_DIR}/*_seed0"))- subs = ["/".join(p.split("/")[-3:-1]) for p in sorted(glob.glob(f"{META}/*/*/metadata.parquet"))]- subs += ["blender_distill", "threejs_distill", "animation2code"]- if a.only:- want = {x.strip() for x in a.only.split(",") if x.strip()}- subs = [x for x in subs if x in want]- if a.limit_subdirs:- subs = subs[: a.limit_subdirs]- print(f"[conv] {len(subs)} subdirs -> {a.out}", flush=True)- os.makedirs(a.out, exist_ok=True)- stats = []- with ProcessPoolExecutor(a.workers, initializer=_init) as ex:- for st in ex.map(convert_one, [(s, a.out, bench) for s in subs]):- stats.append(st)- print(f"[conv] {st.get('subdir'):45s} pairs={st.get('exported_pairs', 0):7,} "- f"issues={ {k: v for k, v in st.items() if k in ('empty_code','short_code','bad_chars','duplicate_code','syntax_bad','no_caption','too_long_gt8192','media_placeholder','benchmark_overlap')} }", flush=True)- df = pd.DataFrame(stats)- df.to_csv(os.path.join(a.out, "quality_report.csv"), index=False)- try:- df.to_excel(os.path.join(a.out, "quality_report.xlsx"), index=False)- except Exception as e:- print("[conv] xlsx skipped:", e)- json.dump(stats, open(os.path.join(a.out, "quality_report.json"), "w"), indent=2)- print(f"[conv] wrote quality_report.{{csv,xlsx,json}} | total exported pairs: {int(df['exported_pairs'].sum()):,}")+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"+ sys.path.insert(0, str(scripts))+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")if __name__ == "__main__":diff --git a/finetune/scripts/drop_old_pair_trees.py b/finetune/scripts/drop_old_pair_trees.pyindex 3e92b10d..a6f32161 100644--- a/finetune/scripts/drop_old_pair_trees.py+++ b/finetune/scripts/drop_old_pair_trees.py@@ -1,26 +1,15 @@-"""Remove the superseded top-level pairs/ and pair_images/ once their replacements are registered.+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""-Both were mine: the pair datasets were first published into a separate top-level tree, which read as if it were-a source of its own. They now live at <source>/<subdir>_llamafactory_{text,img,imgtext} and-<source>/pair_renders/, and dataset_info.json points only at those, so the old tree is safe to drop.-"""-import os, json-from huggingface_hub import HfApi, hf_hub_download, CommitOperationDelete-tok = os.environ["HF_TOKEN"]; REPO = "ilabai/3dcodeverse"; api = HfApi(token=tok)-agg = json.load(open(hf_hub_download(REPO, "dataset_info.json", repo_type="dataset", token=tok)))-still = [k for k, v in agg.items() if str(v.get("folder", "")).startswith(("pairs/", "pair_images/"))]-if still:- raise SystemExit(f"REFUSING: {len(still)} index entries still point into the old tree")-for d in ("pairs", "pair_images"):- files = [e.path for e in api.list_repo_tree(REPO, repo_type="dataset", path_in_repo=d, recursive=True)- if e.__class__.__name__.startswith("RepoFile")]- print(f"[drop] {d}/: {len(files):,} files", flush=True)- B = 400- for i in range(0, len(files), B):- api.create_commit(repo_id=REPO, repo_type="dataset",- operations=[CommitOperationDelete(path_in_repo=f) for f in files[i:i+B]],- commit_message=f"remove the superseded {d}/ ({i+1}-{min(i+B, len(files))} of {len(files)})")- if (i // B) % 20 == 0:- print(f"[drop] {min(i+B, len(files)):,}/{len(files):,}", flush=True)- print(f"[drop] {d}/ removed", flush=True)-print("[drop] DONE", flush=True)+import runpy+import sys+from pathlib import Path+++def main():+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"+ sys.path.insert(0, str(scripts))+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")+++if __name__ == "__main__":+ main()diff --git a/finetune/scripts/shrink_pair_images.py b/finetune/scripts/shrink_pair_images.pyindex 1f4b0e49..a6f32161 100644--- a/finetune/scripts/shrink_pair_images.py+++ b/finetune/scripts/shrink_pair_images.py@@ -1,57 +1,14 @@-"""Re-encode the extracted renders to training resolution.+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""-The originals average 220 KB (640x360 or 1024x1024 PNG) and total 84 GB, which is impractical to ship beside a-dataset. Every vision tower we train downsamples to 448x448 anyway (our own 27B run set image_max_pixels to-448^2), so a <=448px JPEG loses nothing a training run would have seen — and the untouched originals stay in the-corpus tars for anyone who needs them. Measured saving: 95%, so 84 GB becomes about 4 GB.--usage: python scripts/shrink_pair_images.py [--root DIR] [--out DIR] [--max_px 448] [--quality 88] [--workers 64]-"""-import argparse-import glob-import os-from concurrent.futures import ProcessPoolExecutor--from PIL import Image---def one(args):- src, dst, max_px, q = args- if os.path.exists(dst):- return 0, 0, 0- try:- im = Image.open(src)- im = im.convert("RGB")- im.thumbnail((max_px, max_px), Image.LANCZOS)- os.makedirs(os.path.dirname(dst), exist_ok=True)- im.save(dst, "JPEG", quality=q, optimize=True)- return os.path.getsize(src), os.path.getsize(dst), 1- except Exception:- return 0, 0, -1+import runpy+import sys+from pathlib import Pathdef main():- ap = argparse.ArgumentParser()- ap.add_argument("--root", default="/wekafs/ict/hx_624/data/pair_images")- ap.add_argument("--out", default="/wekafs/ict/hx_624/data/pair_images_small")- ap.add_argument("--max_px", type=int, default=448)- ap.add_argument("--quality", type=int, default=88)- ap.add_argument("--workers", type=int, default=64)- a = ap.parse_args()- srcs = glob.glob(os.path.join(a.root, "**", "*.png"), recursive=True)- print(f"[shrink] {len(srcs):,} renders -> {a.out} at <={a.max_px}px q{a.quality}", flush=True)- jobs = [(s, os.path.join(a.out, os.path.relpath(s, a.root)).rsplit(".", 1)[0] + ".jpg", a.max_px, a.quality)- for s in srcs]- big = small = done = failed = 0- with ProcessPoolExecutor(a.workers) as ex:- for i, (b, s, ok) in enumerate(ex.map(one, jobs, chunksize=64), 1):- big += b; small += s- if ok == 1: done += 1- elif ok < 0: failed += 1- if i % 50000 == 0:- print(f"[shrink] {i:,}/{len(jobs):,} | {big/1e9:.1f} GB -> {small/1e9:.1f} GB | failed {failed}", flush=True)- print(f"[shrink] DONE {done:,} written, {failed} failed | {big/1e9:.1f} GB -> {small/1e9:.1f} GB "- f"({(1-small/max(big,1))*100:.1f}% smaller)", flush=True)+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"+ sys.path.insert(0, str(scripts))+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")if __name__ == "__main__":diff --git a/finetune/scripts/sync_repo.sh b/finetune/scripts/sync_repo.shindex 8f3f5a67..fc52a7eb 100755--- a/finetune/scripts/sync_repo.sh+++ b/finetune/scripts/sync_repo.sh@@ -3,7 +3,9 @@# scripts/sync_repo.sh "commit message" (default message if omitted)## What is synced (code + docs only):-# scripts/ configs/ eval/ env.sh -> finetune/+# shell scripts/ configs/ eval/ env.sh -> finetune/+# Python data tools are maintained in the repo's finetune/llamafactory/;+# their compatibility launchers in scripts/ must not be overwritten by old copies.# REPORT.md -> finetune/docs/REPORT.md# README.md -> finetune/docs/PROJECT_README.md# llm_finetune_exps.md -> finetune/llm_finetune_exps.md@@ -18,7 +20,10 @@ DST=$REPO/finetuneMSG=${1:-"finetune: sync scripts, configs, eval stack and docs"}[ -d "$DST" ] || { echo "repo clone missing at $REPO"; exit 1; }-rsync -a --delete --exclude '__pycache__' --exclude '*.pyc' "$SRC/scripts/" "$DST/scripts/"+# Preserve the repo-owned entrypoints/docs and this script itself, so an older+# external sync script cannot restore the pre-consolidation mirroring rules.+rsync -a --delete --exclude '__pycache__' --exclude '*.pyc' --exclude '*.py' \+ --exclude '/sync_repo.sh' --exclude '/README.md' "$SRC/scripts/" "$DST/scripts/"# --delete on configs too: without it a config retired into configs/lf/archive/ was copied to its new# location but the old copy lingered in the repo forever, so a tidy-up never actually reached GitHub.rsync -a --delete --exclude '__pycache__' "$SRC/configs/" "$DST/configs/"diff --git a/finetune/scripts/upload_pairs.py b/finetune/scripts/upload_pairs.pyindex 9af73627..a6f32161 100644--- a/finetune/scripts/upload_pairs.py+++ b/finetune/scripts/upload_pairs.py@@ -1,96 +1,14 @@-"""Publish the pair datasets INTO their source folders, not into a separate top-level tree.+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""-Every converted dataset in this repo lives at <source>/<subdir>_llamafactory; the pair datasets follow the same-rule as <source>/<subdir>_llamafactory_{text,img,imgtext}, and their renders as <source>/pair_renders/... . A-top-level pairs/ or pair_images/ would read as if it were a source of its own, which is the same mistake the-five stray *_llamafactory folders made.--usage: python scripts/upload_pairs.py [--batch 24] [--images] [--drop_old]-"""-import argparse-import glob-import json-import os--from huggingface_hub import HfApi, CommitOperationAdd, CommitOperationDelete, hf_hub_download--REPO = "ilabai/3dcodeverse"-ROOTS = ("/wekafs/ict/hx_624/hf_pairs_shadertoy", "/wekafs/ict/hx_624/hf_pairs", "/wekafs/ict/hx_624/hf_pairs_other")-# the two sources the user renamed after these were built-RENAME = {"blender_distill": "blender_coding_agent", "threejs_distill": "threejs_coding_agent"}---def collect():- out = []- for root in ROOTS:- for d in sorted(glob.glob(f"{root}/*_llamafactory_*")):- q = json.load(open(f"{d}/qc.json"))- if q["rows"] == 0:- continue- src = RENAME.get(q["subdir"].split("/")[0], q["subdir"].split("/")[0])- out.append((d, f"{src}/{os.path.basename(d)}", q))- return out+import runpy+import sys+from pathlib import Pathdef main():- ap = argparse.ArgumentParser()- ap.add_argument("--batch", type=int, default=24)- ap.add_argument("--images", action="store_true")- ap.add_argument("--drop_old", action="store_true", help="remove the earlier top-level pairs/ and pair_images/")- a = ap.parse_args()- api = HfApi(token=os.environ["HF_TOKEN"])- items = collect()- print(f"[up] {len(items)} pair datasets, {sum(q['rows'] for _, _, q in items):,} rows", flush=True)-- for i in range(0, len(items), a.batch):- chunk = items[i:i + a.batch]- ops = []- for d, path, _ in chunk:- for f in ("train.parquet", "dataset_info.json", "qc.json"):- if os.path.exists(f"{d}/{f}"):- ops.append(CommitOperationAdd(path_in_repo=f"{path}/{f}", path_or_fileobj=f"{d}/{f}"))- api.create_commit(repo_id=REPO, repo_type="dataset", operations=ops,- commit_message=f"pairs: publish {i+1}-{i+len(chunk)} of {len(items)} under their sources")- print(f"[up] pushed {i+1}-{i+len(chunk)}", flush=True)-- p = hf_hub_download(REPO, "dataset_info.json", repo_type="dataset", token=os.environ["HF_TOKEN"])- agg = json.load(open(p))- before = len(agg)- # drop the entries that pointed into the old top-level tree, then register the new locations- agg = {k: v for k, v in agg.items() if not str(v.get("folder", "")).startswith("pairs/")}- for d, path, q in items:- info = json.load(open(f"{d}/dataset_info.json"))- for name, attr in info.items():- agg[name] = {"hf_hub_url": REPO, "folder": path, "formatting": "sharegpt", "split": "train",- "columns": attr.get("columns", {"messages": "conversations", "system": "system"})}- out = "/tmp/claude-1035/-wekafs-ict-hx-624/8a069962-27e5-496d-b000-d4e3dd4dac29/scratchpad/dataset_info_v2.json"- json.dump(agg, open(out, "w"), indent=2, ensure_ascii=False, sort_keys=True)- api.upload_file(path_or_fileobj=out, path_in_repo="dataset_info.json", repo_id=REPO, repo_type="dataset",- commit_message=f"dataset_info: point the pair datasets at their source folders ({before} -> {len(agg)})")- print(f"[up] index {before} -> {len(agg)} entries", flush=True)-- if a.images:- for src in sorted(os.listdir("/wekafs/ict/hx_624/data/pair_images_small")):- d = f"/wekafs/ict/hx_624/data/pair_images_small/{src}"- if not os.path.isdir(d):- continue- dst = f"{RENAME.get(src, src)}/pair_renders"- print(f"[up] renders {src} -> {dst}", flush=True)- api.upload_folder(folder_path=d, path_in_repo=dst, repo_id=REPO, repo_type="dataset",- commit_message=f"{dst}: renders at training resolution")-- if a.drop_old:- for old in ("pairs", "pair_images"):- try:- es = list(api.list_repo_tree(REPO, repo_type="dataset", path_in_repo=old, recursive=True))- files = [e.path for e in es if e.__class__.__name__.startswith("RepoFile")]- for i in range(0, len(files), 200):- api.create_commit(repo_id=REPO, repo_type="dataset",- operations=[CommitOperationDelete(path_in_repo=f) for f in files[i:i+200]],- commit_message=f"remove the superseded top-level {old}/ ({i+1}-{min(i+200,len(files))})")- print(f"[up] removed {len(files)} files from {old}/", flush=True)- except Exception as e:- print(f"[up] {old}/: {type(e).__name__} {str(e)[:80]}", flush=True)+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"+ sys.path.insert(0, str(scripts))+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")if __name__ == "__main__":diff --git a/finetune/scripts/verify_unflattened_glsl.py b/finetune/scripts/verify_unflattened_glsl.pyindex 5bb8dc33..a6f32161 100644--- a/finetune/scripts/verify_unflattened_glsl.py+++ b/finetune/scripts/verify_unflattened_glsl.py@@ -1,107 +1,14 @@-"""Compile-verify the 44 shadertoy shards that the flattening pass never reached.+"""Compatibility CLI; the implementation lives in finetune/llamafactory/."""-Those shards' metadata carries no `shader_json`, so there was no renderpass structure to merge a Common tab-from and `flatten_multifile.py` skipped them — exactly 44 of 112, with zero mismatches. The consequence is that-73,281 shaders are verified (99.04% compile) while 46,680 have never been near a compiler. This runs the same-checker and the same minimal repairs over those, and writes the `compiles` / `repair` columns the flattened-folders already carry, so a training mix can filter the whole corpus by one rule instead of two.--usage: python scripts/verify_unflattened_glsl.py [--workers 40] [--limit_shards N] [--out DIR]-"""-import argparse-import collections-import glob-import json-import os-import re+import runpyimport sys--import pandas as pd--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-from qc_repair import glsl_compile, repair_glsl--SRC = "/wekafs/ict/hx_624/hf_subdirs/shadertoy"-FENCE = re.compile(r"^```[a-zA-Z]*\n|\n?```$")---def strip_fence(v):- return FENCE.sub("", v.strip())---def verify_one(code):- """-> (compiles, repair_label, code_to_keep). Never returns a program that does not compile as 'repaired'."""- ok, _ = glsl_compile(code)- if ok:- return True, "original", code- try:- fixed, recipe = repair_glsl(code)- except Exception:- return False, "unfixable", code- if fixed and fixed != code:- ok2, _ = glsl_compile(fixed)- if ok2:- return True, f"repaired:{recipe or 'other'}", fixed- return False, "unfixable", code---def do_shard(args):- sub, out_root = args- p = f"{SRC}/{sub}_llamafactory/train.parquet"- df = pd.read_parquet(p)- # one shader appears once per caption variant; compile each distinct program once- by_code = {}- codes = [strip_fence(c[1]["value"]) for c in df["conversations"]]- for c in set(codes):- by_code[c] = verify_one(c)- okc, howc, conv = [], [], []- for row, c in zip(df.itertuples(), codes):- ok, how, keep = by_code[c]- okc.append(ok); howc.append(how)- turns = [dict(t) for t in row.conversations]- turns[1] = dict(turns[1], value=f"```glsl\n{keep}\n```")- conv.append(turns)- df["conversations"] = conv- df["compiles"] = okc- df["repair"] = howc- d = os.path.join(out_root, f"{sub}_llamafactory")- os.makedirs(d, exist_ok=True)- df.to_parquet(os.path.join(d, "train.parquet"), index=False)- for extra in ("dataset_info.json", "qc.json", "README.md"):- s = f"{SRC}/{sub}_llamafactory/{extra}"- if os.path.exists(s):- open(os.path.join(d, extra), "w").write(open(s).read())- st = {"shard": sub, "rows": len(df), "unique_programs": len(by_code),- "compiles": int(sum(okc)), "repaired": int(sum(1 for h in howc if h.startswith("repaired"))),- "unfixable": int(sum(1 for h in howc if h == "unfixable"))}- json.dump(st, open(os.path.join(d, "verify.json"), "w"), indent=2)- return st+from pathlib import Pathdef main():- ap = argparse.ArgumentParser()- ap.add_argument("--workers", type=int, default=40)- ap.add_argument("--limit_shards", type=int, default=None)- ap.add_argument("--out", default="/wekafs/ict/hx_624/hf_subdirs_verified/shadertoy")- a = ap.parse_args()- flat = {os.path.basename(p)[: -len("_llamafactory_flat")] for p in glob.glob(f"{SRC}/*_llamafactory_flat")}- allp = {os.path.basename(p)[: -len("_llamafactory")] for p in glob.glob(f"{SRC}/*_llamafactory")}- todo = sorted(allp - flat)- if a.limit_shards:- todo = todo[: a.limit_shards]- os.makedirs(a.out, exist_ok=True)- print(f"[verify] {len(todo)} shards never verified -> {a.out}", flush=True)- from concurrent.futures import ProcessPoolExecutor- tot = collections.Counter()- with ProcessPoolExecutor(a.workers) as ex:- for st in ex.map(do_shard, [(s, a.out) for s in todo]):- for k in ("rows", "unique_programs", "compiles", "repaired", "unfixable"):- tot[k] += st[k]- print(f"[verify] {st['shard']:4s} rows={st['rows']:>6,} uniq={st['unique_programs']:>5,} "- f"ok={st['compiles']:>6,} repaired={st['repaired']:>4,} unfixable={st['unfixable']:>4,}", flush=True)- rate = tot["compiles"] / max(tot["rows"], 1) * 100- print(f"[verify] TOTAL rows={tot['rows']:,} unique={tot['unique_programs']:,} "- f"compiles={tot['compiles']:,} ({rate:.2f}%) repaired={tot['repaired']:,} unfixable={tot['unfixable']:,}", flush=True)+ scripts = Path(__file__).resolve().parents[1] / "llamafactory"+ sys.path.insert(0, str(scripts))+ runpy.run_path(str(scripts / Path(__file__).name), run_name="__main__")if __name__ == "__main__":diff --git a/finetune/tests/test_llamafactory_entrypoints.py b/finetune/tests/test_llamafactory_entrypoints.pynew file mode 100644index 00000000..e29acfc8--- /dev/null+++ b/finetune/tests/test_llamafactory_entrypoints.py@@ -0,0 +1,183 @@+"""Offline regression checks for the consolidated dataset scripts and legacy CLIs.++Parquet I/O uses JSON-backed doubles; the actual CLI, audit, sample filtering,+and forwarding code runs without data/model dependencies or network access.+"""+import json+import os+import runpy+import shutil+import subprocess+import sys+import types+from pathlib import Path++import pytest++FINETUNE = Path(__file__).resolve().parents[1]+ENTRY_DIRS = ("llamafactory", "scripts")+LEGACY_SCRIPTS = (+ "audit_pairs.py",+ "build_pairs_3dcodebench.py",+ "convert_subdirs.py",+ "drop_old_pair_trees.py",+ "shrink_pair_images.py",+ "upload_pairs.py",+ "verify_unflattened_glsl.py",+)+++class JsonFrame:+ def __init__(self, rows):+ self.rows = rows++ def __len__(self):+ return len(self.rows)++ def to_dict(self, orient):+ assert orient == "records"+ return self.rows++ def to_parquet(self, path, *, index):+ assert index is False+ Path(path).write_text(json.dumps(self.rows))+++def offline_io(monkeypatch):+ pandas = types.ModuleType("pandas")+ pandas.DataFrame = JsonFrame+ pandas.read_parquet = lambda path: JsonFrame(json.loads(Path(path).read_text()))+ monkeypatch.setitem(sys.modules, "pandas", pandas)+ parquet = types.ModuleType("pyarrow.parquet")+ arrow = types.ModuleType("pyarrow")+ arrow.parquet = parquet+ monkeypatch.setitem(sys.modules, "pyarrow", arrow)+ monkeypatch.setitem(sys.modules, "pyarrow.parquet", parquet)++ def no_network(*args, **kwargs):+ pytest.fail("dataset CLI attempted a network request")++ requests = types.ModuleType("requests")+ requests.get = no_network+ hub = types.ModuleType("huggingface_hub")+ hub.HfApi = hub.hf_hub_download = no_network+ monkeypatch.setitem(sys.modules, "requests", requests)+ monkeypatch.setitem(sys.modules, "huggingface_hub", hub)+ # Legacy entry points add their canonical sibling directory to sys.path.+ monkeypatch.setattr(sys, "path", list(sys.path))+ return parquet+++def run_cli(monkeypatch, entry_dir, filename, *args):+ script = FINETUNE / entry_dir / filename+ monkeypatch.setattr(sys, "argv", [str(script), *map(str, args)])+ try:+ runpy.run_path(str(script), run_name="__main__")+ except SystemExit as exc:+ return exc.code+ return 0++[email protected]("entry_dir", ENTRY_DIRS)[email protected]("kind", ("text", "img", "imgtext"))[email protected]("literal_in_answer", (False, True))+def test_audit_counts_image_tokens_in_assistant_turn(+ tmp_path, monkeypatch, capsys, offline_io, entry_dir, kind, literal_in_answer+):+ root = tmp_path / "datasets"+ dataset = root / f"fixture_llamafactory_{kind}"+ dataset.mkdir(parents=True)+ media = tmp_path / "media"+ media.mkdir()+ (media / "view.jpg").write_bytes(b"local image fixture")+ row = {+ "system": "Write a shader.",+ "conversations": [+ {"from": "human", "value": ("<image>" if kind != "text" else "") + "Draw a sphere."},+ {"from": "gpt", "value": "```glsl\n// " + ("<image>" if literal_in_answer else "sphere") + "\n```"},+ ],+ }+ if kind != "text":+ row["images"] = ["view.jpg"]+ (dataset / "train.parquet").write_text(json.dumps([row]))+ (dataset / "qc.json").write_text(json.dumps({"rows": 1}))++ status = run_cli(monkeypatch, entry_dir, "audit_pairs.py", "--roots", root, "--media_root", media)++ assert status == int(literal_in_answer)+ output = capsys.readouterr().out+ expected = "placeholder_in_text_pair" if kind == "text" else "placeholder_mismatch"+ assert (expected in output) == literal_in_answer+ if not literal_in_answer:+ assert "problems: NONE" in output++[email protected]("entry_dir", ENTRY_DIRS)+def test_builder_drops_literal_image_token_without_dropping_valid_sample(+ tmp_path, monkeypatch, offline_io, entry_dir+):+ code = "void mainImage(out vec4 color, in vec2 xy) { color = vec4(1.0); }"+ rows = [+ {"id": "valid", "code": code, "captions": {"instruction": "A bright sphere against a dark background."}},+ {"id": "literal", "code": code + "\n// <image>", "captions": {"instruction": "A bright sphere against a dark background."}},+ ]+ offline_io.read_table = lambda path: types.SimpleNamespace(to_pylist=lambda: rows)+ exists = os.path.exists+ monkeypatch.setattr(+ os.path,+ "exists",+ lambda path: str(path).endswith("/review_fixture/metadata.parquet") or exists(path),+ )+ images = tmp_path / "images"+ source_images = images / "review_fixture"+ for row in rows:+ sample_dir = source_images / row["id"]+ sample_dir.mkdir(parents=True)+ (sample_dir / "view.png").write_bytes(b"cached local render")+ (source_images / ".scan_complete").write_text("2\n")+ output = tmp_path / "output"++ status = run_cli(+ monkeypatch, entry_dir, "build_pairs_3dcodebench.py",+ "--subdir", "review_fixture", "--dialect", "glsl",+ "--out", output, "--img_root", images, "--workers", "1",+ )++ assert status == 0+ for kind in ("text", "img", "imgtext"):+ dataset = output / f"review_fixture_llamafactory_{kind}"+ built = json.loads((dataset / "train.parquet").read_text())+ assert [row["id"] for row in built] == ["valid"]+ assert built[0]["conversations"][1]["value"] == f"```glsl\n{code}\n```"+ assert sum(t["value"].count("<image>") for t in built[0]["conversations"]) == len(built[0].get("images", []))+ qc = json.loads((dataset / "qc.json").read_text())+ assert qc["literal_image_token"] == 1+ assert qc["source_samples"] == 2+ assert qc["rows"] == qc["usable"] == 1++[email protected]("filename", LEGACY_SCRIPTS)+def test_legacy_cli_forwards_arguments_exit_status_and_sibling_imports(tmp_path, filename):+ component = tmp_path / "checkout with spaces" / "finetune"+ legacy = component / "scripts"+ canonical = component / "llamafactory"+ legacy.mkdir(parents=True)+ canonical.mkdir()+ shutil.copyfile(FINETUNE / "scripts" / filename, legacy / filename)+ (canonical / "forwarder_sibling.py").write_text("VALUE = 'canonical sibling'\n")+ (canonical / filename).write_text(+ "import json, sys\n"+ "from forwarder_sibling import VALUE\n"+ "print(json.dumps({'args': sys.argv[1:], 'sibling': VALUE, 'name': __name__}))\n"+ "raise SystemExit(7)\n"+ )+ args = ["--out", "folder with spaces", "--workers", "3"]++ result = subprocess.run(+ [sys.executable, str(legacy / filename), *args],+ cwd=tmp_path, capture_output=True, text=True, timeout=10, check=False,+ )++ assert result.returncode == 7, result.stderr+ assert json.loads(result.stdout) == {"args": args, "sibling": "canonical sibling", "name": "__main__"}diff --git a/harness/codeverse3d/addons/dataset/refine.py b/harness/codeverse3d/addons/dataset/refine.pyindex c2ef87f9..b085de45 100644--- a/harness/codeverse3d/addons/dataset/refine.py+++ b/harness/codeverse3d/addons/dataset/refine.py@@ -13,7 +13,7 @@ commit. ``instructions`` is ``RoundRecord.instructions``, the task list``build_refine_instructions`` compiled — not the per-group prompt text the sessions werehanded, which is a rendering of it.-The rows are data, not a training format: ``toolkits/llamafactory/build_refine_sft.py``+The rows are data, not a training format: ``finetune/llamafactory/build_refine_sft.py``turns them into the message shape the finetune pipeline consumes. ``--with-code`` inlinesthe changed files so that conversion needs no access to the runs."""diff --git a/harness/docs/ARCHITECTURE.md b/harness/docs/ARCHITECTURE.mdindex eca03d65..5b89c639 100644--- a/harness/docs/ARCHITECTURE.md+++ b/harness/docs/ARCHITECTURE.md@@ -670,7 +670,7 @@ instructions written in response, both code snapshots and whether the score move(the row is `addons/dataset/refine.RefineTransition`; INTERFACES has the fields and thedrop reasons). That is the supervision the harness produces that aone-shot corpus cannot: what a failing artefact looked like, what was wrong with it-in the harness's own words, and what the fix changed. `toolkits/llamafactory/`+in the harness's own words, and what the fix changed. `finetune/llamafactory/`turns those rows into training files; the harness writes the measurement, not thetrainer's format. `flywheel caption` adds{detailed, instruction, factory} captions (image-grounded, brand-free, `--out` fordiff --git a/harness/docs/DECISIONS.md b/harness/docs/DECISIONS.mdindex de746361..7830f071 100644--- a/harness/docs/DECISIONS.md+++ b/harness/docs/DECISIONS.md@@ -478,7 +478,7 @@ Pointers: EVAL, PAPER_WRITING = `eval/docs/*.md`; COST, RUNBOOK, ARCHITECTURE, I`cat-file --batch`, never `git archive` (a planted `filter.<name>.smudge` executes and no flagturns it off); pinned by a test in `tests/flywheel_cli` (`record/_git.py`).* **The refine transitions are the corpus this harness is for** (`3dcode flywheel refine`: one- row per refined round); format conversion lives in `toolkits/llamafactory/`, not in the harness+ row per refined round); format conversion lives in `finetune/llamafactory/`, not in the harness(owner's boundary, 2026-09-03).* **`<mimic>` changes the sweep, which is what it was for**: on the 8 of 14 prompts that declarea coupling the sampler drives a median 3 joints per pose instead of 6; the `joint_sweep`diff --git a/harness/docs/RUNBOOK.md b/harness/docs/RUNBOOK.mdindex 1dba7000..365947f3 100644--- a/harness/docs/RUNBOOK.md+++ b/harness/docs/RUNBOOK.md@@ -196,7 +196,7 @@ unless you type `--host` yourself; it never serves a path outside the declared r# one row per round the harness asked to change; the row# schema is codeverse3d/addons/dataset/refine.RefineTransition and# INTERFACES has the call signatures. Training formats live- # in toolkits/llamafactory/, not here.+ # in finetune/llamafactory/, not here.3dcode flywheel caption <slug> [--model …] [--out caps/] # --out = side-car mode, run untouched3dcode flywheel index runs/ runs_index.sqlitepython -m codeverse3d.addons.calibration runs/<slug> [runs/<slug2> …] --model gemini:gemini-3.1-pro-preview --n 3 --out out/diff --git a/toolkits/README.md b/toolkits/README.mdindex 410b8d3b..8ed536b9 100644--- a/toolkits/README.md+++ b/toolkits/README.md@@ -1,32 +1,29 @@-# toolkits/ — the tools that turn raw 3D projects into trainable data+# toolkits/ — general data CLI and corpus curation-Everything here operates on **data**, not on models. The pipeline they cover, end to end:--```-contributor's project ──▶ curation ──▶ 3DCodeVerse on the Hub ──▶ LLaMA-Factory datasets ──▶ finetune/- code + renders validate, per-source subdirs per-subdir ShareGPT, training runs- + meta.json dedupe, (metadata.parquet + tars) verified & flagged and evaluation- render, pack-```+These tools turn contributed or source projects into the 3DCodeVerse corpus: validate, execute,+render, deduplicate, attach metadata, and package the results for publication.| folder | stage | what it does ||---|---|---|| [`3dcode_cli/`](3dcode_cli/) | contributor → corpus | the installable `3dcode-data` CLI: validate, fingerprint/dedupe, execute, render and push a project to the staging bucket. Per-dialect modules for Blender-Python, CadQuery, build123d, FreeCAD and OpenSCAD || [`curation/`](curation/) | raw → corpus | per-source curation: convert a source's native format to runnable projects, execute/render them, compute ground-truth meshes, dedupe, pack to tar+parquet, upload to the Hub |-| [`llamafactory/`](llamafactory/) | corpus → training data | turn every corpus subdirectory into a LLaMA-Factory dataset, verify it with the real compilers/runtimes, flatten multi-file samples, build multimodal (render + instruction) variants, and sample training mixes |-`3dcode_cli/` is a complete copy of **[gaoypeng/3dcode_toolkit](https://github.com/gaoypeng/3dcode_toolkit)**-(74 files, byte-for-byte apart from `.git`), kept here so the whole data toolchain lives in one place. It is still-`pipx install`-able from either location:+LLaMA-Factory conversion, training-data QC, multimodal pairs, and training mixes live in+[`../finetune/llamafactory/`](../finetune/llamafactory/). Its README contains the training-data script+catalog and usage examples. Training configs, environments, and launchers remain under `finetune/`.++## 3dcode_cli/++`3dcode_cli/` is a repository copy of+**[gaoypeng/3dcode_toolkit](https://github.com/gaoypeng/3dcode_toolkit)**. It can be installed from+either location:```bash-pipx install "git+https://github.com/gaoypeng/3dcode_toolkit" # upstream-pipx install ./toolkits/3dcode_cli # this copy+pipx install "git+https://github.com/gaoypeng/3dcode_toolkit" # upstream+pipx install ./toolkits/3dcode_cli # from this repository's root```-If both are edited they will drift — pick one as canonical and mirror the other.-----+If both copies are edited they can drift; coordinate changes with the upstream repository.## curation/ — per source@@ -35,68 +32,7 @@ If both are edited they will drift — pick one as canonical and mirror the othe| `deepcad/` | `convert_vec_to_cq.py`, `batch_convert.py`, `cq_check{,_batch}.py`, `gen_gt_stl.py`, `iou_compute.py`, `render_all.py`, `dedup_check.py`, `attach_text.py`, `pack_tar_parquet.py`, `finalize.py`, `upload_to_hf.py` | DeepCAD vector sequences → CadQuery programs, executed and checked, GT meshes + IoU, captions attached, packed and uploaded || `articraft/` | `articraft_cq.py`, `assemble_shim.py`, … | articulated assets → CadQuery / URDF projects |-The shape of a curation pass, which any new source should follow: **convert → execute → render → compute a-ground truth → dedupe → attach text → pack → upload**. Every step writes files, so a failed step can be rerun-without redoing the pipeline.--## llamafactory/ — corpus → trainable datasets--| script | what it does |-|---|---|-| `convert_subdirs.py` | one `<subdir>_llamafactory/` per corpus subdirectory: `train.parquet` (ShareGPT) + `dataset_info.json` + `qc.json` + README. Runs the per-sample quality checks (empty/short/mojibake code, in-subdir duplicates, missing captions, **benchmark overlap**, syntax, over-length, literal media placeholders) |-| `flatten_multifile.py` | makes multi-file samples single-file: merges a Shadertoy shader's **Common tab** into each pass exactly as Shadertoy does, and recovers `code.urdf` for the articraft URDF subdirs (whose `code` column is empty in the metadata) |-| `qc_repair.py` | validates with the **real** compiler (glslangValidator for GLSL, `ast.parse` for Python dialects), attempts minimal semantics-preserving repairs, re-validates each one, and reports exact-duplicate groups within and across subdirs |-| `glsl_render_batch.py`, `glsl_render.py`, `render_verify_glsl.py` | compiling is not drawing: render every shader in a real headless WebGL2 context, sample two frames, and record whether the image is non-uniform and whether it animates |-| `finalize_glsl.py` | consolidates: prefer the author's untouched program, fall back to a repair only if it compiles, otherwise mark `compiles=False`. Writes `compiles` / `repair` columns |-| `build_mm_dataset.py` | the multimodal datasets — `mm_img` (renders only), `mm_img_text` (renders + instruction), `mm_text` (control), from the same samples with the same split so they can be compared |-| `build_image_dataset.py` | earlier single-variant image builder (kept for reproducing the first pilot) |-| `build_md_max.py`, `sample_mix.py` | build the full corpus (every source, every caption variant) and sample training mixes from it with a per-dialect token budget and a captions-per-sample cap |-| `to_llamafactory.py` | jsonl → ShareGPT JSON + a `dataset_info.json` entry |-| `build_hf_dataset.py` | package corpus + preference sets + held-out tests as a Hub dataset |-| `cache_ntok.py`, `ntok_cache.py` | cache per-sample token counts so re-mixing is seconds, not half an hour |-| `extract.py` | dialect-aware code extraction from free-form model output (used by evaluation, included here because dataset QC uses the same rules) |-| `analyze_3dcodeverse.py`, `fetch_hf_files.py` | corpus inventory; selective file download from the Hub |--### Two rules these tools encode--1. **Suspect the checker before the data.** Of 4,192 shaders that "failed", only 588 actually needed a code- change; the rest exposed defects in the validation harness (guessing sampler types from syntax, a missing- `iFrameRate` uniform, a regex that deleted real code from minified lines, the wrong GLSL ES profile).- Every checker in here has a known-good control that must pass.-2. **Mark, don't silently drop.** Rows that cannot be used for a particular purpose — `benchmark_factory`- (3DCodeBench contamination), `compiles=False`, over-length — stay in the dataset with a column that lets a- training mix filter them out. The one exception is exact duplicates, which are removed and listed in- `duplicates_<source>.json` so the upstream corpus can be cleaned too.--### Typical use--```bash-# 1. corpus -> one dataset per subdirectory, with QC-python convert_subdirs.py --out /path/to/out --workers 16--# 2. make multi-file samples single-file, recover code that lives in tars-python flatten_multifile.py --shadertoy --out /path/to/out-python flatten_multifile.py --urdf_tars /path/to/tars --out /path/to/out--# 3. verify with the real toolchain, repair minimally, consolidate-python qc_repair.py --pass all --root /path/to/out-python render_verify_glsl.py --workers 10 --write-python finalize_glsl.py--# 4. build a training mix (per-dialect token budget, one caption per sample)-python sample_mix.py --out mix/ --budget "blender=all,cadquery=25M,glsl=25M" --captions_per_sample 1--# 5. multimodal variants (renders + instruction)-python build_mm_dataset.py --views 4 --out mm/-```--Results produced with these tools: [`finetune/llm_finetune_exps.md`](../finetune/llm_finetune_exps.md) and-[`finetune/docs/REPORT.md`](../finetune/docs/REPORT.md).-------## Layout note--Nothing outside this folder hardcodes the name `toolkits/`: the scripts take their input and output paths as-arguments, and the only references are the links in this file, `../README.md` and `../finetune/README.md`.-Renaming the folder is therefore a `git mv` plus those three links.+The shape of a curation pass is **convert → execute → render → compute ground truth → dedupe →+attach text → pack → upload**. Scripts retain intermediate files; inspect their completion checks+before rerunning a partially completed pass. Some scripts contain paths from the original workstation;+adapt their input, output, and tool paths before running them elsewhere.diff --git a/toolkits/llamafactory/README.md b/toolkits/llamafactory/README.mddeleted file mode 100644index 9710236f..00000000--- a/toolkits/llamafactory/README.md+++ /dev/null@@ -1,71 +0,0 @@-# llamafactory/ — 3DCodeVerse → LLaMA-Factory datasets--Scripts that turn the corpus on the Hub into datasets a training run can consume directly, and that verify the-data with the same runtimes the evaluation uses. See [`../README.md`](../README.md) for the table of every script-and the two rules they encode; the sections below are the details you need to run them.--## Output layout--```-<subdir>_llamafactory/ one per corpus subdirectory, next to the source- train.parquet ShareGPT: system + conversations[human,gpt], plus id / caption_type / n_tokens- dataset_info.json drop-in registry entry -> dataset: 3dcv_<subdir>- qc.json per-subdir counters (drops, syntax, token stats)- README.md what it is and what was dropped, with reasons-<subdir>_llamafactory_flat/ flattened multi-file variants (Shadertoy passes with Common merged)- train.parquet + compiles (bool) and repair (original / repaired:<cat> / unfixable)-```--Multimodal datasets (`build_mm_dataset.py`) add an `images` column; the number of `<image>` placeholders in the-user turn **must** equal `len(images)` — LLaMA-Factory's `mm_plugin._validate_input` raises otherwise. Multimodal-rows also cannot be packed: set `packing: false` and `neat_packing: false`.--## Dialects covered--| dialect | runtime used for verification | notes |-|---|---|---|-| Blender-Python | Blender 5.0.1 headless (needs `scipy` in its bundled Python) | 15 of the 212 3DCodeBench reference scripts import scipy |-| CadQuery | CadQuery 2.8 | the runner captures `show_object`/exports and prefers `result`/`solid` variables |-| OpenSCAD | OpenSCAD AppImage (`--export-format binstl`) | decode with sampling: greedy collapses into the Customizer parameter block |-| GLSL | glslangValidator **and** headless WebGL2 | try ES 3.00 and 3.10 × the channel-type assignments a shader actually uses |-| three.js / web | headless Chromium (Playwright) | judge blankness from the composited screenshot, not `gl.readPixels` |-| URDF | XML parse + link/joint counts | code recovered from the source tars |--## Requirements--Python 3.12 with `pandas`, `pyarrow`, `transformers` (tokenizer only), `playwright` (+ `playwright install chromium`),-and `huggingface_hub` for the upload paths; `glslangValidator` on PATH (conda-forge `glslang`); Blender / OpenSCAD-as system binaries. Paths to the binaries are constants at the top of each script — change them there.-| `build_pairs_3dcodebench.py` | the three pair types for one source subdirectory — text→code, image→code, image+text→code — with one caption per sample rather than one row per caption variant. Renders are read by **scanning each tar**, because the byte offsets in `metadata.parquet` cannot be trusted (for `3dcodebench/instances_geo` only 577 of 1,953 point at a real tar header; the archives were repacked after the metadata was written, and following them silently cost 70% of the image pairs) |-| `verify_unflattened_glsl.py` | compile-verifies the 44 Shadertoy shards `flatten_multifile.py` never reached — their metadata carries no `shader_json`, so there was no renderpass structure to merge a Common tab from. Writes the same `compiles` / `repair` columns the flattened shards have, so one filter covers the whole GLSL corpus |-| `runners/` | the per-source drivers that call the above over every subdirectory |--### A third rule these tools now encode--3. **A conversion must never decide what a benchmark forbids.** `convert_subdirs.py` used to drop any sample- overlapping the 212 3DCodeBench factories, which silently emptied both `3dcodebench/instances_*` folders —- 100% of their rows were overlaps. The overlap is now *counted* in `qc.json` and the rows are kept, so the- split is a decision the training run makes, not one the converter already made.-| `shrink_pair_images.py` | re-encodes the extracted renders to training resolution (≤448px JPEG). Measured 90.8% smaller — 113.7 GB becomes 10.5 GB — and loses nothing a training run would have seen, because every vision tower downsamples to 448² anyway; the originals stay in the corpus tars |-| `upload_pairs.py` | publishes the pair datasets INTO their source folders (`<source>/<subdir>_llamafactory_{text,img,imgtext}`) and their renders as `<source>/pair_renders/`, then registers everything in the root `dataset_info.json` |-| `drop_old_pair_trees.py` | removes a superseded tree only after verifying the index no longer points into it — it refuses to run otherwise |--### A fourth rule, learned the expensive way--4. **Same name does not mean same content — compare before deleting.** Twice a folder appeared both at the- repository root and inside its source, and both times the root copy was the NEWER one (1,008 vs 949 samples,- and earlier 949 vs 318). Deleting the apparent duplicate would have silently discarded 1,239 samples. The- safe order is always: compare, promote the newer copy, then delete — in a single commit, so the repository is- never seen with the data in two places or in neither.-| `audit_pairs.py` | checks every pair dataset for the invariants that fail only once training has started: absolute paths, wrong extension, image files that do not exist, `<image>` count vs `len(images)`, turn shape, fenced code, empty system, row count vs `qc.json`. Exits non-zero on any finding, so it can gate a publish |--### A fifth rule--5. **Sampling proves a problem exists; it cannot prove one does not.** Checking four datasets said "every path is- relative". Scanning all 411 found **36,964 absolute paths** in three of them — introduced by a later rebuild- that reused already-extracted renders and took a code path which skipped the relativisation. For a global- invariant, scan globally: `audit_pairs.py` exists for exactly this.-- The same audit script then reported all 1,948,765 rows as malformed, which was a defect in the checker- (parquet returns numpy arrays, and `isinstance(x, list)` is False for them) — rule 1 applies to your own- checkers too.