Official Codebase of 3DCodeVerse
Python
finetune: publish the training data as a HuggingFace dataset in LLaMA-Factory format (scripts/build_hf_dataset.py -> ilabai/3dcodeverse-llamafactory: per-dialect SFT corpus, bootstrapped Blender set, 7 preference-pair sets, held-out tests, dataset_info.json, mixes.json); add the GPU supervisor + experiment-log generator
gaoypeng31a8ed4272e6
diff --git a/finetune/README.md b/finetune/README.mdindex f27093e8..15a2a562 100644--- a/finetune/README.md+++ b/finetune/README.md@@ -21,6 +21,10 @@ API models; this folder trains and evaluates open models on that data.| `env.sh` | shell env (caches, secrets file) used by all scripts || `docs/REPORT.md` | full experiment report (setup, data audit, ablations, DPO, multi-dialect, scale-up, training-amount curves, qualitative checks); `docs/PROJECT_README.md` is the original working README; `docs/assets/` renders |+## Data on the Hub++The training data in LLaMA-Factory (ShareGPT) format is published as **[`ilabai/3dcodeverse-llamafactory`](https://huggingface.co/datasets/ilabai/3dcodeverse-llamafactory)** (private; same CC-BY-NC-SA / non-commercial terms as the parent `ilabai/3dcodeverse`): the corpus sharded by dialect (`sft/<dialect>/`), the 15.7k execution-verified bootstrapped Blender samples, seven preference-pair sets (execution feedback and geometry feedback), the held-out test sets, a drop-in `dataset_info.json`, and `mixes.json` with the exact budget spec of every training mix in the report. Rebuild/refresh it with `scripts/build_hf_dataset.py`; regenerate any mix with `scripts/sample_mix.py`.+## Results* **[`llm_finetune_exps.md`](llm_finetune_exps.md)** — the full experiment log: every training run with its config and wall time, every 3DCodeBench and per-dialect evaluation, and a summary of which levers moved the numbers (auto-generated tables from `scripts/build_exp_log.py` + curated commentary).diff --git a/finetune/scripts/build_hf_dataset.py b/finetune/scripts/build_hf_dataset.pynew file mode 100644index 00000000..f5a08d6f--- /dev/null+++ b/finetune/scripts/build_hf_dataset.py@@ -0,0 +1,231 @@+"""Package the finetuning data as a HuggingFace dataset in LLaMA-Factory (ShareGPT) format.++Design: publish the *corpus* + the *recipes* + the *derived* sets, not the 15 near-duplicate training mixes+(they are all samples of the same corpus and would multiply the size for nothing — `scripts/sample_mix.py`+regenerates any of them from a one-line budget spec).++ sft/<dialect>/train-*.parquet the md_max corpus (768k prompt→code pairs), sharded by dialect+ sft/blender_bootstrapped/ 15.7k execution-verified Blender samples generated by the flywheel+ dpo/<name>.parquet execution-feedback and geometry-feedback preference pairs+ test/<dialect>.parquet the held-out evaluation sets (never trained on)+ dataset_info.json drop-in LLaMA-Factory registry for everything above+ mixes.json the exact budget spec of every mix used in the report+ README.md dataset card++usage: python scripts/build_hf_dataset.py [--out /wekafs/ict/hx_624/hf_ds] [--push ilabai/3dcodeverse-llamafactory] [--private]+"""+import argparse+import collections+import glob+import json+import os++import pandas as pd++ROOT = "/wekafs/ict/hx_624/llm-ft"+SHARD_ROWS = 50000++MIXES = {+ "md_xl": {"desc": "9B best SFT mix (3DCodeBench 90.1%)", "budget": "blender=all,openscad=all(x4),threejs=all,cadquery=all,glsl=all", "captions_per_sample": 1,+ "extra": "+15.7k bootstrapped blender", "pairs": 270173, "tokens": "293M"},+ "md_big": {"desc": "half-size predecessor of md_xl (84.9%)", "budget": "cadquery=40k rows,glsl=40k rows,openscad=all(x4),blender=all", "captions_per_sample": 1,+ "extra": "+15.7k bootstrapped blender", "pairs": 101851, "tokens": "107M"},+ "md_27b_v2s": {"desc": "27B best mix (3DCodeBench 93.4%) — blender-heavy", "budget": "blender=all(x2)+bootstrapped, cadquery=25M tok, glsl=25M tok, openscad=all(x4), threejs=all, web=all",+ "captions_per_sample": 2, "extra": "filtered to <=3400 tokens", "pairs": 103770, "tokens": "75M"},+ "md_27b_v3": {"desc": "27B rebalanced (cadquery restored)", "budget": "cadquery=45M tok, glsl=20M tok, blender=all+bootstrapped, openscad/threejs/web=all",+ "captions_per_sample": 1, "extra": "filtered to <=3400 tokens", "pairs": 107366, "tokens": "73M"},+ "md_max_9b": {"desc": "caption-augmentation ablation — NEGATIVE result (3DCodeBench 34.4%)", "budget": "cadquery=140M tok, glsl=140M tok, rest=all",+ "captions_per_sample": 3, "extra": "no bootstrapped data", "pairs": 363708, "tokens": "304M"},+ "md_v3": {"desc": "control for the above (86.8%)", "budget": "cadquery=140M tok, glsl=140M tok, rest=all", "captions_per_sample": 1,+ "extra": "+15.7k bootstrapped blender", "pairs": 272802, "tokens": "236M"},+}++DPO_SETS = {+ "exec_feedback_blender_r1": ("data/lf/dpo_exec_pairs.json", "2,590 (runs, fails) pairs from LoRA-v1 samples executed in Blender → 75%→93.9% on 3DCodeBench"),+ "exec_feedback_blender_r2": ("data/lf/dpo_exec_pairs_r2.json", "second round on the round-1 model → 96.2%"),+ "exec_feedback_multidialect_md_mixed": ("data/lf/dpo_md_pairs.json", "581 pairs across 4 dialects (md_mixed policy)"),+ "exec_feedback_multidialect_md_big": ("data/lf/dpo_md_big_pairs.json", "1,052 pairs (md_big policy)"),+ "exec_feedback_multidialect_md_xl": ("data/lf/dpo_md_xl_pairs.json", "909 pairs (md_xl policy)"),+ "exec_feedback_27b": ("data/lf/dpo_27b_pairs.json", "756 pairs from the tuned Qwen3.8-27B (three.js 92.5%→100%)"),+ "geometry_feedback": ("data/lf/dpo_geo_pairs.json", "2,357 pairs ranked by [email protected] against the reference mesh — the only signal that moved geometric fidelity"),+}+++def write_parquet(rows, path_prefix, cols=None):+ os.makedirs(os.path.dirname(path_prefix), exist_ok=True)+ n = 0+ for i in range(0, len(rows), SHARD_ROWS):+ chunk = rows[i: i + SHARD_ROWS]+ df = pd.DataFrame(chunk, columns=cols) if cols else pd.DataFrame(chunk)+ suffix = f"-{i // SHARD_ROWS:05d}" if len(rows) > SHARD_ROWS else ""+ df.to_parquet(f"{path_prefix}{suffix}.parquet", index=False, compression="zstd")+ n += 1+ return n+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--out", default="/wekafs/ict/hx_624/hf_ds")+ ap.add_argument("--push", default=None, help="repo id, e.g. ilabai/3dcodeverse-llamafactory")+ ap.add_argument("--private", action="store_true")+ a = ap.parse_args()+ os.makedirs(a.out, exist_ok=True)+ info, stats = {}, {}++ # ---- 1. the corpus, sharded by dialect (ShareGPT: conversations + system) ----+ by = collections.defaultdict(list)+ for split in ("train", "val"):+ for l in open(f"{ROOT}/data/md_max/{split}.jsonl"):+ r = json.loads(l)+ msgs = r["messages"]+ by[r["dialect"]].append({+ "id": r["id"], "subset": r["subset"], "caption_type": r["caption"], "split": split,+ "system": msgs[0]["content"],+ "conversations": [{"from": "human", "value": msgs[1]["content"]},+ {"from": "gpt", "value": msgs[2]["content"]}],+ })+ for dia, rows in by.items():+ nsh = write_parquet(rows, f"{a.out}/sft/{dia}/train")+ info[f"3dcv_{dia}"] = {"file_name": f"sft/{dia}", "formatting": "sharegpt", "columns": {"messages": "conversations", "system": "system"}}+ stats[f"sft/{dia}"] = {"rows": len(rows), "shards": nsh}+ print(f"[hf] sft/{dia}: {len(rows)} rows, {nsh} shard(s)")++ # ---- 2. the bootstrapped Blender set (execution-verified, generated by our own flywheel) ----+ boot = []+ for l in open(f"{ROOT}/data/sft_v13_boot/train.jsonl"):+ if '"source": "boot_' not in l:+ continue+ m = json.loads(l)["messages"]+ boot.append({"system": m[0]["content"],+ "conversations": [{"from": "human", "value": m[1]["content"]}, {"from": "gpt", "value": m[2]["content"]}]})+ write_parquet(boot, f"{a.out}/sft/blender_bootstrapped/train")+ info["3dcv_blender_bootstrapped"] = {"file_name": "sft/blender_bootstrapped", "formatting": "sharegpt", "columns": {"messages": "conversations", "system": "system"}}+ stats["sft/blender_bootstrapped"] = {"rows": len(boot), "shards": 1}+ print(f"[hf] sft/blender_bootstrapped: {len(boot)} rows")++ # ---- 3. preference sets ----+ for name, (path, desc) in DPO_SETS.items():+ p = os.path.join(ROOT, path)+ if not os.path.exists(p):+ print(f"[hf] skip {name} (missing)")+ continue+ rows = json.load(open(p))+ write_parquet(rows, f"{a.out}/dpo/{name}")+ info[f"3dcv_dpo_{name}"] = {"file_name": f"dpo/{name}.parquet", "ranking": True, "formatting": "sharegpt",+ "columns": {"messages": "conversations", "system": "system", "chosen": "chosen", "rejected": "rejected"}}+ stats[f"dpo/{name}"] = {"rows": len(rows), "desc": desc}+ print(f"[hf] dpo/{name}: {len(rows)} pairs")++ # ---- 4. held-out test sets ----+ for f in sorted(glob.glob(f"{ROOT}/data/multidialect/*/test.jsonl")):+ dia = f.split("/")[-2]+ rows = []+ for l in open(f):+ r = json.loads(l)+ m = r["messages"]+ rows.append({"id": r["id"], "system": m[0]["content"], "prompt": m[1]["content"], "reference_code": m[2]["content"]})+ write_parquet(rows, f"{a.out}/test/{dia}")+ stats[f"test/{dia}"] = {"rows": len(rows)}+ print(f"[hf] test/{dia}: {len(rows)} rows")++ json.dump(info, open(f"{a.out}/dataset_info.json", "w"), indent=2, ensure_ascii=False)+ json.dump({"mixes": MIXES, "contents": stats}, open(f"{a.out}/mixes.json", "w"), indent=2, ensure_ascii=False)++ total = sum(v.get("rows", 0) for k, v in stats.items() if k.startswith("sft/"))+ card = f"""---+license: cc-by-nc-sa-4.0+task_categories:+ - text-generation+language:+ - en+tags:+ - 3d+ - code-generation+ - blender+ - cadquery+ - openscad+ - glsl+ - threejs+ - llama-factory+ - sft+ - dpo+size_categories:+ - 100K<n<1M+---++# 3DCodeVerse — LLaMA-Factory edition++Text→3D-code training data in **LLaMA-Factory ShareGPT format**, derived from+[`ilabai/3dcodeverse`](https://huggingface.co/datasets/ilabai/3dcodeverse). Same license and non-commercial+research terms as the parent dataset; the per-source licenses (Infinigen/BSD-3, Bioinspired3D/MIT, Shadertoy/CC BY-NC-SA,+Thingiverse, DeepCAD/MIT) still apply.++Training and evaluation code: <https://github.com/gaoypeng/3dcodeverse/tree/main/finetune>.+Full experiment log: [`llm_finetune_exps.md`](https://github.com/gaoypeng/3dcodeverse/blob/main/finetune/llm_finetune_exps.md).++## Contents++| path | what | rows |+|---|---|---|+| `sft/blender/` | Blender-Python (bioinspired3d, 3dcodebench factories, blender_distill) | {stats.get('sft/blender', {}).get('rows', 0):,} |+| `sft/cadquery/` | CadQuery (deepcad, articraft) | {stats.get('sft/cadquery', {}).get('rows', 0):,} |+| `sft/glsl/` | GLSL fragment shaders (shadertoy) | {stats.get('sft/glsl', {}).get('rows', 0):,} |+| `sft/openscad/` | OpenSCAD (thingiverse) | {stats.get('sft/openscad', {}).get('rows', 0):,} |+| `sft/threejs/` | single-file three.js pages | {stats.get('sft/threejs', {}).get('rows', 0):,} |+| `sft/web/` | animated web pages (animation2code) | {stats.get('sft/web', {}).get('rows', 0):,} |+| `sft/blender_bootstrapped/` | **execution-verified** Blender samples generated by a finetuned model on novel prompts and kept only if they ran | {stats.get('sft/blender_bootstrapped', {}).get('rows', 0):,} |+| `dpo/*.parquet` | execution-feedback and geometry-feedback preference pairs (see below) | 8k pairs total |+| `test/*.parquet` | held-out evaluation sets, never trained on | 593 |++Total SFT pairs: **{total:,}** (one row per prompt variant; each source sample has up to 3 caption styles —+`caption_type` is `instruction` / `detailed` / `factory`).++## How to use with LLaMA-Factory++```bash+huggingface-cli download ilabai/3dcodeverse-llamafactory --repo-type dataset --local-dir data/3dcv+cp data/3dcv/dataset_info.json data/3dcv/ # already there; point dataset_dir at data/3dcv+llamafactory-cli train your_config.yaml # with: dataset_dir: data/3dcv, dataset: 3dcv_blender,3dcv_cadquery,...+```++`dataset_info.json` registers every subset (`3dcv_<dialect>`, `3dcv_blender_bootstrapped`, `3dcv_dpo_<name>`).++## Reproducing the mixes from the report++The report's training mixes are *samples* of this corpus, so they are described by a budget rather than shipped+as duplicated files (`mixes.json` has the exact specs; `scripts/sample_mix.py` in the repo applies them):++| mix | pairs | tokens | captions/sample | result |+|---|---|---|---|---|+{chr(10).join(f"| `{k}` | {v['pairs']:,} | {v['tokens']} | {v['captions_per_sample']} | {v['desc']} |" for k, v in MIXES.items())}++## Two findings that matter when using this data++1. **Do not use all three caption variants of the same code sample.** At a fixed token budget, 3 captions per sample+ scored 34.4% on 3DCodeBench versus 90.1% for one caption per sample plus the bootstrapped set — what generalises+ is the number of *distinct runnable programs*, not the number of prompts.+2. **`sft/blender_bootstrapped/` is the highest-value subset per token.** Adding it (and raising the Blender share to+ ~40%) took a 27B model from 57.6% to 93.4% on 3DCodeBench.++## Evaluation++Execution-based: every generated program is run (Blender 5.0.1 headless / CadQuery / OpenSCAD / glslangValidator /+headless Chromium) and the resulting mesh compared to the reference with Chamfer distance and F-score. Scripts:+`finetune/eval/` in the GitHub repo. Note: Blender's bundled Python needs `scipy` — 15 of the 212 3DCodeBench+reference scripts import it.+"""+ open(f"{a.out}/README.md", "w").write(card)+ print(f"[hf] wrote {a.out} ({sum(os.path.getsize(p) for p in glob.glob(a.out + '/**/*', recursive=True) if os.path.isfile(p))/1e9:.2f} GB)")++ if a.push:+ from huggingface_hub import HfApi+ tok = os.environ.get("HF_TOKEN")+ api = HfApi(token=tok)+ api.create_repo(a.push, repo_type="dataset", private=a.private, exist_ok=True)+ api.upload_folder(folder_path=a.out, repo_id=a.push, repo_type="dataset",+ commit_message="3DCodeVerse in LLaMA-Factory format: SFT corpus by dialect, bootstrapped Blender set, DPO preference sets, held-out tests")+ print(f"[hf] pushed to https://huggingface.co/datasets/{a.push}")+++if __name__ == "__main__":+ main()diff --git a/finetune/scripts/queue.txt b/finetune/scripts/queue.txtnew file mode 100644index 00000000..f69b7fa8--- /dev/null+++ b/finetune/scripts/queue.txt@@ -0,0 +1,2 @@+v3_smoke|GPUS=$(scripts/free_gpus.sh) scripts/lf_train.sh configs/lf/lora_27b_v3.yaml max_steps=6 save_steps=100000 eval_steps=100000 output_dir=/wekafs/ict/hx_624/llm-ft/runs/_smoke_v3 > logs/smoke_v3.log 2>&1; grep -q train_runtime logs/smoke_v3.log && { echo SMOKE_OK; rm -rf runs/_smoke_v3; } || { echo SMOKE_OOM; grep -o "Tried to allocate [0-9.]* GiB" logs/smoke_v3.log | tail -1; exit 1; }+v3_full|scripts/run_27b_v3.shdiff --git a/finetune/scripts/supervisor.sh b/finetune/scripts/supervisor.shnew file mode 100755index 00000000..890d3176--- /dev/null+++ b/finetune/scripts/supervisor.sh@@ -0,0 +1,34 @@+#!/bin/bash+# Keeps GPUs 0-3 busy: runs the pending jobs in scripts/queue.txt one at a time, restarts nothing that succeeded,+# and reports every state change to logs/supervisor.log. One line per job: "<label>|<command>".+set -uo pipefail; cd /wekafs/ict/hx_624/llm-ft+Q=scripts/queue.txt; STATE=logs/.supervisor_done+touch $STATE+while true; do+ PENDING=""+ while IFS='|' read -r label cmd; do+ [ -z "${label:-}" ] && continue+ case "$label" in \#*) continue;; esac+ grep -qx "$label" $STATE || { PENDING="$label|$cmd"; break; }+ done < $Q+ if [ -z "$PENDING" ]; then echo "[sup] $(date) queue empty, exiting"; break; fi+ label="${PENDING%%|*}"; cmd="${PENDING#*|}"+ # wait until no training/inference of mine is running and the GPUs are idle (3 confirmations)+ STABLE=0+ while [ $STABLE -lt 3 ]; do+ # count only REAL trainer/inference processes (python/torchrun): a plain `pgrep -f` also matches+ # monitoring shells whose own command line contains these strings, which deadlocks the queue+ MINE=0+ for pid in $(pgrep -u $USER -f "llamafactory-cli train|generate_multi.py|generate_vllm.py" 2>/dev/null); do+ case "$(ps -o comm= -p $pid 2>/dev/null)" in python*|torchrun*|pt_main*) MINE=$((MINE+1));; esac+ done+ F=$(scripts/free_gpus.sh); N=$(echo "$F" | awk -F, 'NF&&$1!=""{print NF}')+ if [ "$MINE" -eq 0 ] && [ "${N:-0}" -ge 2 ]; then STABLE=$((STABLE+1)); else STABLE=0; fi+ sleep 30+ done+ echo "[sup] $(date) starting: $label"+ bash -c "$cmd" >> logs/sup_$label.log 2>&1+ RC=$?+ if [ $RC -eq 0 ]; then echo "[sup] $(date) OK: $label"; else echo "[sup] $(date) FAILED (rc=$RC): $label — see logs/sup_$label.log"; fi+ echo "$label" >> $STATE+done