Official Codebase of 3DCodeVerse
Python
stop-DPO: the termination pairs transfer, and generating per-model ones is not worth it
gaoypeng01a820fc584b
diff --git a/finetune/configs/lf/dpo_stop_plus_own.yaml b/finetune/configs/lf/dpo_stop_plus_own.yamlnew file mode 100644index 00000000..39ecbde7--- /dev/null+++ b/finetune/configs/lf/dpo_stop_plus_own.yaml@@ -0,0 +1,42 @@+### DPO with execution-feedback preference pairs, on top of LoRA-v1 (merged)+model_name_or_path: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus/merged+trust_remote_code: true+flash_attn: fa2+stage: dpo+do_train: true+finetuning_type: lora+lora_rank: 16+lora_alpha: 32+lora_dropout: 0.05+lora_target: all+freeze_vision_tower: true+freeze_multi_modal_projector: true+pref_beta: 0.1+pref_loss: sigmoid+dataset_dir: /wekafs/ict/hx_624/llm-ft/data/lf+dataset: stop_dpo_pairs_plus+template: qwen3_5+enable_thinking: false+cutoff_len: 2048+preprocessing_num_workers: 16+dataloader_num_workers: 2+output_dir: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus_stopown+logging_steps: 5+save_steps: 1000+save_only_model: true+plot_loss: true+overwrite_output_dir: true+report_to: tensorboard+per_device_train_batch_size: 1+gradient_accumulation_steps: 8+learning_rate: 5.0e-6+num_train_epochs: 2.0+lr_scheduler_type: cosine+warmup_ratio: 0.1+bf16: true+gradient_checkpointing: true+ddp_timeout: 180000000+val_size: 0.02+per_device_eval_batch_size: 1+eval_strategy: steps+eval_steps: 100diff --git a/finetune/configs/lf/export_lf_qwen35_9b_plus_stop.yaml b/finetune/configs/lf/export_lf_qwen35_9b_plus_stop.yamlnew file mode 100644index 00000000..104622d9--- /dev/null+++ b/finetune/configs/lf/export_lf_qwen35_9b_plus_stop.yaml@@ -0,0 +1,12 @@+### Merge a LoRA adapter into the base model (LLaMA-Factory): llamafactory-cli export <this file>+model_name_or_path: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus/merged+adapter_name_or_path: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus_stop+template: qwen3_5+enable_thinking: false+trust_remote_code: true+finetuning_type: lora++export_dir: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus_stop/merged+export_size: 5+export_device: cpu+export_legacy_format: falsediff --git a/finetune/configs/lf/export_lf_qwen35_9b_plus_stopown.yaml b/finetune/configs/lf/export_lf_qwen35_9b_plus_stopown.yamlnew file mode 100644index 00000000..1b786185--- /dev/null+++ b/finetune/configs/lf/export_lf_qwen35_9b_plus_stopown.yaml@@ -0,0 +1,12 @@+### Merge a LoRA adapter into the base model (LLaMA-Factory): llamafactory-cli export <this file>+model_name_or_path: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus/merged+adapter_name_or_path: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus_stopown+template: qwen3_5+enable_thinking: false+trust_remote_code: true+finetuning_type: lora++export_dir: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus_stopown/merged+export_size: 5+export_device: cpu+export_legacy_format: falsediff --git a/finetune/data_specs/dataset_info.json b/finetune/data_specs/dataset_info.jsonindex 1f30d41b..b17d1313 100644--- a/finetune/data_specs/dataset_info.json+++ b/finetune/data_specs/dataset_info.json@@ -639,5 +639,16 @@"messages": "conversations","system": "system"}+ },+ "stop_dpo_pairs_plus": {+ "file_name": "stop_dpo_pairs_plus.json",+ "ranking": true,+ "formatting": "sharegpt",+ "columns": {+ "messages": "conversations",+ "system": "system",+ "chosen": "chosen",+ "rejected": "rejected"+ }}}\ No newline at end of filediff --git a/finetune/docs/REPORT.md b/finetune/docs/REPORT.mdindex 5d7a212a..81057038 100644--- a/finetune/docs/REPORT.md+++ b/finetune/docs/REPORT.md@@ -85,8 +85,16 @@ OpenSCAD 和 three.js 的问题不是训练方法,是**语料里只有一千输出长度同步坍缩:OpenSCAD 从 30,166 token 降到 410。-**三条延伸结论:**-1. **可迁移** — 用 mmmix2 生成的配对训练另一个模型(dpo_exec_v2),OpenSCAD 44% → 82%,bench 涨到 95.8%+**四条延伸结论:**+1. **可迁移,而且自己生成不划算。** 同一批 mmmix2 产出的 1,605 对,在三个不同模型上都有效:++ | 被训练的模型 | OpenSCAD | bench |+ |---|---|---|+ | mmmix2(自产) | 6% → 94% | 93.9% → 89.1% |+ | dpo_exec_v2(迁移) | 44% → 82% | 94.3% → 95.8% |+ | md_9b_plus(迁移) | 8% → 70% | 79.2% → 85.9% |++ 在 md_9b_plus 上做了单变量对照——同一基座、同样超参数,只换配对来源:迁移版 bench 85.9 / OpenSCAD 70.0 / three.js 75.0,自产版 85.4 / 50.0 / 72.5,**迁移赢四项自产赢两项,差距都在噪声量级**。而自产要多花 2.5 小时生成。所以正确用法是**用一个健康模型产出一批终止样本,当作可复用资产分发**2. **训练后贪心优于采样**(82.0% vs 76.0%)——说明是真修好了,不是换解码绕过去3. **回头解释了旧现象** — 此前执行反馈 DPO 带来的大幅提升(93.9% → 96.2%),很可能有一半功劳是它顺带教会了终止:在 dpo_exec_v2 上跑配对生成,3,687 个 prompt 里 3,247 个贪心本来就正常,**产出 0 对**@@ -133,6 +141,7 @@ OpenSCAD 和 three.js 的问题不是训练方法,是**语料里只有一千* **Qwen3.5/3.8 有线性注意力层**,vLLM 每个并发序列要一个 Mamba cache block。27B 只剩 179 块而默认 `max_num_seqs` 是 1024,必须显式调小,否则引擎起不来——而报错信息和显存不足完全不同。* **多模态混合数据集时,图像数据必须排在文本前面。** HuggingFace 按第一个数据集确定 schema,文本在前会把 `images` 列推断成 null 类型,后面的图像数据 cast 不进去。* **超过 2 GB 的数据集必须存成 parquet,不能用 JSON。** Arrow 读单个大 JSON 会抛 `OverflowError: value too large to convert to int32_t`,报错完全看不出是文件体积问题。3.05 GB 的 JSON 转 parquet 后是 1.04 GB,问题消失。+* **共用机器上的五个坑**,每一个在独占机器上都不存在:(1) 别人常驻的占位进程可以吃掉 135 GB,而且声明 16 GiB 实占 33.8 GB;(2) `gpu_memory_utilization` 是总显存的比例,不是空闲的;(3) 27B 的四次 OOM 全部发生在与他人共享的那张卡上,换到全空的卡就能跑;(4) 选卡脚本硬编码只看前四张,扩到八卡后一直看不见后四张;(5) **选卡与用卡之间存在竞态**——卡在选中时空闲、启动引擎时已被占满,引擎死掉且不产生任何输出。评测入口现在会在启动前重新确认并换卡重试。* **绝不修改正在运行的脚本。** bash 按字节偏移读取文件,改动会让它执行到错位的字节。这在项目里发生过两次。---diff --git a/finetune/eval/eval_all_run2.sh b/finetune/eval/eval_all_run2.shindex 52338160..478c0316 100755--- a/finetune/eval/eval_all_run2.sh+++ b/finetune/eval/eval_all_run2.sh@@ -52,9 +52,28 @@ for s in suites:json.dump(jobs, open(spec_path, "w"))print("[eval_all] suites:", [j["name"] for j in jobs])PY-python eval/generate_multi.py --model $MODEL --spec @$SPEC --tp $TP $NOTHINK \- --max_new_tokens $MAXNEW --max_model_len $MAXLEN --temperature $TEMP --seed $SEED 2>&1 \- | grep -E "gen-multi|extract\]|Error|Traceback" | tail -40+# On a shared box a card that was free when it was picked can be full by the time vLLM starts, and the engine+# dies with no generations at all. Re-check right before launching, and fall back to whichever card is free now.+for ATTEMPT in 1 2 3; do+ FREE_NOW=$(scripts/free_gpus.sh 40000)+ case ",$FREE_NOW," in+ *",$GPU,"*) ;;+ *) NEW=$(echo "$FREE_NOW" | cut -d, -f1)+ if [ -n "$NEW" ]; then+ echo "[eval_all] GPU $GPU filled up since it was picked; switching to $NEW"+ GPU=$NEW; export CUDA_VISIBLE_DEVICES=$GPU+ else+ echo "[eval_all] no card has 40 GB free; waiting 120s (attempt $ATTEMPT/3)"; sleep 120; continue+ fi;;+ esac+ python eval/generate_multi.py --model $MODEL --spec @$SPEC --tp $TP $NOTHINK \+ --max_new_tokens $MAXNEW --max_model_len $MAXLEN --temperature $TEMP --seed $SEED 2>&1 \+ | grep -E "gen-multi|extract\]|Error|Traceback" | tail -40+ # one generation directory per task; anything less means the engine never ran+ if [ "$(find eval/out/${NAME}_$(echo $SUITES | awk '{print $1}') -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l)" -ge 5 ]; then break; fi+ echo "[eval_all] generation produced nothing on attempt $ATTEMPT; retrying"+ sleep 60+donerm -f $SPEC# ---------- execution + metrics (llmft env) ----------diff --git a/finetune/scripts/run_plus_stop_eval.sh b/finetune/scripts/run_plus_stop_eval.shnew file mode 100755index 00000000..40268e2c--- /dev/null+++ b/finetune/scripts/run_plus_stop_eval.sh@@ -0,0 +1,42 @@+#!/bin/bash+# Same base, same hyper-parameters, only the pair source differs: mmmix2's pairs (transfer) vs md_9b_plus's own.+# Answers whether the 2.5 hours of per-model pair generation buys anything over reusing an existing set.+set -uo pipefail; cd /wekafs/ict/hx_624/llm-ft+source /wekafs/ict/hx_624/anaconda3/etc/profile.d/conda.sh+BASE=$(readlink -f runs/lf_qwen35_9b_plus/merged)+for SPEC in "lf_qwen35_9b_plus_stop:q9b_plus_stop:transfer" "lf_qwen35_9b_plus_stopown:q9b_plus_stopown:own"; do+ IFS=: read -r RUNDIR NAME TAG <<< "$SPEC"+ RUN=runs/$RUNDIR+ for _ in $(seq 1 90); do [ -f $RUN/train_results.json ] && break; sleep 60; done+ [ -f $RUN/train_results.json ] || { echo "[pstop-$TAG] training never finished"; continue; }+ if [ ! -d $RUN/merged ]; then+ sed -e "s|model_name_or_path:.*|model_name_or_path: $BASE|" -e "s|adapter_name_or_path:.*|adapter_name_or_path: $(readlink -f $RUN)|" -e "s|export_dir:.*|export_dir: $(readlink -f $RUN)/merged|" configs/lf/export_lora_template.yaml > configs/lf/export_$RUNDIR.yaml+ scripts/lf_export.sh configs/lf/export_$RUNDIR.yaml > logs/export_$RUNDIR.log 2>&1 || { echo "[pstop-$TAG] EXPORT FAILED"; continue; }+ fi+ G=""; for _ in $(seq 1 60); do G=$(scripts/free_gpus.sh 40000 | cut -d, -f1); [ -n "$G" ] && break; sleep 30; done+ [ -z "$G" ] && { echo "[pstop-$TAG] no GPU"; continue; }+ echo "[pstop-$TAG] $(date) evaluating on GPU $G"+ GPUS=$G TP=1 bash eval/eval_all_run2.sh $RUN/merged ${NAME}_greedy 2>&1 | tail -9+ GPUS=$G TP=1 bash eval/eval_all_run2.sh $RUN/merged ${NAME}_t07 --temp 0.7 --seed 1 2>&1 | tail -9+ echo "[pstop-$TAG] DONE"+done+conda activate llmft+python - <<'PY'+import json, os+suites = ["bench", "blender", "cadquery", "openscad", "glsl", "threejs"]+sets = [("md_9b_plus greedy", "q9b_plus_greedy"), ("md_9b_plus T=0.7", "q9b_plus_t07"),+ ("+transfer pairs greedy", "q9b_plus_stop_greedy"), ("+transfer pairs T=0.7", "q9b_plus_stop_t07"),+ ("+own pairs greedy", "q9b_plus_stopown_greedy"), ("+own pairs T=0.7", "q9b_plus_stopown_t07")]+for title, key in (("exec rate", "exec_rate"), ("mean output tokens", "mean_new_tokens")):+ print(f" --- {title} ---")+ print(f" {'model':26s}" + "".join(f"{s[:8]:>10s}" for s in suites))+ for label, n in sets:+ p = f"eval/out/{n}_report.json"+ if not os.path.exists(p): print(f" {label:26s} (missing)"); continue+ d = {x["suite"]: x for x in json.load(open(p))["suites"]}+ if key == "exec_rate":+ print(f" {label:26s}" + "".join(f"{d[s][key]*100:9.1f}%" if s in d else f"{'-':>10s}" for s in suites))+ else:+ print(f" {label:26s}" + "".join(f"{d[s].get(key,0):10,.0f}" if s in d else f"{'-':>10s}" for s in suites))+PY+echo "[pstop] ALL DONE"