Skip to content

Official Codebase of 3DCodeVerse

Python

stop-DPO: the termination pairs transfer, and generating per-model ones is not worth it

gaoypeng01a820fc584b
diff --git a/finetune/configs/lf/dpo_stop_plus_own.yaml b/finetune/configs/lf/dpo_stop_plus_own.yaml
new file mode 100644
index 00000000..39ecbde7
--- /dev/null
+++ b/finetune/configs/lf/dpo_stop_plus_own.yaml
@@ -0,0 +1,42 @@
+### DPO with execution-feedback preference pairs, on top of LoRA-v1 (merged)
+model_name_or_path: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus/merged
+trust_remote_code: true
+flash_attn: fa2
+stage: dpo
+do_train: true
+finetuning_type: lora
+lora_rank: 16
+lora_alpha: 32
+lora_dropout: 0.05
+lora_target: all
+freeze_vision_tower: true
+freeze_multi_modal_projector: true
+pref_beta: 0.1
+pref_loss: sigmoid
+dataset_dir: /wekafs/ict/hx_624/llm-ft/data/lf
+dataset: stop_dpo_pairs_plus
+template: qwen3_5
+enable_thinking: false
+cutoff_len: 2048
+preprocessing_num_workers: 16
+dataloader_num_workers: 2
+output_dir: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus_stopown
+logging_steps: 5
+save_steps: 1000
+save_only_model: true
+plot_loss: true
+overwrite_output_dir: true
+report_to: tensorboard
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 8
+learning_rate: 5.0e-6
+num_train_epochs: 2.0
+lr_scheduler_type: cosine
+warmup_ratio: 0.1
+bf16: true
+gradient_checkpointing: true
+ddp_timeout: 180000000
+val_size: 0.02
+per_device_eval_batch_size: 1
+eval_strategy: steps
+eval_steps: 100
diff --git a/finetune/configs/lf/export_lf_qwen35_9b_plus_stop.yaml b/finetune/configs/lf/export_lf_qwen35_9b_plus_stop.yaml
new file mode 100644
index 00000000..104622d9
--- /dev/null
+++ b/finetune/configs/lf/export_lf_qwen35_9b_plus_stop.yaml
@@ -0,0 +1,12 @@
+### Merge a LoRA adapter into the base model (LLaMA-Factory): llamafactory-cli export <this file>
+model_name_or_path: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus/merged
+adapter_name_or_path: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus_stop
+template: qwen3_5
+enable_thinking: false
+trust_remote_code: true
+finetuning_type: lora
+
+export_dir: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus_stop/merged
+export_size: 5
+export_device: cpu
+export_legacy_format: false
diff --git a/finetune/configs/lf/export_lf_qwen35_9b_plus_stopown.yaml b/finetune/configs/lf/export_lf_qwen35_9b_plus_stopown.yaml
new file mode 100644
index 00000000..1b786185
--- /dev/null
+++ b/finetune/configs/lf/export_lf_qwen35_9b_plus_stopown.yaml
@@ -0,0 +1,12 @@
+### Merge a LoRA adapter into the base model (LLaMA-Factory): llamafactory-cli export <this file>
+model_name_or_path: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus/merged
+adapter_name_or_path: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus_stopown
+template: qwen3_5
+enable_thinking: false
+trust_remote_code: true
+finetuning_type: lora
+
+export_dir: /wekafs/ict/hx_624/llm-ft/runs/lf_qwen35_9b_plus_stopown/merged
+export_size: 5
+export_device: cpu
+export_legacy_format: false
diff --git a/finetune/data_specs/dataset_info.json b/finetune/data_specs/dataset_info.json
index 1f30d41b..b17d1313 100644
--- a/finetune/data_specs/dataset_info.json
+++ b/finetune/data_specs/dataset_info.json
@@ -639,5 +639,16 @@
"messages": "conversations",
"system": "system"
}
+ },
+ "stop_dpo_pairs_plus": {
+ "file_name": "stop_dpo_pairs_plus.json",
+ "ranking": true,
+ "formatting": "sharegpt",
+ "columns": {
+ "messages": "conversations",
+ "system": "system",
+ "chosen": "chosen",
+ "rejected": "rejected"
+ }
}
}
\ No newline at end of file
diff --git a/finetune/docs/REPORT.md b/finetune/docs/REPORT.md
index 5d7a212a..81057038 100644
--- a/finetune/docs/REPORT.md
+++ b/finetune/docs/REPORT.md
@@ -85,8 +85,16 @@ OpenSCAD 和 three.js 的问题不是训练方法,是**语料里只有一千
输出长度同步坍缩:OpenSCAD 从 30,166 token 降到 410。
-**三条延伸结论:**
-1. **可迁移** — 用 mmmix2 生成的配对训练另一个模型(dpo_exec_v2),OpenSCAD 44% → 82%,bench 涨到 95.8%
+**四条延伸结论:**
+1. **可迁移,而且自己生成不划算。** 同一批 mmmix2 产出的 1,605 对,在三个不同模型上都有效:
+
+ | 被训练的模型 | OpenSCAD | bench |
+ |---|---|---|
+ | mmmix2(自产) | 6% → 94% | 93.9% → 89.1% |
+ | dpo_exec_v2(迁移) | 44% → 82% | 94.3% → 95.8% |
+ | md_9b_plus(迁移) | 8% → 70% | 79.2% → 85.9% |
+
+ 在 md_9b_plus 上做了单变量对照——同一基座、同样超参数,只换配对来源:迁移版 bench 85.9 / OpenSCAD 70.0 / three.js 75.0,自产版 85.4 / 50.0 / 72.5,**迁移赢四项自产赢两项,差距都在噪声量级**。而自产要多花 2.5 小时生成。所以正确用法是**用一个健康模型产出一批终止样本,当作可复用资产分发**
2. **训练后贪心优于采样**(82.0% vs 76.0%)——说明是真修好了,不是换解码绕过去
3. **回头解释了旧现象** — 此前执行反馈 DPO 带来的大幅提升(93.9% → 96.2%),很可能有一半功劳是它顺带教会了终止:在 dpo_exec_v2 上跑配对生成,3,687 个 prompt 里 3,247 个贪心本来就正常,**产出 0 对**
@@ -133,6 +141,7 @@ OpenSCAD 和 three.js 的问题不是训练方法,是**语料里只有一千
* **Qwen3.5/3.8 有线性注意力层**,vLLM 每个并发序列要一个 Mamba cache block。27B 只剩 179 块而默认 `max_num_seqs` 是 1024,必须显式调小,否则引擎起不来——而报错信息和显存不足完全不同。
* **多模态混合数据集时,图像数据必须排在文本前面。** HuggingFace 按第一个数据集确定 schema,文本在前会把 `images` 列推断成 null 类型,后面的图像数据 cast 不进去。
* **超过 2 GB 的数据集必须存成 parquet,不能用 JSON。** Arrow 读单个大 JSON 会抛 `OverflowError: value too large to convert to int32_t`,报错完全看不出是文件体积问题。3.05 GB 的 JSON 转 parquet 后是 1.04 GB,问题消失。
+* **共用机器上的五个坑**,每一个在独占机器上都不存在:(1) 别人常驻的占位进程可以吃掉 135 GB,而且声明 16 GiB 实占 33.8 GB;(2) `gpu_memory_utilization` 是总显存的比例,不是空闲的;(3) 27B 的四次 OOM 全部发生在与他人共享的那张卡上,换到全空的卡就能跑;(4) 选卡脚本硬编码只看前四张,扩到八卡后一直看不见后四张;(5) **选卡与用卡之间存在竞态**——卡在选中时空闲、启动引擎时已被占满,引擎死掉且不产生任何输出。评测入口现在会在启动前重新确认并换卡重试。
* **绝不修改正在运行的脚本。** bash 按字节偏移读取文件,改动会让它执行到错位的字节。这在项目里发生过两次。
---
diff --git a/finetune/eval/eval_all_run2.sh b/finetune/eval/eval_all_run2.sh
index 52338160..478c0316 100755
--- a/finetune/eval/eval_all_run2.sh
+++ b/finetune/eval/eval_all_run2.sh
@@ -52,9 +52,28 @@ for s in suites:
json.dump(jobs, open(spec_path, "w"))
print("[eval_all] suites:", [j["name"] for j in jobs])
PY
-python eval/generate_multi.py --model $MODEL --spec @$SPEC --tp $TP $NOTHINK \
- --max_new_tokens $MAXNEW --max_model_len $MAXLEN --temperature $TEMP --seed $SEED 2>&1 \
- | grep -E "gen-multi|extract\]|Error|Traceback" | tail -40
+# On a shared box a card that was free when it was picked can be full by the time vLLM starts, and the engine
+# dies with no generations at all. Re-check right before launching, and fall back to whichever card is free now.
+for ATTEMPT in 1 2 3; do
+ FREE_NOW=$(scripts/free_gpus.sh 40000)
+ case ",$FREE_NOW," in
+ *",$GPU,"*) ;;
+ *) NEW=$(echo "$FREE_NOW" | cut -d, -f1)
+ if [ -n "$NEW" ]; then
+ echo "[eval_all] GPU $GPU filled up since it was picked; switching to $NEW"
+ GPU=$NEW; export CUDA_VISIBLE_DEVICES=$GPU
+ else
+ echo "[eval_all] no card has 40 GB free; waiting 120s (attempt $ATTEMPT/3)"; sleep 120; continue
+ fi;;
+ esac
+ python eval/generate_multi.py --model $MODEL --spec @$SPEC --tp $TP $NOTHINK \
+ --max_new_tokens $MAXNEW --max_model_len $MAXLEN --temperature $TEMP --seed $SEED 2>&1 \
+ | grep -E "gen-multi|extract\]|Error|Traceback" | tail -40
+ # one generation directory per task; anything less means the engine never ran
+ if [ "$(find eval/out/${NAME}_$(echo $SUITES | awk '{print $1}') -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l)" -ge 5 ]; then break; fi
+ echo "[eval_all] generation produced nothing on attempt $ATTEMPT; retrying"
+ sleep 60
+done
rm -f $SPEC
# ---------- execution + metrics (llmft env) ----------
diff --git a/finetune/scripts/run_plus_stop_eval.sh b/finetune/scripts/run_plus_stop_eval.sh
new file mode 100755
index 00000000..40268e2c
--- /dev/null
+++ b/finetune/scripts/run_plus_stop_eval.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+# Same base, same hyper-parameters, only the pair source differs: mmmix2's pairs (transfer) vs md_9b_plus's own.
+# Answers whether the 2.5 hours of per-model pair generation buys anything over reusing an existing set.
+set -uo pipefail; cd /wekafs/ict/hx_624/llm-ft
+source /wekafs/ict/hx_624/anaconda3/etc/profile.d/conda.sh
+BASE=$(readlink -f runs/lf_qwen35_9b_plus/merged)
+for SPEC in "lf_qwen35_9b_plus_stop:q9b_plus_stop:transfer" "lf_qwen35_9b_plus_stopown:q9b_plus_stopown:own"; do
+ IFS=: read -r RUNDIR NAME TAG <<< "$SPEC"
+ RUN=runs/$RUNDIR
+ for _ in $(seq 1 90); do [ -f $RUN/train_results.json ] && break; sleep 60; done
+ [ -f $RUN/train_results.json ] || { echo "[pstop-$TAG] training never finished"; continue; }
+ if [ ! -d $RUN/merged ]; then
+ sed -e "s|model_name_or_path:.*|model_name_or_path: $BASE|" -e "s|adapter_name_or_path:.*|adapter_name_or_path: $(readlink -f $RUN)|" -e "s|export_dir:.*|export_dir: $(readlink -f $RUN)/merged|" configs/lf/export_lora_template.yaml > configs/lf/export_$RUNDIR.yaml
+ scripts/lf_export.sh configs/lf/export_$RUNDIR.yaml > logs/export_$RUNDIR.log 2>&1 || { echo "[pstop-$TAG] EXPORT FAILED"; continue; }
+ fi
+ G=""; for _ in $(seq 1 60); do G=$(scripts/free_gpus.sh 40000 | cut -d, -f1); [ -n "$G" ] && break; sleep 30; done
+ [ -z "$G" ] && { echo "[pstop-$TAG] no GPU"; continue; }
+ echo "[pstop-$TAG] $(date) evaluating on GPU $G"
+ GPUS=$G TP=1 bash eval/eval_all_run2.sh $RUN/merged ${NAME}_greedy 2>&1 | tail -9
+ GPUS=$G TP=1 bash eval/eval_all_run2.sh $RUN/merged ${NAME}_t07 --temp 0.7 --seed 1 2>&1 | tail -9
+ echo "[pstop-$TAG] DONE"
+done
+conda activate llmft
+python - <<'PY'
+import json, os
+suites = ["bench", "blender", "cadquery", "openscad", "glsl", "threejs"]
+sets = [("md_9b_plus greedy", "q9b_plus_greedy"), ("md_9b_plus T=0.7", "q9b_plus_t07"),
+ ("+transfer pairs greedy", "q9b_plus_stop_greedy"), ("+transfer pairs T=0.7", "q9b_plus_stop_t07"),
+ ("+own pairs greedy", "q9b_plus_stopown_greedy"), ("+own pairs T=0.7", "q9b_plus_stopown_t07")]
+for title, key in (("exec rate", "exec_rate"), ("mean output tokens", "mean_new_tokens")):
+ print(f" --- {title} ---")
+ print(f" {'model':26s}" + "".join(f"{s[:8]:>10s}" for s in suites))
+ for label, n in sets:
+ p = f"eval/out/{n}_report.json"
+ if not os.path.exists(p): print(f" {label:26s} (missing)"); continue
+ d = {x["suite"]: x for x in json.load(open(p))["suites"]}
+ if key == "exec_rate":
+ print(f" {label:26s}" + "".join(f"{d[s][key]*100:9.1f}%" if s in d else f"{'-':>10s}" for s in suites))
+ else:
+ print(f" {label:26s}" + "".join(f"{d[s].get(key,0):10,.0f}" if s in d else f"{'-':>10s}" for s in suites))
+PY
+echo "[pstop] ALL DONE"