Skip to content

Official Codebase of 3DCodeVerse

Python
3.8 KB
Raw Blame

Released weights

Two LoRA adapters trained with this component are public on the Hugging Face Hub, grouped with the paper and dataset in the 3DCodeVerse collection.

repo base model training run size
3dcodeverse/3dcodeverse-qwen3.6-27b-lora Qwen/Qwen3.6-27B qwen3.6-27b_lora_3dcv_text_A: 3dcv_text_all mix, 5,515 steps, global batch 48, eval loss 0.252 467 MB
3dcodeverse/3dcodeverse-qwen3.8-27b-lora Qwen/Qwen3.8-27B qwen3.8-27b_lora_3dcv_text_A: 4,322 steps, final train loss ≈ 0.25 467 MB

Both adapters use LoRA rank 16, alpha 32, dropout 0, on every attention, linear-attention (in_proj_qkv, in_proj_a/b/z, out_proj), and MLP projection; peak LR 1e-4, cosine schedule, 3% warmup, 1 epoch, unmodified LLaMA-Factory with PEFT 0.18.1. Only the adapters are published. The base models load from Qwen's repositories.

The weights are released under CC BY-NC-SA 4.0, matching the 3dcodeverse/3dcodeverse-2m data license. The base models are Apache 2.0.

Which runs were released

The working repository williamium/3dcodeverse holds six adapters. One adapter per base model was chosen. The choice used training logs only; no execution-based scores existed for these runs:

run data steps loss released
qwen3.6-27b_lora_3dcv_text_A 3dcv_text_all (full mix) 5,515 eval 0.2521 yes
qwen3.6-27b_lora_3dcv_text_B 3dcv_text_B (subset) 2,389 eval 0.3097 no
qwen3.6-27b_lora_3dcv_text_C 3dcv_text_run1 (smaller subset) 624 eval 0.3068 no
qwen3.8-27b_lora_3dcv_text_A full mix 4,322 train ≈ 0.248 (last 50 logs) yes
qwen3.8-27b_lora_3dcv_text_A_v3 same data as A 4,322 train ≈ 0.272 (last 50 logs) no
qwen3.8-27b_lora_3dcv_text_B subset B 2,492 — no

Eval losses of different data subsets are measured on different held-out sets and are not directly comparable. A was chosen because it saw the most distinct programs, which is the main lever in REPORT.md §2.1. Run ../../eval/llm/ before stating benchmark numbers for these adapters.

Using them

Load with PEFT on top of the base checkpoint (Qwen3_5ForConditionalGeneration; the adapter only touches model.language_model):

from transformers import AutoModelForImageTextToText
from peft import PeftModel
base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "3dcodeverse/3dcodeverse-qwen3.6-27b-lora")

For evaluation or serving, merge first. configs/lf/export_lora_template.yaml works with Hub ids in place of local paths (model_name_or_path: Qwen/Qwen3.6-27B, adapter_name_or_path: 3dcodeverse/3dcodeverse-qwen3.6-27b-lora); then evaluate the merged directory with python -m llm.run_eval --backend vllm --model <merged> from eval/. Lower vLLM max_num_seqs if the engine fails to allocate Mamba cache blocks (see REPORT.md §6).

Prompts follow the training records: a dialect-specific system prompt plus a one-turn description, answered by one fenced code block, with thinking disabled. The model cards on the Hub include a full example.

Re-publishing

The card text, loss plots, and the upload were produced from a local staging folder; the logo files are ../../assets/wordmark.png and ../../assets/logo.png. To update a card, edit README.md in the Hub repo directly. The adapter weights should not change after release.