Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

Keep the original checkpoint key names and fix Qwen4 vLLM inference (#2327)

Wenhua Chengaba0abdd80ab
diff --git a/README.md b/README.md
index 8dc397fa2..e6a36bff9 100644
--- a/README.md
+++ b/README.md
@@ -36,44 +36,29 @@ See our papers [SignRoundV1](https://arxiv.org/pdf/2309.05516) and [SignRoundV2]
## 🆕 What's New
-* [2026/08] We experimentally support **algorithm composition** (e.g., `--algs awq,signround` or `--algs hadamard,awq,signround`) to improve accuracy [*Overview*](./docs/algorithm_combinations.md). We welcome any practical, deployable algorithms that are ready for real-world use. Feel free to submit a PR or leave a comment in Issues.
-
-* [2026/08] AutoScheme WOQ deployment on vLLM is experimentally restored, thanks to Humming Kernel: [*vLLM PR*](https://github.com/vllm-project/vllm/pull/52890), [*Example Model*](https://huggingface.co/Intel/Qwen3.8-27B-bpw2.8-AutoRound). Note: shared layers must be configured per vLLM's fusion patterns.
-* [2026/07] `torch.compile` is enabled in most scenarios to accelerate quantization, at the cost of ~10G extra RAM usage. Minor numerical differences compared with the non-compiled path are expected due to compiler optimizations. To disable it, pass `enable_torch_compile=False` to the Python API or use `--disable_torch_compile` on the CLI.
-
-* [2026/06] AutoScheme has been refined to improve accuracy for gguf format. See [AutoScheme Accuracy](./docs/auto_scheme_acc.md) for details. This enhancement incurs additional tuning cost.
+* [2026/09] We improved MoE tuning speed by more than 3× and significantly reduced RAM usage. See the [doc](./docs/moe.md) for detailed results. Please note that this optimization may introduce some regressions or issues.
-* [2026/06] AutoRound is now fully integrated into vLLM-Omni: [*vLLM blog*](https://vllm.ai/blog/2026-06-02-vllm-omni-autoround)
+* [2026/09] We now support 5/6/7-bit WOQ models in vLLM and Transformers on CUDA devices, thanks to Humming Kernel.
-* [2026/05] We provide **free** devices for calibration-free quantization; please visit [Intel Low Bit Open LLM Leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard) for more details.
+* [2026/08] We experimentally support **algorithm composition** (e.g., `--algs awq,signround` or `--algs hadamard,awq,signround`) to improve accuracy [*Overview*](./docs/algorithm_combinations.md). We welcome any practical, deployable algorithms that are ready for real-world use. Feel free to submit a PR or leave a comment in Issues.
-* [2026/05] **Model free** quantization is available, `auto-round-rtn` will now default to using the model-free approach: [Doc](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#model-free-mode).
+* [2026/06] AutoScheme has been refined to improve accuracy for gguf format. See [AutoScheme Accuracy](./docs/auto_scheme_acc.md) for details. This enhancement incurs additional tuning cost.
-* [2026/03] **Block-wise FP8** quantization is available and rtn mode is recommended. `auto-round-rtn --scheme FP8_BLOCK`.
+* [2026/06] **Community adoption update.** AutoRound is now fully integrated into vLLM-Omni [*vLLM blog*](https://vllm.ai/blog/2026-06-02-vllm-omni-autoround), following its adoption by Transformers, vLLM, and SGLang. See the [*vLLM blog*](https://vllm.ai/blog/2026-06-02-vllm-omni-autoround).
-* [2026/03] **MTP layer quantization** has been supported in this [PR](https://github.com/intel/auto-round/pull/1526)
+* [2026/05] We provide **free** devices for quantization; please visit [Intel Low Bit Open LLM Leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard) for more details.
* [2025/12] The **SignRoundV2** paper is available. Turn on `enable_alg_ext` and use the **AutoScheme** API for mixed-precision quantization to reproduce the results: [*Paper*](http://arxiv.org/abs/2512.04746), [*Notes for evaluating LLaMA models*](./docs/alg_202508.md).
-* [2025/11] AutoRound has landed in **LLM-Compressor**: [*Usage*](https://github.com/vllm-project/llm-compressor/tree/main/examples/autoround/README.md), [*vLLM blog*](https://blog.vllm.ai/2025/12/09/intel-autoround-llmc.html), [*RedHat blog*](https://developers.redhat.com/articles/2025/12/09/advancing-low-bit-quantization-llms-autoround-x-llm-compressor), [*X post*](https://x.com/vllm_project/status/1998710451312771532), [*Intel blog*](https://community.intel.com/t5/Blogs/Products-and-Solutions/HPC/Advancing-Low-Bit-Quantization-for-LLMs-AutoRound-x-LLM/post/1729336), [*Linkedin*](https://www.linkedin.com/posts/vllm-project_advancing-lowbit-quantization-for-llms-activity-7404478053768441856-ru8f/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAapNW8BLnAdCAr57GOwSCJXjf76ZvOEOAg), [*微信*](https://mp.weixin.qq.com/s/l5WA-1_4ipffQN6GOH2Iqg), [*知乎*](https://zhuanlan.zhihu.com/p/1982167638315664412).
-
-* [2025/11] An **enhanced GGUF** quantization algorithm is available via `--enable_alg_ext`: [*Accuracy*](./docs/gguf_alg_ext_acc.md).
-
-* [2025/10] AutoRound has been integrated into **SGLang**: [*Usage*](https://docs.sglang.io/advanced_features/quantization.html#using-auto-round), [*LMSYS Blog*](https://lmsys.org/blog/2025-11-13-AutoRound/), [*X post*](https://x.com/lmsysorg/status/1991977019220148650?s=20), [*Intel blog*](https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/AutoRound-Meets-SGLang-Enabling-Quantized-Model-Inference-with/post/1727196), [*Linkedin*](https://www.linkedin.com/feed/update/urn:li:activity:7397742859354857472).
-
-* [2025/10] A **mixed precision** algorithm is available to generate schemes in minutes: [*Usage*](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#autoscheme), [*Accuracy*](./docs/auto_scheme_acc.md).
-
-* [2025/09] **MXFP4** and **NVFP4** dtypes is available: [*Accuracy*](./docs/mxnv_acc.md).
+* [2025/10] A AutoScheme is available to generate mixed-bit recipes in minutes: [*Usage*](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#autoscheme), [*Accuracy*](./docs/auto_scheme_acc.md).
* [2025/08] An **improved INT2** algorithm is available via `--enable_alg_ext`: [*Accuracy*](./docs/alg_202508.md)
+*
+* [2025/09] **MXFP4** and **NVFP4** dtypes is available: [*Accuracy*](./docs/mxnv_acc.md).
* [2025/07] **GGUF** format is supported: [*Usage*](./docs/step_by_step.md#gguf-format).
-* [2025/05] AutoRound has been integrated into **vLLM**: [*Usage*](https://docs.vllm.ai/en/latest/features/quantization/inc/), [*Medium blog*](https://medium.com/@NeuralCompressor/accelerating-vllm-and-sglang-deployment-using-autoround-45fdc0b2683e), [*小红书*](https://www.xiaohongshu.com/explore/69396bc6000000000d03e473?note_flow_source=wechat&xsec_token=CB6G3F_yM99q8XfusvyRlJqm8Db4Es2k0kYIHdIUiSQ9g=).
-
-* [2025/05] AutoRound has been integrated into **Transformers**: [*Blog*](https://huggingface.co/blog/autoround).
-
* [2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy: [*Model*](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).
diff --git a/README_CN.md b/README_CN.md
index a09734492..39858c7af 100644
--- a/README_CN.md
+++ b/README_CN.md
@@ -35,6 +35,10 @@ AutoRound 是专为大语言模型(LLMs)和视觉-语言模型(VLMs)设
## 🆕 最新进展
+* [2026/09] 我们将 MoE量化速度提升了 3 倍以上,并显著降低了 RAM 使用量。详细结果请参阅[文档](./docs/moe.md)。请注意,该优化可能会引入一些回归问题或其他问题。
+
+* [2026/09] 现在支持在 CUDA 设备上通过 vLLM 和 Transformers 使用 5/6/7-bit WOQ 模型,感谢 Humming Kernel 的支持。
+
* [2026/08] 我们实验性地支持**算法组合**(例如 `--algs awq,signround` 或 `--algs hadamard,awq,signround`),以提升精度:[*总览*](./docs/algorithm_combinations_CN.md). 我们欢迎能真正落地的任何算法组合,欢迎提交 PR 或在 Issues 中留言。
* [2026/08] 感谢 Humming Kernel,AutoScheme WOQ 在 vLLM 上的部署已实验性恢复:[*vLLM PR*](https://github.com/vllm-project/vllm/pull/52890),[*示例模型*](https://huggingface.co/Intel/Qwen3.8-27B-bpw2.8-AutoRound)。注意:共享层需按 vLLM 的融合模式进行配置。
diff --git a/auto_round/cli/main.py b/auto_round/cli/main.py
index 8441bf5bb..de5f778b7 100644
--- a/auto_round/cli/main.py
+++ b/auto_round/cli/main.py
@@ -254,6 +254,13 @@ def tune(args):
if args.eval_bs is None:
args.eval_bs = "auto"
+ if getattr(args, "num_hidden_layers", None) is not None:
+ # Debug helper: load only the first N decoder layers. Propagated to the
+ # model loader via an env var so every load path picks it up.
+ from auto_round import envs
+
+ envs.set_config(AR_DEBUG_LAYER_NUM=args.num_hidden_layers)
+
from transformers.utils.versions import require_version
if args.tasks is not None:
diff --git a/auto_round/cli/parser.py b/auto_round/cli/parser.py
index 40f0f4559..68e360d19 100644
--- a/auto_round/cli/parser.py
+++ b/auto_round/cli/parser.py
@@ -107,6 +107,15 @@ def build_quantize_parser(*, prog: str = "auto_round quantize") -> argparse.Argu
)
rt.add_argument("--model_dtype", default=None, help="Model dtype used when loading the model.")
rt.add_argument("--platform", default="hf", help="Model loading platform. Options: hf or model_scope.")
+ rt.add_argument(
+ "--num_hidden_layers",
+ "--debug_layer_num",
+ default=None,
+ type=int,
+ help="Debug only: load only the first N decoder layers of the model. Useful for isolating and "
+ "debugging issues on very large models with a fraction of the load time and memory. The resulting "
+ "model is partial and must not be used for a real quantization run.",
+ )
rt.add_argument(
"--batch_size", "--train_bs", "--bs", default=None, type=int, help="Batch size for calibration and tuning."
)
diff --git a/auto_round/compressors/base.py b/auto_round/compressors/base.py
index 788bbfec2..bda4df32d 100644
--- a/auto_round/compressors/base.py
+++ b/auto_round/compressors/base.py
@@ -66,12 +66,14 @@ from auto_round.utils import (
find_matching_blocks,
get_block_names,
get_reverse_checkpoint_conversion_mapping,
+ get_reverse_weight_transforms,
is_debug_mode,
is_hpex_available,
is_quantized_input_module,
memory_monitor,
preserve_original_visual_block_name,
revert_checkpoint_conversion_mapping,
+ revert_name_with_weight_transforms,
)
from auto_round.utils.device import (
_force_trim_malloc,
@@ -1235,6 +1237,8 @@ class BaseOrchestrator(object):
RTN and optimized RTN quantize each layer in a single pass, and very short
SignRound runs (``iters < MIN_ITERS_FOR_TORCH_COMPILE``) finish before the
compilation cost is amortized, so ``torch.compile`` only adds overhead there.
+ The short-iters rule is skipped for MoE models, whose many expert linears reuse
+ the same compiled quant function, amortizing compilation even at small iters.
This only adjusts the *default*: when the user explicitly passed
``enable_torch_compile``, their choice is always honored. Pass
@@ -1263,7 +1267,16 @@ class BaseOrchestrator(object):
iters = getattr(quantize_config, "iters", None)
if iters is not None and iters < MIN_ITERS_FOR_TORCH_COMPILE:
- return f"`iters`={iters} is below {MIN_ITERS_FOR_TORCH_COMPILE}"
+ # MoE models reuse the same compiled quant function across a large number
+ # of expert linears, so the one-time compilation cost is amortized even for
+ # very short SignRound runs. Skip the low-iters block only for MoE.
+ model = getattr(getattr(self, "model_context", None), "model", None)
+ if model is None:
+ model = getattr(self, "model", None)
+ from auto_round.utils.model import is_moe_model
+
+ if model is None or not is_moe_model(model):
+ return f"`iters`={iters} is below {MIN_ITERS_FOR_TORCH_COMPILE}"
return None
@@ -1975,22 +1988,27 @@ class BaseOrchestrator(object):
if isinstance(original_to_quant_block_names, list):
original_to_quant_block_names = original_to_quant_block_names[:]
- # to match the original name
+ # to match the original name. Prefer transformers' scope-aware reverse
+ # transforms (they honour each transform's scope / anchors) so a text
+ # sub-model prefix rule cannot double ``language_model`` or nest the
+ # sibling vision tower; fall back to the flattened regex mapping.
+ reverse_weight_transforms = get_reverse_weight_transforms(self.model)
reverse_checkpoint_conversion_mapping = get_reverse_checkpoint_conversion_mapping(self.model)
+ def _revert_block_name(block_name):
+ if reverse_weight_transforms is not None:
+ return revert_name_with_weight_transforms(block_name, reverse_weight_transforms)
+ return revert_checkpoint_conversion_mapping(block_name, reverse_checkpoint_conversion_mapping)
+
if isinstance(serialization_dict["to_quant_block_names"], str):
- reverted_block_name = revert_checkpoint_conversion_mapping(
- serialization_dict["to_quant_block_names"], reverse_checkpoint_conversion_mapping
- )
+ reverted_block_name = _revert_block_name(serialization_dict["to_quant_block_names"])
serialization_dict["to_quant_block_names"] = preserve_original_visual_block_name(
original_to_quant_block_names, reverted_block_name
)
elif isinstance(serialization_dict["to_quant_block_names"], list):
for idx in range(len(serialization_dict["to_quant_block_names"])):
- reverted_block_name = revert_checkpoint_conversion_mapping(
- serialization_dict["to_quant_block_names"][idx], reverse_checkpoint_conversion_mapping
- )
+ reverted_block_name = _revert_block_name(serialization_dict["to_quant_block_names"][idx])
original_block_name = None
if isinstance(original_to_quant_block_names, list) and idx < len(original_to_quant_block_names):
original_block_name = original_to_quant_block_names[idx]
diff --git a/auto_round/compressors/shard_writer.py b/auto_round/compressors/shard_writer.py
index 3aaf72058..6232b3d88 100644
--- a/auto_round/compressors/shard_writer.py
+++ b/auto_round/compressors/shard_writer.py
@@ -30,7 +30,9 @@ from auto_round.utils import (
get_lm_head_name,
get_module,
get_reverse_checkpoint_conversion_mapping,
+ get_reverse_weight_transforms,
revert_checkpoint_conversion_mapping,
+ revert_name_with_weight_transforms,
)
DEFAULT_MAX_SHARD_SIZE = "5GB"
@@ -75,6 +77,13 @@ class ShardWriter:
self.shard_meta = [] # List of {tmp_file: str, params: list}
self.global_weight_map = {}
self.shard_counter = 0
+ # Prefer transformers' own scope-aware reverse transforms (only attached to
+ # ``from_pretrained`` models). They revert a parameter name exactly the way
+ # transformers would when saving, honouring each transform's scope / anchors
+ # so a text-model prefix rule cannot leak onto a sibling vision tower or
+ # double-apply on an already-prefixed key. Fall back to the flattened regex
+ # mapping for models built from config (no ``_weight_conversions``).
+ self.reverse_weight_transforms = get_reverse_weight_transforms(self.model)
self.reverse_checkpoint_conversion_mapping = get_reverse_checkpoint_conversion_mapping(self.model)
# Persistent set of all parameter names already flushed to a shard file.
@@ -257,6 +266,28 @@ class ShardWriter:
return None
return list(expanded.items())
+ def _split_merged_concat(self, name: str, tensor: torch.Tensor) -> list[tuple[str, torch.Tensor]] | None:
+ """Split a merged model-side concat param back into its checkpoint shards.
+
+ Some families store a parameter as several numbered shards on disk and
+ concatenate them into one model-side tensor on load (e.g. Qwen3-Next
+ "Flash" PLE n-gram embeddings: ``...ngram_embedding.shard_<i>.weight`` ->
+ ``...ngram_embedding.weight``). ``save_pretrained`` reverses this
+ automatically; this immediate-saving path must replay the inverse so the
+ checkpoint keeps its original shards instead of one unusable blob.
+
+ Returns a list of ``(shard_name, shard_tensor)`` pairs, or ``None`` when
+ *name* is not such a merged concat parameter.
+ """
+ from auto_round.utils.disk_stream_util import split_merged_concat_tensor
+
+ config = getattr(self.model, "config", None)
+ try:
+ return split_merged_concat_tensor(config, name, tensor)
+ except Exception as e: # pragma: no cover - never break saving on a split attempt
+ logger.warning("Failed to split merged concat tensor '%s' (%s); saving it as-is.", name, e)
+ return None
+
def _add_tensor(self, name: str, tensor: torch.Tensor):
if is_attention_calibration_tensor_name(name):
return
@@ -278,8 +309,23 @@ class ShardWriter:
self._add_tensor(sub_name, sub_tensor)
return
+ # Split a merged model-side concat parameter (e.g. Qwen3-Next "Flash" PLE
+ # n-gram embedding, merged from ``shard_<i>.weight`` on load) back into its
+ # numbered checkpoint shards. ``save_pretrained`` does this automatically,
+ # but this immediate-saving path bypasses it, so replay the inverse here --
+ # otherwise a single, unusable ``[sum_rows, dim]`` blob is written.
+ split = self._split_merged_concat(name, tensor)
+ if split is not None:
+ self._all_saved.add(name)
+ for sub_name, sub_tensor in split:
+ self._add_tensor(sub_name, sub_tensor)
+ return
+
# transformers will handle _checkpoint_conversion_mapping automatically if is_immediate_saving=False
- name = revert_checkpoint_conversion_mapping(name, self.reverse_checkpoint_conversion_mapping)
+ if self.reverse_weight_transforms is not None:
+ name = revert_name_with_weight_transforms(name, self.reverse_weight_transforms)
+ else:
+ name = revert_checkpoint_conversion_mapping(name, self.reverse_checkpoint_conversion_mapping)
t_size = tensor.nbytes
self.total_param_elems += tensor.numel()
diff --git a/auto_round/context/model.py b/auto_round/context/model.py
index 63657b526..017b67ba5 100644
--- a/auto_round/context/model.py
+++ b/auto_round/context/model.py
@@ -31,6 +31,7 @@ from auto_round.special_model_handler import _handle_special_model, update_modul
from auto_round.utils import (
check_and_mark_quantized_module,
diffusion_load_model,
+ install_debug_layer_config_patch,
is_diffusion_model,
is_mllm_model,
is_moe_model,
@@ -156,6 +157,10 @@ class ModelContext(BaseContext):
device_manager.device = value
def _load_model(self):
+ # Debug helper: when AR_DEBUG_LAYER_NUM is set, patch transformers config
+ # loading so every branch below (llm / mllm / diffusion / meta skeleton)
+ # loads only the first N decoder layers. No-op otherwise.
+ install_debug_layer_config_patch()
if is_diffusion_model(self.model):
self.is_diffusion = True
self.preloaded_diffusion_pipeline = not isinstance(self.model, str)
@@ -299,11 +304,16 @@ class ModelContext(BaseContext):
materialize one block at a time from the checkpoint. Every other model (dense, or
a MoE that already ships as ``ModuleList`` of ``Linear``) has nothing to gain and
keeps the ordinary load path. ``AR_DISK_STREAM_MODEL=1`` forces this on for any
- model; ``AR_DISABLE_AUTO_META_LOAD=1`` turns the automatic choice off.
+ model; ``AR_DISABLE_META_LOAD=1`` turns the automatic choice off.
"""
if envs.AR_DISK_STREAM_MODEL:
return True
- if envs.AR_DISABLE_AUTO_META_LOAD:
+ if envs.AR_DISABLE_META_LOAD:
+ return False
+ # The debug "load only N layers" path (AR_DEBUG_LAYER_NUM) relies on the
+ # normal from_pretrained load so the truncated config is honored; the meta
+ # skeleton streams the full checkpoint block-by-block and would ignore it.
+ if envs.AR_DEBUG_LAYER_NUM is not None:
return False
if not isinstance(self.model, str):
return False
@@ -331,7 +341,7 @@ class ModelContext(BaseContext):
self.disk_stream_model_dir = checkpoint_dir
logger.info(
"Fused-MoE checkpoint detected: building a meta skeleton and materializing weights per block "
- "(set `AR_DISABLE_AUTO_META_LOAD=1` to load the whole model on CPU instead)."
+ "(set `AR_DISABLE_META_LOAD=1` to load the whole model on CPU instead)."
)
return True
diff --git a/auto_round/envs.py b/auto_round/envs.py
index d0d18da0a..6ffedd38b 100644
--- a/auto_round/envs.py
+++ b/auto_round/envs.py
@@ -32,7 +32,7 @@ if TYPE_CHECKING:
AR_ENABLE_AUTO_SCHEME_PARALLEL: bool = True
AR_NVFP4_E5M3_CACHE_HP_WEIGHT: bool = False
AR_DISK_STREAM_MODEL: bool = False
- AR_DISABLE_AUTO_META_LOAD: bool = False
+ AR_DISABLE_META_LOAD: bool = False
AR_RESUME_DIR: Optional[str] = None
AR_FORCE_MOE_ROUTING_ALL_EXPERTS: bool = False
AR_NVFP4_FUSED_LAYER_GLOBAL_SCALE: bool = True
@@ -131,7 +131,7 @@ environment_variables: dict[str, Callable[[], Any]] = {
# (fused 3D experts must be split into per-expert Linear to be quantizable, and doing
# that on real tensors costs ~2x one experts module on top of a fully resident model).
# Set this to fall back to the old behavior of loading the whole model on CPU first.
- "AR_DISABLE_AUTO_META_LOAD": lambda: os.getenv("AR_DISABLE_AUTO_META_LOAD", "0").lower() in ("1", "true", "yes"),
+ "AR_DISABLE_META_LOAD": lambda: os.getenv("AR_DISABLE_META_LOAD", "0").lower() in ("1", "true", "yes"),
# When set to a directory path, the per-block tuning loop checkpoints its
# progress there after each completed block, and resumes from the first
# not-yet-completed block on a fresh run against the same directory --
@@ -198,6 +198,13 @@ environment_variables: dict[str, Callable[[], Any]] = {
"AR_NVFP4_FUSED_LAYER_GLOBAL_SCALE": lambda: os.getenv("AR_NVFP4_FUSED_LAYER_GLOBAL_SCALE", "1").lower()
not in ("0", "false", "no", "off"),
"AR_ALLOW_W8_ASYM": lambda: os.getenv("AR_ALLOW_W8_ASYM", "0").lower() in ("1", "true", "yes"),
+ # Debug helper: when set to a positive integer N, models are loaded with only
+ # the first N decoder layers (config.num_hidden_layers is truncated to N before
+ # the weights are instantiated). This makes it possible to isolate and debug
+ # issues on very large models with a fraction of the load time and memory.
+ # Exposed on the CLI as ``--num_hidden_layers``. The resulting model is a
+ # partial model and must not be used for a real/production quantization run.
+ "AR_DEBUG_LAYER_NUM": lambda: _get_optional_positive_int_env("AR_DEBUG_LAYER_NUM"),
}
diff --git a/auto_round/export/utils.py b/auto_round/export/utils.py
index d62192999..c1aa42205 100644
--- a/auto_round/export/utils.py
+++ b/auto_round/export/utils.py
@@ -225,6 +225,75 @@ def resolve_pipeline_export_layout(model: nn.Module, output_dir: str) -> tuple[s
return model_output_dir, processor_output_dir, True
+def _load_source_config_dict(source_dir: str) -> dict | None:
+ """Load the source checkpoint's ``config.json`` as a dict (local dir or HF repo id)."""
+ if not isinstance(source_dir, str) or not source_dir:
+ return None
+ config_path = None
+ if os.path.isdir(source_dir):
+ candidate = os.path.join(source_dir, "config.json")
+ config_path = candidate if os.path.exists(candidate) else None
+ else:
+ try:
+ from huggingface_hub import hf_hub_download
+
+ config_path = hf_hub_download(source_dir, "config.json")
+ except Exception:
+ config_path = None
+ if config_path is None or not os.path.exists(config_path):
+ return None
+ with open(config_path, "r", encoding="utf-8") as f:
+ return json.load(f)
+
+
+def _restore_original_layer_types(save_dir: str, source_dir: str) -> None:
+ """Restore the original ``layer_types`` from the source checkpoint into the saved config.
+
+ transformers normalizes ``layer_types`` (e.g. legacy name remapping) when a config is
+ loaded and re-saved. For hybrid-attention models this rewrites the strings the user
+ authored in their checkpoint. This copies the original ``layer_types`` back so the
+ exported ``config.json`` matches the source. Both the top-level config and a nested
+ ``text_config`` (VL models) are handled. The value is only restored when it differs and
+ the number of entries matches, to avoid tripping transformers' length validation.
+ """
+ saved_path = os.path.join(save_dir, "config.json")
+ if not os.path.exists(saved_path):
+ return
+ source_config = _load_source_config_dict(source_dir)
+ if source_config is None:
+ return
+
+ with open(saved_path, "r", encoding="utf-8") as f:
+ saved_config = json.load(f)
+
+ changed = False
+ for src_scope, dst_scope, scope_name in (
+ (source_config, saved_config, "config"),
+ (source_config.get("text_config"), saved_config.get("text_config"), "text_config"),
+ ):
+ if not isinstance(src_scope, dict) or not isinstance(dst_scope, dict):
+ continue
+ original = src_scope.get("layer_types")
+ current = dst_scope.get("layer_types")
+ if not isinstance(original, list) or original == current:
+ continue
+ if isinstance(current, list) and len(current) != len(original):
+ logger.warning(
+ "Not restoring original layer_types for %s: source has %d entries but exported has %d.",
+ scope_name,
+ len(original),
+ len(current),
+ )
+ continue
+ dst_scope["layer_types"] = original
+ changed = True
+ logger.info("Restored original layer_types from source checkpoint for %s.", scope_name)
+
+ if changed:
+ with open(saved_path, "w", encoding="utf-8") as f:
+ json.dump(saved_config, f, indent=2)
+
+
def save_model(
model: nn.Module,
save_dir: str,
@@ -299,6 +368,16 @@ def save_model(
with open(config_path, "w") as file:
json.dump(data, file, indent=2)
+ # transformers' PreTrainedConfig normalizes ``layer_types`` on load/save (via
+ # ``remap_legacy_layer_types`` and dataclass post-init), so ``model.save_pretrained``
+ # can rewrite the strings (e.g. hybrid-attention Qwen models). Restore the original
+ # ``layer_types`` from the source checkpoint so the exported config stays faithful.
+ if source_dir is not None:
+ try:
+ _restore_original_layer_types(save_dir, source_dir)
+ except Exception as e: # pragma: no cover - best-effort, never block export
+ logger.warning("Skipping restore of original layer_types due to error: %s", e)
+
config_file = "quantization_config.json"
if hasattr(model, "config") and hasattr(model.config, "quantization_config"):
with open(os.path.join(save_dir, config_file), "w", encoding="utf-8") as f:
diff --git a/auto_round/modeling/fused_moe/replace_modules.py b/auto_round/modeling/fused_moe/replace_modules.py
index 3896402c2..f40867bb8 100644
--- a/auto_round/modeling/fused_moe/replace_modules.py
+++ b/auto_round/modeling/fused_moe/replace_modules.py
@@ -184,9 +184,9 @@ def materialize_model_(model: torch.nn.Module) -> None:
preview = ", ".join(still_meta[:8]) + (f" (+{len(still_meta) - 8} more)" if len(still_meta) > 8 else "")
# The auto meta-skeleton is only enabled for fused-MoE checkpoints; an explicit
# AR_DISK_STREAM_MODEL=1 forces it for anything else. Point at whichever toggle
- # actually turned it on so the hint stays correct (AR_DISABLE_AUTO_META_LOAD is
+ # actually turned it on so the hint stays correct (AR_DISABLE_META_LOAD is
# the MoE-only auto path).
- hint = "unset AR_DISK_STREAM_MODEL" if envs.AR_DISK_STREAM_MODEL else "set AR_DISABLE_AUTO_META_LOAD=1"
+ hint = "unset AR_DISK_STREAM_MODEL" if envs.AR_DISK_STREAM_MODEL else "set AR_DISABLE_META_LOAD=1"
raise RuntimeError(
f"Failed to materialize {len(still_meta)} checkpoint tensor(s), still on meta: {preview}. "
f"To load the whole model on CPU instead, {hint} (uses more RAM)."
diff --git a/auto_round/special_model_handler.py b/auto_round/special_model_handler.py
index 73750a8de..ba344a39d 100644
--- a/auto_round/special_model_handler.py
+++ b/auto_round/special_model_handler.py
@@ -1087,7 +1087,12 @@ register_ignore_layers(
matchers=[
ModelTypeMatcher(r"glm5_next", mode="full"),
],
- ignore_layers=[get_glm_flash_ignore_layers, "weights_proj", "indexer"], # get_glm_flash_ignore_layers: vllm issue
+ ignore_layers=[
+ get_glm_flash_ignore_layers, # vllm issue
+ "weights_proj",
+ "indexer",
+ "self_attn",
+ ],
)
# step3p5
@@ -1114,6 +1119,18 @@ register_ignore_layers(
],
)
+# qwen4: keep hyper_connection and MoE gate modules in full precision.
+register_ignore_layers(
+ matchers=[
+ ArchitectureMatcher(r"Qwen4", mode="in"),
+ ],
+ ignore_layers=[
+ "hyper_connection",
+ "mlp.gate", # MoE router gate
+ "shared_expert",
+ ],
+)
+
def get_bagel_ignore_layers(model) -> list[str]:
"""Keep BAGEL generation-path modules in FP16.
diff --git a/auto_round/utils/common.py b/auto_round/utils/common.py
index 284bfee94..075e92879 100644
--- a/auto_round/utils/common.py
+++ b/auto_round/utils/common.py
@@ -1216,16 +1216,149 @@ def get_reverse_checkpoint_conversion_mapping(model):
v: k for k, v in getattr(model, "_checkpoint_conversion_mapping", {}).items()
}
- if hasattr(model, "_weight_conversions"):
- weight_conversions = model._weight_conversions
+ weight_conversions = getattr(model, "_weight_conversions", None)
+
+ # Models built from config (e.g. AutoRound's meta / disk-stream skeleton in
+ # ``build_meta_model`` via ``model_cls(config)``) are NOT created through
+ # ``from_pretrained``, so transformers never attaches ``_weight_conversions``.
+ # Without it the per-family renames registered centrally in
+ # ``transformers/conversion_mapping.py`` (transformers >= 5.x) are lost, and
+ # module-side names such as ``attn_hc.base`` / ``self_attn.forget_gate.*``
+ # (glm5_next, deepseek_v4, ...) get written to the checkpoint verbatim instead
+ # of their original ``hc_attn_base`` / ``self_attn.*`` names, breaking reload
+ # (e.g. vLLM ``KeyError: 'layers.0.attn_hc.base'``). Fall back to the central
+ # registry and reverse those entries ourselves in that case.
+ if not weight_conversions and hasattr(transformers, "conversion_mapping"):
+ try:
+ from transformers.conversion_mapping import (
+ get_checkpoint_conversion_mapping as transformers_get_checkpoint_conversion_mapping,
+ )
+ except ImportError: # pragma: no cover - transformers < 5
+ transformers_get_checkpoint_conversion_mapping = None
+
+ config = getattr(model, "config", None)
+ if transformers_get_checkpoint_conversion_mapping is not None and config is not None:
+ # Include the text sub-model type for composite models (e.g. VLMs), where
+ # the MoE / hyper-connection renames are registered on the text config.
+ model_types = []
+ for candidate in (
+ getattr(config, "model_type", None),
+ getattr(getattr(config, "text_config", None), "model_type", None),
+ ):
+ if candidate and candidate not in model_types:
+ model_types.append(candidate)
+
+ weight_conversions = []
+ seen = set()
+ for model_type in model_types:
+ mapping = transformers_get_checkpoint_conversion_mapping(model_type)
+ if not mapping:
+ continue
+ for entry in mapping:
+ sig = (tuple(entry.source_patterns), tuple(entry.target_patterns))
+ if sig in seen:
+ continue
+ seen.add(sig)
+ weight_conversions.append(entry)
+
+ if weight_conversions:
for weight_conversion in weight_conversions:
- reverse_conversion_mapping = weight_conversion.reverse_transform()
+ try:
+ reverse_conversion_mapping = weight_conversion.reverse_transform()
+ except Exception: # pragma: no cover - not every transform is reversible (e.g. quantized converters)
+ continue
for source_pattern in reverse_conversion_mapping.source_patterns:
reverse_checkpoint_conversion_mapping[source_pattern] = reverse_conversion_mapping.target_patterns
return reverse_checkpoint_conversion_mapping
+def get_reverse_weight_transforms(model):
+ """Build transformers' scope-aware reverse weight transforms for ``model``.
+
+ Returns a ``(renamings, converters)`` tuple of reversed ``WeightTransform``
+ objects, or ``None`` when transforms are unavailable (transformers too old,
+ or nothing to revert).
+
+ This mirrors ``transformers.core_model_loading.revert_weight_conversion`` so
+ AutoRound reverts parameter names exactly the way transformers itself does
+ when saving, honouring each transform's ``scope_prefix`` / ``base_model_prefix``
+ and ``^`` anchors. Two sources are used, in order:
+
+ * ``model._weight_conversions`` -- attached only to ``from_pretrained`` models;
+ the exact, already-scoped transforms that were used at load time.
+ * ``get_model_conversion_mapping(model, add_legacy=False)`` -- for models built
+ ``from_config`` (e.g. AutoRound's meta / disk-stream skeleton), rebuilt from
+ the model's real module tree so every transform is correctly scoped. As
+ transformers does in this case, ``PrefixChange`` transforms are dropped:
+ because the model was not loaded from a checkpoint we cannot know whether a
+ prefix was present, and re-adding it corrupts keys -- doubling
+ ``language_model`` (``model.language_model.language_model.layers.*``) or
+ nesting the sibling vision tower (``model.language_model.visual.*``).
+
+ The flattened ``{source: target}`` regex path in
+ :func:`revert_checkpoint_conversion_mapping`, by contrast, drops both the
+ ``^`` anchor and the per-sub-model scope, which is precisely what triggers the
+ prefix-doubling / vision-nesting corruption above.
+ """
+ try:
+ from transformers.core_model_loading import PrefixChange, WeightConverter, WeightRenaming
+ except Exception: # pragma: no cover - transformers < 5 has no such primitives
+ return None
+
+ weight_conversions = getattr(model, "_weight_conversions", None)
+
+ if not weight_conversions:
+ # Model not created via ``from_pretrained`` -> rebuild scoped transforms
+ # from the real module tree and drop ``PrefixChange`` (see docstring).
+ try:
+ from transformers.conversion_mapping import get_model_conversion_mapping
+
+ weight_conversions = get_model_conversion_mapping(model, add_legacy=False)
+ except Exception: # pragma: no cover - no module tree / older transformers
+ return None
+ weight_conversions = [c for c in weight_conversions if not isinstance(c, PrefixChange)]
+
+ if not weight_conversions:
+ return None
+
+ try:
+ # Mirror transformers.core_model_loading.revert_weight_conversion: reverse
+ # the conversion order first, then reverse each individual transform.
+ reversed_conversions = [conversion.reverse_transform() for conversion in list(weight_conversions)[::-1]]
+ except Exception: # pragma: no cover - not every transform is reversible
+ return None
+
+ renamings = [entry for entry in reversed_conversions if isinstance(entry, WeightRenaming)]
+ converters = [entry for entry in reversed_conversions if isinstance(entry, WeightConverter)]
+ if not renamings and not converters:
+ return None
+ return renamings, converters
+
+
+def revert_name_with_weight_transforms(name: str, transforms) -> str:
+ """Revert ``name`` to its original checkpoint form using scope-aware transforms.
+
+ ``transforms`` is the ``(renamings, converters)`` tuple returned by
+ :func:`get_reverse_weight_transforms`. Applies transformers'
+ ``rename_source_key`` (reverse mode) so the reverse honours each transform's
+ ``scope_prefix`` / ``base_model_prefix`` and ``^`` anchors, avoiding the
+ prefix-leak / double-prefix corruption of the flattened regex fallback. On
+ any failure it returns ``name`` unchanged so saving never crashes.
+ """
+ if "," in name:
+ return ",".join(revert_name_with_weight_transforms(part, transforms) for part in name.split(","))
+
+ renamings, converters = transforms
+ try:
+ from transformers.core_model_loading import rename_source_key
+
+ renamed_key, _ = rename_source_key(name, renamings, converters, reverse=True)
+ return renamed_key
+ except Exception: # pragma: no cover - defensive: never break saving on rename
+ return name
+
+
def revert_checkpoint_conversion_mapping(name: str, key_mapping: dict[str, str]) -> str:
if "," in name:
return ",".join(revert_checkpoint_conversion_mapping(part, key_mapping) for part in name.split(","))
@@ -1238,6 +1371,16 @@ def revert_checkpoint_conversion_mapping(name: str, key_mapping: dict[str, str])
# Skip stripping the capture group if the target backreferences it,
# otherwise re.subn raises "invalid group reference".
if not re.search(r"\\g?<?\d+>?", target_pattern):
+ # A capture group that matches a numeric index (e.g.
+ # ``ngram_embedding.shard_(\d+).weight``) must NOT be stripped:
+ # dropping it and applying the rule would rewrite every
+ # ``shard_0, shard_1, ... shard_N`` onto the *same* reverted name,
+ # silently collapsing hundreds of distinct tensors into one (all
+ # but the first are then dropped by the saver's dedup guard).
+ # Such a rule cannot be expressed without a backreference, so skip
+ # it entirely and keep the original (indexed) name intact.
+ if re.search(r"\([^)]*(?:\\d|\[0-9\])[^)]*\)", source_pattern):
+ continue
source_pattern = re.sub(r"\(.*\)", "", source_pattern)
# Weight-conversion reverse mappings may expose bare tensor names
diff --git a/auto_round/utils/disk_stream_util.py b/auto_round/utils/disk_stream_util.py
index f8de6614d..05fefe638 100644
--- a/auto_round/utils/disk_stream_util.py
+++ b/auto_round/utils/disk_stream_util.py
@@ -492,6 +492,128 @@ def _wildcard_concat_converters_for(model_type):
return tuple(converters)
+@lru_cache(maxsize=None)
+def _wildcard_split_converters_for(model_type):
+ """Save-side inverse of :func:`_wildcard_concat_converters_for`.
+
+ Returns the same wildcard shard-concat converters, but also surfaces the
+ ``Concatenate`` op's ``num_shards_attribute`` so the merged model-side
+ parameter can be split back into the exact number of checkpoint shards at
+ save time.
+
+ Returns a tuple of ``(target_suffix, source_suffix, concat_dim,
+ num_shards_attribute)`` tuples; ``source_suffix`` still contains the ``*``
+ shard wildcard and ``num_shards_attribute`` may be ``None`` when the op does
+ not declare one (the caller then infers the count another way).
+ """
+ if not model_type:
+ return ()
+ try:
+ from transformers.conversion_mapping import get_checkpoint_conversion_mapping
+ from transformers.core_model_loading import Concatenate, WeightConverter
+ except ImportError: # pragma: no cover - transformers < 5
+ return ()
+
+ mapping = get_checkpoint_conversion_mapping(model_type)
+ if not mapping:
+ return ()
+
+ converters = []
+ for entry in mapping:
+ if not isinstance(entry, WeightConverter):
+ continue
+ if len(entry.source_patterns) != 1 or len(entry.target_patterns) != 1:
+ continue
+ if len(entry.operations) != 1 or not isinstance(entry.operations[0], Concatenate):
+ continue
+ source = entry.source_patterns[0].rstrip("$")
+ target = entry.target_patterns[0].rstrip("$")
+ if "*" not in source or "*" in target:
+ continue
+ op = entry.operations[0]
+ converters.append((target, source, getattr(op, "dim", 0), getattr(op, "num_shards_attribute", None)))
+ return tuple(converters)
+
+
+def _config_model_types(config):
+ """Collect ``model_type`` values from a config and its ``text_config`` (if any)."""
+ model_types = []
+ for candidate in (config, getattr(config, "text_config", None)):
+ model_type = getattr(candidate, "model_type", None)
+ if model_type and model_type not in model_types:
+ model_types.append(model_type)
+ return model_types
+
+
+def _resolve_num_shards(config, num_shards_attribute):
+ """Read the shard count declared by a ``Concatenate`` converter from the config."""
+ if not num_shards_attribute:
+ return None
+ for candidate in (config, getattr(config, "text_config", None)):
+ value = getattr(candidate, num_shards_attribute, None)
+ if isinstance(value, int) and value > 0:
+ return value
+ return None
+
+
+def split_merged_concat_tensor(config, full_name: str, tensor: "torch.Tensor"):
+ """Split a merged model-side concat parameter back into its checkpoint shards.
+
+ Inverse of the load-time :func:`_assemble_sharded_tensor`. Qwen3-Next "Flash"
+ (``qwen4_exp_text``) merges the per-layer PLE n-gram embedding shards
+ (``...ngram_embedding.shard_<i>.weight``) into a single
+ ``...ngram_embedding.weight`` on load; at save time (immediate ShardWriter
+ path, which does not go through ``save_pretrained``) that merge must be
+ reversed so the checkpoint keeps its original numbered shards -- otherwise a
+ single, unusable ``[sum_rows, dim]`` blob is written and reload fails.
+
+ Args:
+ config: The model config (used for ``model_type`` and the shard count).
+ full_name: The model-side parameter name (e.g. ``....ngram_embedding.weight``).
+ tensor: The merged tensor.
+
+ Returns:
+ A list of ``(shard_name, shard_tensor)`` pairs, or ``None`` when
+ *full_name* is not a wildcard-concat target (nothing to split).
+ """
+ if config is None:
+ return None
+ for model_type in _config_model_types(config):
+ for target_suffix, source_suffix, dim, num_shards_attribute in _wildcard_split_converters_for(model_type):
+ if not full_name.endswith(target_suffix):
+ continue
+ prefix = full_name[: -len(target_suffix)]
+ # Require a clean path boundary so a suffix does not match mid-token.
+ if prefix and not prefix.endswith("."):
+ continue
+ num_shards = _resolve_num_shards(config, num_shards_attribute)
+ if not num_shards:
+ # Fall back to an even split by the merged size if the row count is
+ # divisible by a plausible shard count declared elsewhere; without a
+ # count we cannot safely reconstruct shards, so leave the tensor as-is.
+ logger.warning(
+ "Cannot split merged concat tensor %r: shard count "
+ "(config.%s) is unavailable; leaving it merged.",
+ full_name,
+ num_shards_attribute,
+ )
+ return None
+ if tensor.shape[dim] % num_shards != 0:
+ logger.warning(
+ "Cannot evenly split merged concat tensor %r (dim %d size %d) "
+ "into %d shards; leaving it merged.",
+ full_name,
+ dim,
+ tensor.shape[dim],
+ num_shards,
+ )
+ return None
+ chunks = torch.chunk(tensor, num_shards, dim=dim)
+ source_prefix = prefix + source_suffix # still contains the '*' wildcard
+ return [(source_prefix.replace("*", str(i)), chunk) for i, chunk in enumerate(chunks)]
+ return None
+
+
def _assemble_sharded_tensor(index, full_name: str, device: str):
"""Reconstruct a fused model-side param stored on disk as numbered shards.
@@ -833,7 +955,7 @@ def materialize_module(module: nn.Module, module_name: str, index: SafetensorsIn
raise RuntimeError(
f"Failed to materialize fused MoE parameter '{module_name}.{name}' from the checkpoint "
f"(value shape {tuple(value.shape)}). This usually means the model's fused-expert layout "
- "is not yet recognised by the meta-skeleton loader. Set AR_DISABLE_AUTO_META_LOAD=1 to load "
+ "is not yet recognised by the meta-skeleton loader. Set AR_DISABLE_META_LOAD=1 to load "
"the whole model on CPU instead (uses more RAM)."
) from exc
_fused_cache.clear()
diff --git a/auto_round/utils/model.py b/auto_round/utils/model.py
index 010b3669e..35fd1fe5b 100644
--- a/auto_round/utils/model.py
+++ b/auto_round/utils/model.py
@@ -353,6 +353,89 @@ def _is_fp8_model(model_path, trust_remote_code=True):
return quant_method == "fp8" and model_type in _FP8_SUPPORTED_MODEL_TYPES
+def _maybe_truncate_debug_layers(config) -> bool:
+ """Debug helper: truncate the number of decoder layers on a config in place.
+
+ Controlled by the ``AR_DEBUG_LAYER_NUM`` env var (exposed on the CLI as
+ ``--num_hidden_layers``). When set to a positive integer N, only the first N
+ decoder layers are kept so that very large models can be loaded quickly with
+ a fraction of the memory for isolating and debugging issues.
+
+ Handles both flat configs (``num_hidden_layers``) and the nested sub-configs
+ used by multimodal models (e.g. ``text_config`` on a Qwen*-VL config). Every
+ reachable ``PretrainedConfig`` that exposes ``num_hidden_layers`` is truncated.
+
+ Returns ``True`` if the config was modified.
+ """
+ debug_layer_num = envs.AR_DEBUG_LAYER_NUM
+ if debug_layer_num is None or config is None:
+ return False
+
+ # Collect the config itself plus any nested sub-config it carries. Multimodal
+ # models keep the decoder layer count on a sub-config (text_config /
+ # thinker_config / ...), so scan the attribute dict generically instead of
+ # hard-coding one name.
+ targets = [config]
+ seen = {id(config)}
+ for value in list(getattr(config, "__dict__", {}).values()):
+ if hasattr(value, "num_hidden_layers") and id(value) not in seen:
+ targets.append(value)
+ seen.add(id(value))
+
+ changed = False
+ for cfg in targets:
+ current = getattr(cfg, "num_hidden_layers", None)
+ if isinstance(current, int) and debug_layer_num < current:
+ cfg.num_hidden_layers = debug_layer_num
+ changed = True
+
+ if changed:
+ logger.warning_once(
+ f"AR_DEBUG_LAYER_NUM={debug_layer_num} is set: loading only the first {debug_layer_num} "
+ "decoder layer(s) for debugging. The resulting model is partial and must not be used for a "
+ "real quantization run."
+ )
+ return changed
+
+
+def install_debug_layer_config_patch() -> None:
+ """Make ``AR_DEBUG_LAYER_NUM`` apply to *every* model-loading path.
+
+ The various loaders (``llm_load_model`` / ``mllm_load_model`` /
+ ``diffusion_load_model`` / the meta-skeleton builder) each let ``transformers``
+ resolve the config internally, so there is no single AutoRound call site to
+ intercept. They do all funnel through ``PretrainedConfig.from_dict`` (used by
+ ``AutoConfig.from_pretrained``, every concrete config's ``from_pretrained``,
+ and therefore every ``model.from_pretrained``), so patch that one classmethod
+ to truncate the decoder-layer count on the freshly built config.
+
+ No-op unless ``AR_DEBUG_LAYER_NUM`` is set. Idempotent.
+ """
+ if envs.AR_DEBUG_LAYER_NUM is None:
+ return
+
+ from transformers import PretrainedConfig
+
+ original = PretrainedConfig.__dict__.get("from_dict")
+ if original is None or getattr(original.__func__, "_ar_debug_patched", False):
+ return
+ original_func = original.__func__
+
+ def _patched_from_dict(cls, config_dict, **kwargs):
+ result = original_func(cls, config_dict, **kwargs)
+ # from_dict returns either the config or a (config, unused_kwargs) tuple.
+ config = result[0] if isinstance(result, tuple) else result
+ try:
+ _maybe_truncate_debug_layers(config)
+ except Exception as exc: # pragma: no cover - best-effort debug path
+ logger.debug(f"AR_DEBUG_LAYER_NUM truncation skipped for {cls}: {exc}")
+ return result
+
+ _patched_from_dict._ar_debug_patched = True
+ PretrainedConfig.from_dict = classmethod(_patched_from_dict)
+ logger.debug("Installed AR_DEBUG_LAYER_NUM config patch on PretrainedConfig.from_dict")
+
+
def llm_load_model(
pretrained_model_name_or_path: str,
platform: str = "hf",
@@ -406,6 +489,10 @@ def llm_load_model(
}
load_kwargs.update(kwargs)
+ # Debug helper: honor AR_DEBUG_LAYER_NUM (load only the first N decoder
+ # layers) on every load path via a transformers config patch.
+ install_debug_layer_config_patch()
+
if version.parse(transformers.__version__) >= version.parse("4.56.0"):
is_fp8 = _is_fp8_model(pretrained_model_name_or_path, trust_remote_code=trust_remote_code)
if is_fp8 and "quantization_config" not in load_kwargs:
diff --git a/docs/moe.md b/docs/moe.md
new file mode 100644
index 000000000..344ed11a5
--- /dev/null
+++ b/docs/moe.md
@@ -0,0 +1,13 @@
+torch 2.14 is used (<2.14 will cause torch compile exception), H200 with 140G or A100 with 80g
+x denote exceptions
+
+| Model | AR Version | Device | ram | vram | time cost | comment |
+|---------------------------------------------------|------------|----------|-------|-----------|-----------|-------------------------------------------------------------|
+| Qwen/Qwen3.6-35B-A3B | 0.15 | A100 | 25GB | 25GB | ~200m | |
+| Qwen/Qwen3.6-35B-A3B | 0.16 | A100 | 30G | 28G | ~60m | |
+| nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 | 0.15 | A100 | >124G | ❌ | ❌ | '_ExpertContainer' object has no attribute 'gate_proj' |
+| nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 | 0.16 | A100 | 21G | 40G | ~50m | vllm inference issue, probably not related to AutoRound |
+| Qwen3.8-Flash-Next 180B | 0.15 | H200 | >256G | ❌ | ❌ | group_size issue |
+| Qwen3.8-Flash-Next 180B | 0.16 | H200 | ~120G | ~100G | 9h | 95GB ngrams are on cpu, full attention layer is much slower |
+| zai-org/GLM-5.3-Flash-BF16 320B | 0.15 | H200 X 2 | 50 | (100,120) | ~8H | |
+| zai-org/GLM-5.3-Flash-BF16 320B | 0.16 | H200 X 2 | 40 | (110,120) | ~2.5h | vllm inference issue, probably not related to AutoRound |
diff --git a/test/unit/common/utils/test_utils.py b/test/unit/common/utils/test_utils.py
index 795356083..ea94ca031 100644
--- a/test/unit/common/utils/test_utils.py
+++ b/test/unit/common/utils/test_utils.py
@@ -7,6 +7,7 @@ import torch
import auto_round.utils.device as auto_round_utils
from auto_round.utils.common import (
compress_layer_names,
+ get_reverse_checkpoint_conversion_mapping,
preserve_original_visual_block_name,
revert_checkpoint_conversion_mapping,
)
@@ -110,6 +111,40 @@ def test_revert_checkpoint_conversion_mapping_handles_backreference_patterns():
assert converted == "model.layers.0.self_attn.q_proj.weight"
+def test_get_reverse_checkpoint_conversion_mapping_falls_back_to_central_registry():
+ # Models built from config (AutoRound's meta / disk-stream skeleton) are not
+ # created through ``from_pretrained`` and therefore carry no
+ # ``_weight_conversions``. The reverse map must still be recovered from
+ # transformers' central per-family registry (transformers >= 5.x), otherwise
+ # module-side names such as ``attn_hc.base`` are written to the checkpoint
+ # verbatim and break reload (e.g. vLLM ``KeyError: 'layers.0.attn_hc.base'``).
+ pytest.importorskip("transformers.conversion_mapping")
+ from transformers.conversion_mapping import get_checkpoint_conversion_mapping as _tf_mapping
+
+ if not _tf_mapping("glm5_next"):
+ pytest.skip("installed transformers has no glm5_next conversion mapping")
+
+ model = SimpleNamespace(config=SimpleNamespace(model_type="glm5_next"))
+ reverse_mapping = get_reverse_checkpoint_conversion_mapping(model)
+
+ # attn_hc.base -> hc_attn_base is the exact rename that broke vLLM deployment.
+ assert (
+ revert_checkpoint_conversion_mapping("model.language_model.layers.0.attn_hc.base", reverse_mapping)
+ == "model.language_model.layers.0.hc_attn_base"
+ )
+ assert (
+ revert_checkpoint_conversion_mapping("model.language_model.layers.3.ffn_hc.scale", reverse_mapping)
+ == "model.language_model.layers.3.hc_ffn_scale"
+ )
+ # The forget-gate submodule renames must reverse too, to avoid the next KeyError.
+ assert (
+ revert_checkpoint_conversion_mapping(
+ "model.language_model.layers.0.self_attn.forget_gate.A_log", reverse_mapping
+ )
+ == "model.language_model.layers.0.self_attn.A_log"
+ )
+
+
def test_preserve_original_visual_block_name():
# Single visual block name
assert preserve_original_visual_block_name("model.visual.blocks", "visual.blocks") == "model.visual.blocks"