PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Keep the original checkpoint key names and fix Qwen4 vLLM inference (#2327)
Wenhua Chengaba0abdd80ab
diff --git a/README.md b/README.mdindex 8dc397fa2..e6a36bff9 100644--- a/README.md+++ b/README.md@@ -36,44 +36,29 @@ See our papers [SignRoundV1](https://arxiv.org/pdf/2309.05516) and [SignRoundV2]## 🆕 What's New-* [2026/08] We experimentally support **algorithm composition** (e.g., `--algs awq,signround` or `--algs hadamard,awq,signround`) to improve accuracy [*Overview*](./docs/algorithm_combinations.md). We welcome any practical, deployable algorithms that are ready for real-world use. Feel free to submit a PR or leave a comment in Issues.--* [2026/08] AutoScheme WOQ deployment on vLLM is experimentally restored, thanks to Humming Kernel: [*vLLM PR*](https://github.com/vllm-project/vllm/pull/52890), [*Example Model*](https://huggingface.co/Intel/Qwen3.8-27B-bpw2.8-AutoRound). Note: shared layers must be configured per vLLM's fusion patterns.-* [2026/07] `torch.compile` is enabled in most scenarios to accelerate quantization, at the cost of ~10G extra RAM usage. Minor numerical differences compared with the non-compiled path are expected due to compiler optimizations. To disable it, pass `enable_torch_compile=False` to the Python API or use `--disable_torch_compile` on the CLI.--* [2026/06] AutoScheme has been refined to improve accuracy for gguf format. See [AutoScheme Accuracy](./docs/auto_scheme_acc.md) for details. This enhancement incurs additional tuning cost.+* [2026/09] We improved MoE tuning speed by more than 3× and significantly reduced RAM usage. See the [doc](./docs/moe.md) for detailed results. Please note that this optimization may introduce some regressions or issues.-* [2026/06] AutoRound is now fully integrated into vLLM-Omni: [*vLLM blog*](https://vllm.ai/blog/2026-06-02-vllm-omni-autoround)+* [2026/09] We now support 5/6/7-bit WOQ models in vLLM and Transformers on CUDA devices, thanks to Humming Kernel.-* [2026/05] We provide **free** devices for calibration-free quantization; please visit [Intel Low Bit Open LLM Leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard) for more details.+* [2026/08] We experimentally support **algorithm composition** (e.g., `--algs awq,signround` or `--algs hadamard,awq,signround`) to improve accuracy [*Overview*](./docs/algorithm_combinations.md). We welcome any practical, deployable algorithms that are ready for real-world use. Feel free to submit a PR or leave a comment in Issues.-* [2026/05] **Model free** quantization is available, `auto-round-rtn` will now default to using the model-free approach: [Doc](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#model-free-mode).+* [2026/06] AutoScheme has been refined to improve accuracy for gguf format. See [AutoScheme Accuracy](./docs/auto_scheme_acc.md) for details. This enhancement incurs additional tuning cost.-* [2026/03] **Block-wise FP8** quantization is available and rtn mode is recommended. `auto-round-rtn --scheme FP8_BLOCK`.+* [2026/06] **Community adoption update.** AutoRound is now fully integrated into vLLM-Omni [*vLLM blog*](https://vllm.ai/blog/2026-06-02-vllm-omni-autoround), following its adoption by Transformers, vLLM, and SGLang. See the [*vLLM blog*](https://vllm.ai/blog/2026-06-02-vllm-omni-autoround).-* [2026/03] **MTP layer quantization** has been supported in this [PR](https://github.com/intel/auto-round/pull/1526)+* [2026/05] We provide **free** devices for quantization; please visit [Intel Low Bit Open LLM Leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard) for more details.* [2025/12] The **SignRoundV2** paper is available. Turn on `enable_alg_ext` and use the **AutoScheme** API for mixed-precision quantization to reproduce the results: [*Paper*](http://arxiv.org/abs/2512.04746), [*Notes for evaluating LLaMA models*](./docs/alg_202508.md).-* [2025/11] AutoRound has landed in **LLM-Compressor**: [*Usage*](https://github.com/vllm-project/llm-compressor/tree/main/examples/autoround/README.md), [*vLLM blog*](https://blog.vllm.ai/2025/12/09/intel-autoround-llmc.html), [*RedHat blog*](https://developers.redhat.com/articles/2025/12/09/advancing-low-bit-quantization-llms-autoround-x-llm-compressor), [*X post*](https://x.com/vllm_project/status/1998710451312771532), [*Intel blog*](https://community.intel.com/t5/Blogs/Products-and-Solutions/HPC/Advancing-Low-Bit-Quantization-for-LLMs-AutoRound-x-LLM/post/1729336), [*Linkedin*](https://www.linkedin.com/posts/vllm-project_advancing-lowbit-quantization-for-llms-activity-7404478053768441856-ru8f/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAapNW8BLnAdCAr57GOwSCJXjf76ZvOEOAg), [*微信*](https://mp.weixin.qq.com/s/l5WA-1_4ipffQN6GOH2Iqg), [*知乎*](https://zhuanlan.zhihu.com/p/1982167638315664412).--* [2025/11] An **enhanced GGUF** quantization algorithm is available via `--enable_alg_ext`: [*Accuracy*](./docs/gguf_alg_ext_acc.md).--* [2025/10] AutoRound has been integrated into **SGLang**: [*Usage*](https://docs.sglang.io/advanced_features/quantization.html#using-auto-round), [*LMSYS Blog*](https://lmsys.org/blog/2025-11-13-AutoRound/), [*X post*](https://x.com/lmsysorg/status/1991977019220148650?s=20), [*Intel blog*](https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/AutoRound-Meets-SGLang-Enabling-Quantized-Model-Inference-with/post/1727196), [*Linkedin*](https://www.linkedin.com/feed/update/urn:li:activity:7397742859354857472).--* [2025/10] A **mixed precision** algorithm is available to generate schemes in minutes: [*Usage*](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#autoscheme), [*Accuracy*](./docs/auto_scheme_acc.md).--* [2025/09] **MXFP4** and **NVFP4** dtypes is available: [*Accuracy*](./docs/mxnv_acc.md).+* [2025/10] A AutoScheme is available to generate mixed-bit recipes in minutes: [*Usage*](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#autoscheme), [*Accuracy*](./docs/auto_scheme_acc.md).* [2025/08] An **improved INT2** algorithm is available via `--enable_alg_ext`: [*Accuracy*](./docs/alg_202508.md)+*+* [2025/09] **MXFP4** and **NVFP4** dtypes is available: [*Accuracy*](./docs/mxnv_acc.md).* [2025/07] **GGUF** format is supported: [*Usage*](./docs/step_by_step.md#gguf-format).-* [2025/05] AutoRound has been integrated into **vLLM**: [*Usage*](https://docs.vllm.ai/en/latest/features/quantization/inc/), [*Medium blog*](https://medium.com/@NeuralCompressor/accelerating-vllm-and-sglang-deployment-using-autoround-45fdc0b2683e), [*小红书*](https://www.xiaohongshu.com/explore/69396bc6000000000d03e473?note_flow_source=wechat&xsec_token=CB6G3F_yM99q8XfusvyRlJqm8Db4Es2k0kYIHdIUiSQ9g=).--* [2025/05] AutoRound has been integrated into **Transformers**: [*Blog*](https://huggingface.co/blog/autoround).-* [2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy: [*Model*](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).diff --git a/README_CN.md b/README_CN.mdindex a09734492..39858c7af 100644--- a/README_CN.md+++ b/README_CN.md@@ -35,6 +35,10 @@ AutoRound 是专为大语言模型(LLMs)和视觉-语言模型(VLMs)设## 🆕 最新进展+* [2026/09] 我们将 MoE量化速度提升了 3 倍以上,并显著降低了 RAM 使用量。详细结果请参阅[文档](./docs/moe.md)。请注意,该优化可能会引入一些回归问题或其他问题。++* [2026/09] 现在支持在 CUDA 设备上通过 vLLM 和 Transformers 使用 5/6/7-bit WOQ 模型,感谢 Humming Kernel 的支持。+* [2026/08] 我们实验性地支持**算法组合**(例如 `--algs awq,signround` 或 `--algs hadamard,awq,signround`),以提升精度:[*总览*](./docs/algorithm_combinations_CN.md). 我们欢迎能真正落地的任何算法组合,欢迎提交 PR 或在 Issues 中留言。* [2026/08] 感谢 Humming Kernel,AutoScheme WOQ 在 vLLM 上的部署已实验性恢复:[*vLLM PR*](https://github.com/vllm-project/vllm/pull/52890),[*示例模型*](https://huggingface.co/Intel/Qwen3.8-27B-bpw2.8-AutoRound)。注意:共享层需按 vLLM 的融合模式进行配置。diff --git a/auto_round/cli/main.py b/auto_round/cli/main.pyindex 8441bf5bb..de5f778b7 100644--- a/auto_round/cli/main.py+++ b/auto_round/cli/main.py@@ -254,6 +254,13 @@ def tune(args):if args.eval_bs is None:args.eval_bs = "auto"+ if getattr(args, "num_hidden_layers", None) is not None:+ # Debug helper: load only the first N decoder layers. Propagated to the+ # model loader via an env var so every load path picks it up.+ from auto_round import envs++ envs.set_config(AR_DEBUG_LAYER_NUM=args.num_hidden_layers)+from transformers.utils.versions import require_versionif args.tasks is not None:diff --git a/auto_round/cli/parser.py b/auto_round/cli/parser.pyindex 40f0f4559..68e360d19 100644--- a/auto_round/cli/parser.py+++ b/auto_round/cli/parser.py@@ -107,6 +107,15 @@ def build_quantize_parser(*, prog: str = "auto_round quantize") -> argparse.Argu)rt.add_argument("--model_dtype", default=None, help="Model dtype used when loading the model.")rt.add_argument("--platform", default="hf", help="Model loading platform. Options: hf or model_scope.")+ rt.add_argument(+ "--num_hidden_layers",+ "--debug_layer_num",+ default=None,+ type=int,+ help="Debug only: load only the first N decoder layers of the model. Useful for isolating and "+ "debugging issues on very large models with a fraction of the load time and memory. The resulting "+ "model is partial and must not be used for a real quantization run.",+ )rt.add_argument("--batch_size", "--train_bs", "--bs", default=None, type=int, help="Batch size for calibration and tuning.")diff --git a/auto_round/compressors/base.py b/auto_round/compressors/base.pyindex 788bbfec2..bda4df32d 100644--- a/auto_round/compressors/base.py+++ b/auto_round/compressors/base.py@@ -66,12 +66,14 @@ from auto_round.utils import (find_matching_blocks,get_block_names,get_reverse_checkpoint_conversion_mapping,+ get_reverse_weight_transforms,is_debug_mode,is_hpex_available,is_quantized_input_module,memory_monitor,preserve_original_visual_block_name,revert_checkpoint_conversion_mapping,+ revert_name_with_weight_transforms,)from auto_round.utils.device import (_force_trim_malloc,@@ -1235,6 +1237,8 @@ class BaseOrchestrator(object):RTN and optimized RTN quantize each layer in a single pass, and very shortSignRound runs (``iters < MIN_ITERS_FOR_TORCH_COMPILE``) finish before thecompilation cost is amortized, so ``torch.compile`` only adds overhead there.+ The short-iters rule is skipped for MoE models, whose many expert linears reuse+ the same compiled quant function, amortizing compilation even at small iters.This only adjusts the *default*: when the user explicitly passed``enable_torch_compile``, their choice is always honored. Pass@@ -1263,7 +1267,16 @@ class BaseOrchestrator(object):iters = getattr(quantize_config, "iters", None)if iters is not None and iters < MIN_ITERS_FOR_TORCH_COMPILE:- return f"`iters`={iters} is below {MIN_ITERS_FOR_TORCH_COMPILE}"+ # MoE models reuse the same compiled quant function across a large number+ # of expert linears, so the one-time compilation cost is amortized even for+ # very short SignRound runs. Skip the low-iters block only for MoE.+ model = getattr(getattr(self, "model_context", None), "model", None)+ if model is None:+ model = getattr(self, "model", None)+ from auto_round.utils.model import is_moe_model++ if model is None or not is_moe_model(model):+ return f"`iters`={iters} is below {MIN_ITERS_FOR_TORCH_COMPILE}"return None@@ -1975,22 +1988,27 @@ class BaseOrchestrator(object):if isinstance(original_to_quant_block_names, list):original_to_quant_block_names = original_to_quant_block_names[:]- # to match the original name+ # to match the original name. Prefer transformers' scope-aware reverse+ # transforms (they honour each transform's scope / anchors) so a text+ # sub-model prefix rule cannot double ``language_model`` or nest the+ # sibling vision tower; fall back to the flattened regex mapping.+ reverse_weight_transforms = get_reverse_weight_transforms(self.model)reverse_checkpoint_conversion_mapping = get_reverse_checkpoint_conversion_mapping(self.model)+ def _revert_block_name(block_name):+ if reverse_weight_transforms is not None:+ return revert_name_with_weight_transforms(block_name, reverse_weight_transforms)+ return revert_checkpoint_conversion_mapping(block_name, reverse_checkpoint_conversion_mapping)+if isinstance(serialization_dict["to_quant_block_names"], str):- reverted_block_name = revert_checkpoint_conversion_mapping(- serialization_dict["to_quant_block_names"], reverse_checkpoint_conversion_mapping- )+ reverted_block_name = _revert_block_name(serialization_dict["to_quant_block_names"])serialization_dict["to_quant_block_names"] = preserve_original_visual_block_name(original_to_quant_block_names, reverted_block_name)elif isinstance(serialization_dict["to_quant_block_names"], list):for idx in range(len(serialization_dict["to_quant_block_names"])):- reverted_block_name = revert_checkpoint_conversion_mapping(- serialization_dict["to_quant_block_names"][idx], reverse_checkpoint_conversion_mapping- )+ reverted_block_name = _revert_block_name(serialization_dict["to_quant_block_names"][idx])original_block_name = Noneif isinstance(original_to_quant_block_names, list) and idx < len(original_to_quant_block_names):original_block_name = original_to_quant_block_names[idx]diff --git a/auto_round/compressors/shard_writer.py b/auto_round/compressors/shard_writer.pyindex 3aaf72058..6232b3d88 100644--- a/auto_round/compressors/shard_writer.py+++ b/auto_round/compressors/shard_writer.py@@ -30,7 +30,9 @@ from auto_round.utils import (get_lm_head_name,get_module,get_reverse_checkpoint_conversion_mapping,+ get_reverse_weight_transforms,revert_checkpoint_conversion_mapping,+ revert_name_with_weight_transforms,)DEFAULT_MAX_SHARD_SIZE = "5GB"@@ -75,6 +77,13 @@ class ShardWriter:self.shard_meta = [] # List of {tmp_file: str, params: list}self.global_weight_map = {}self.shard_counter = 0+ # Prefer transformers' own scope-aware reverse transforms (only attached to+ # ``from_pretrained`` models). They revert a parameter name exactly the way+ # transformers would when saving, honouring each transform's scope / anchors+ # so a text-model prefix rule cannot leak onto a sibling vision tower or+ # double-apply on an already-prefixed key. Fall back to the flattened regex+ # mapping for models built from config (no ``_weight_conversions``).+ self.reverse_weight_transforms = get_reverse_weight_transforms(self.model)self.reverse_checkpoint_conversion_mapping = get_reverse_checkpoint_conversion_mapping(self.model)# Persistent set of all parameter names already flushed to a shard file.@@ -257,6 +266,28 @@ class ShardWriter:return Nonereturn list(expanded.items())+ def _split_merged_concat(self, name: str, tensor: torch.Tensor) -> list[tuple[str, torch.Tensor]] | None:+ """Split a merged model-side concat param back into its checkpoint shards.++ Some families store a parameter as several numbered shards on disk and+ concatenate them into one model-side tensor on load (e.g. Qwen3-Next+ "Flash" PLE n-gram embeddings: ``...ngram_embedding.shard_<i>.weight`` ->+ ``...ngram_embedding.weight``). ``save_pretrained`` reverses this+ automatically; this immediate-saving path must replay the inverse so the+ checkpoint keeps its original shards instead of one unusable blob.++ Returns a list of ``(shard_name, shard_tensor)`` pairs, or ``None`` when+ *name* is not such a merged concat parameter.+ """+ from auto_round.utils.disk_stream_util import split_merged_concat_tensor++ config = getattr(self.model, "config", None)+ try:+ return split_merged_concat_tensor(config, name, tensor)+ except Exception as e: # pragma: no cover - never break saving on a split attempt+ logger.warning("Failed to split merged concat tensor '%s' (%s); saving it as-is.", name, e)+ return None+def _add_tensor(self, name: str, tensor: torch.Tensor):if is_attention_calibration_tensor_name(name):return@@ -278,8 +309,23 @@ class ShardWriter:self._add_tensor(sub_name, sub_tensor)return+ # Split a merged model-side concat parameter (e.g. Qwen3-Next "Flash" PLE+ # n-gram embedding, merged from ``shard_<i>.weight`` on load) back into its+ # numbered checkpoint shards. ``save_pretrained`` does this automatically,+ # but this immediate-saving path bypasses it, so replay the inverse here --+ # otherwise a single, unusable ``[sum_rows, dim]`` blob is written.+ split = self._split_merged_concat(name, tensor)+ if split is not None:+ self._all_saved.add(name)+ for sub_name, sub_tensor in split:+ self._add_tensor(sub_name, sub_tensor)+ return+# transformers will handle _checkpoint_conversion_mapping automatically if is_immediate_saving=False- name = revert_checkpoint_conversion_mapping(name, self.reverse_checkpoint_conversion_mapping)+ if self.reverse_weight_transforms is not None:+ name = revert_name_with_weight_transforms(name, self.reverse_weight_transforms)+ else:+ name = revert_checkpoint_conversion_mapping(name, self.reverse_checkpoint_conversion_mapping)t_size = tensor.nbytesself.total_param_elems += tensor.numel()diff --git a/auto_round/context/model.py b/auto_round/context/model.pyindex 63657b526..017b67ba5 100644--- a/auto_round/context/model.py+++ b/auto_round/context/model.py@@ -31,6 +31,7 @@ from auto_round.special_model_handler import _handle_special_model, update_modulfrom auto_round.utils import (check_and_mark_quantized_module,diffusion_load_model,+ install_debug_layer_config_patch,is_diffusion_model,is_mllm_model,is_moe_model,@@ -156,6 +157,10 @@ class ModelContext(BaseContext):device_manager.device = valuedef _load_model(self):+ # Debug helper: when AR_DEBUG_LAYER_NUM is set, patch transformers config+ # loading so every branch below (llm / mllm / diffusion / meta skeleton)+ # loads only the first N decoder layers. No-op otherwise.+ install_debug_layer_config_patch()if is_diffusion_model(self.model):self.is_diffusion = Trueself.preloaded_diffusion_pipeline = not isinstance(self.model, str)@@ -299,11 +304,16 @@ class ModelContext(BaseContext):materialize one block at a time from the checkpoint. Every other model (dense, ora MoE that already ships as ``ModuleList`` of ``Linear``) has nothing to gain andkeeps the ordinary load path. ``AR_DISK_STREAM_MODEL=1`` forces this on for any- model; ``AR_DISABLE_AUTO_META_LOAD=1`` turns the automatic choice off.+ model; ``AR_DISABLE_META_LOAD=1`` turns the automatic choice off."""if envs.AR_DISK_STREAM_MODEL:return True- if envs.AR_DISABLE_AUTO_META_LOAD:+ if envs.AR_DISABLE_META_LOAD:+ return False+ # The debug "load only N layers" path (AR_DEBUG_LAYER_NUM) relies on the+ # normal from_pretrained load so the truncated config is honored; the meta+ # skeleton streams the full checkpoint block-by-block and would ignore it.+ if envs.AR_DEBUG_LAYER_NUM is not None:return Falseif not isinstance(self.model, str):return False@@ -331,7 +341,7 @@ class ModelContext(BaseContext):self.disk_stream_model_dir = checkpoint_dirlogger.info("Fused-MoE checkpoint detected: building a meta skeleton and materializing weights per block "- "(set `AR_DISABLE_AUTO_META_LOAD=1` to load the whole model on CPU instead)."+ "(set `AR_DISABLE_META_LOAD=1` to load the whole model on CPU instead).")return Truediff --git a/auto_round/envs.py b/auto_round/envs.pyindex d0d18da0a..6ffedd38b 100644--- a/auto_round/envs.py+++ b/auto_round/envs.py@@ -32,7 +32,7 @@ if TYPE_CHECKING:AR_ENABLE_AUTO_SCHEME_PARALLEL: bool = TrueAR_NVFP4_E5M3_CACHE_HP_WEIGHT: bool = FalseAR_DISK_STREAM_MODEL: bool = False- AR_DISABLE_AUTO_META_LOAD: bool = False+ AR_DISABLE_META_LOAD: bool = FalseAR_RESUME_DIR: Optional[str] = NoneAR_FORCE_MOE_ROUTING_ALL_EXPERTS: bool = FalseAR_NVFP4_FUSED_LAYER_GLOBAL_SCALE: bool = True@@ -131,7 +131,7 @@ environment_variables: dict[str, Callable[[], Any]] = {# (fused 3D experts must be split into per-expert Linear to be quantizable, and doing# that on real tensors costs ~2x one experts module on top of a fully resident model).# Set this to fall back to the old behavior of loading the whole model on CPU first.- "AR_DISABLE_AUTO_META_LOAD": lambda: os.getenv("AR_DISABLE_AUTO_META_LOAD", "0").lower() in ("1", "true", "yes"),+ "AR_DISABLE_META_LOAD": lambda: os.getenv("AR_DISABLE_META_LOAD", "0").lower() in ("1", "true", "yes"),# When set to a directory path, the per-block tuning loop checkpoints its# progress there after each completed block, and resumes from the first# not-yet-completed block on a fresh run against the same directory --@@ -198,6 +198,13 @@ environment_variables: dict[str, Callable[[], Any]] = {"AR_NVFP4_FUSED_LAYER_GLOBAL_SCALE": lambda: os.getenv("AR_NVFP4_FUSED_LAYER_GLOBAL_SCALE", "1").lower()not in ("0", "false", "no", "off"),"AR_ALLOW_W8_ASYM": lambda: os.getenv("AR_ALLOW_W8_ASYM", "0").lower() in ("1", "true", "yes"),+ # Debug helper: when set to a positive integer N, models are loaded with only+ # the first N decoder layers (config.num_hidden_layers is truncated to N before+ # the weights are instantiated). This makes it possible to isolate and debug+ # issues on very large models with a fraction of the load time and memory.+ # Exposed on the CLI as ``--num_hidden_layers``. The resulting model is a+ # partial model and must not be used for a real/production quantization run.+ "AR_DEBUG_LAYER_NUM": lambda: _get_optional_positive_int_env("AR_DEBUG_LAYER_NUM"),}diff --git a/auto_round/export/utils.py b/auto_round/export/utils.pyindex d62192999..c1aa42205 100644--- a/auto_round/export/utils.py+++ b/auto_round/export/utils.py@@ -225,6 +225,75 @@ def resolve_pipeline_export_layout(model: nn.Module, output_dir: str) -> tuple[sreturn model_output_dir, processor_output_dir, True+def _load_source_config_dict(source_dir: str) -> dict | None:+ """Load the source checkpoint's ``config.json`` as a dict (local dir or HF repo id)."""+ if not isinstance(source_dir, str) or not source_dir:+ return None+ config_path = None+ if os.path.isdir(source_dir):+ candidate = os.path.join(source_dir, "config.json")+ config_path = candidate if os.path.exists(candidate) else None+ else:+ try:+ from huggingface_hub import hf_hub_download++ config_path = hf_hub_download(source_dir, "config.json")+ except Exception:+ config_path = None+ if config_path is None or not os.path.exists(config_path):+ return None+ with open(config_path, "r", encoding="utf-8") as f:+ return json.load(f)+++def _restore_original_layer_types(save_dir: str, source_dir: str) -> None:+ """Restore the original ``layer_types`` from the source checkpoint into the saved config.++ transformers normalizes ``layer_types`` (e.g. legacy name remapping) when a config is+ loaded and re-saved. For hybrid-attention models this rewrites the strings the user+ authored in their checkpoint. This copies the original ``layer_types`` back so the+ exported ``config.json`` matches the source. Both the top-level config and a nested+ ``text_config`` (VL models) are handled. The value is only restored when it differs and+ the number of entries matches, to avoid tripping transformers' length validation.+ """+ saved_path = os.path.join(save_dir, "config.json")+ if not os.path.exists(saved_path):+ return+ source_config = _load_source_config_dict(source_dir)+ if source_config is None:+ return++ with open(saved_path, "r", encoding="utf-8") as f:+ saved_config = json.load(f)++ changed = False+ for src_scope, dst_scope, scope_name in (+ (source_config, saved_config, "config"),+ (source_config.get("text_config"), saved_config.get("text_config"), "text_config"),+ ):+ if not isinstance(src_scope, dict) or not isinstance(dst_scope, dict):+ continue+ original = src_scope.get("layer_types")+ current = dst_scope.get("layer_types")+ if not isinstance(original, list) or original == current:+ continue+ if isinstance(current, list) and len(current) != len(original):+ logger.warning(+ "Not restoring original layer_types for %s: source has %d entries but exported has %d.",+ scope_name,+ len(original),+ len(current),+ )+ continue+ dst_scope["layer_types"] = original+ changed = True+ logger.info("Restored original layer_types from source checkpoint for %s.", scope_name)++ if changed:+ with open(saved_path, "w", encoding="utf-8") as f:+ json.dump(saved_config, f, indent=2)++def save_model(model: nn.Module,save_dir: str,@@ -299,6 +368,16 @@ def save_model(with open(config_path, "w") as file:json.dump(data, file, indent=2)+ # transformers' PreTrainedConfig normalizes ``layer_types`` on load/save (via+ # ``remap_legacy_layer_types`` and dataclass post-init), so ``model.save_pretrained``+ # can rewrite the strings (e.g. hybrid-attention Qwen models). Restore the original+ # ``layer_types`` from the source checkpoint so the exported config stays faithful.+ if source_dir is not None:+ try:+ _restore_original_layer_types(save_dir, source_dir)+ except Exception as e: # pragma: no cover - best-effort, never block export+ logger.warning("Skipping restore of original layer_types due to error: %s", e)+config_file = "quantization_config.json"if hasattr(model, "config") and hasattr(model.config, "quantization_config"):with open(os.path.join(save_dir, config_file), "w", encoding="utf-8") as f:diff --git a/auto_round/modeling/fused_moe/replace_modules.py b/auto_round/modeling/fused_moe/replace_modules.pyindex 3896402c2..f40867bb8 100644--- a/auto_round/modeling/fused_moe/replace_modules.py+++ b/auto_round/modeling/fused_moe/replace_modules.py@@ -184,9 +184,9 @@ def materialize_model_(model: torch.nn.Module) -> None:preview = ", ".join(still_meta[:8]) + (f" (+{len(still_meta) - 8} more)" if len(still_meta) > 8 else "")# The auto meta-skeleton is only enabled for fused-MoE checkpoints; an explicit# AR_DISK_STREAM_MODEL=1 forces it for anything else. Point at whichever toggle- # actually turned it on so the hint stays correct (AR_DISABLE_AUTO_META_LOAD is+ # actually turned it on so the hint stays correct (AR_DISABLE_META_LOAD is# the MoE-only auto path).- hint = "unset AR_DISK_STREAM_MODEL" if envs.AR_DISK_STREAM_MODEL else "set AR_DISABLE_AUTO_META_LOAD=1"+ hint = "unset AR_DISK_STREAM_MODEL" if envs.AR_DISK_STREAM_MODEL else "set AR_DISABLE_META_LOAD=1"raise RuntimeError(f"Failed to materialize {len(still_meta)} checkpoint tensor(s), still on meta: {preview}. "f"To load the whole model on CPU instead, {hint} (uses more RAM)."diff --git a/auto_round/special_model_handler.py b/auto_round/special_model_handler.pyindex 73750a8de..ba344a39d 100644--- a/auto_round/special_model_handler.py+++ b/auto_round/special_model_handler.py@@ -1087,7 +1087,12 @@ register_ignore_layers(matchers=[ModelTypeMatcher(r"glm5_next", mode="full"),],- ignore_layers=[get_glm_flash_ignore_layers, "weights_proj", "indexer"], # get_glm_flash_ignore_layers: vllm issue+ ignore_layers=[+ get_glm_flash_ignore_layers, # vllm issue+ "weights_proj",+ "indexer",+ "self_attn",+ ],)# step3p5@@ -1114,6 +1119,18 @@ register_ignore_layers(],)+# qwen4: keep hyper_connection and MoE gate modules in full precision.+register_ignore_layers(+ matchers=[+ ArchitectureMatcher(r"Qwen4", mode="in"),+ ],+ ignore_layers=[+ "hyper_connection",+ "mlp.gate", # MoE router gate+ "shared_expert",+ ],+)+def get_bagel_ignore_layers(model) -> list[str]:"""Keep BAGEL generation-path modules in FP16.diff --git a/auto_round/utils/common.py b/auto_round/utils/common.pyindex 284bfee94..075e92879 100644--- a/auto_round/utils/common.py+++ b/auto_round/utils/common.py@@ -1216,16 +1216,149 @@ def get_reverse_checkpoint_conversion_mapping(model):v: k for k, v in getattr(model, "_checkpoint_conversion_mapping", {}).items()}- if hasattr(model, "_weight_conversions"):- weight_conversions = model._weight_conversions+ weight_conversions = getattr(model, "_weight_conversions", None)++ # Models built from config (e.g. AutoRound's meta / disk-stream skeleton in+ # ``build_meta_model`` via ``model_cls(config)``) are NOT created through+ # ``from_pretrained``, so transformers never attaches ``_weight_conversions``.+ # Without it the per-family renames registered centrally in+ # ``transformers/conversion_mapping.py`` (transformers >= 5.x) are lost, and+ # module-side names such as ``attn_hc.base`` / ``self_attn.forget_gate.*``+ # (glm5_next, deepseek_v4, ...) get written to the checkpoint verbatim instead+ # of their original ``hc_attn_base`` / ``self_attn.*`` names, breaking reload+ # (e.g. vLLM ``KeyError: 'layers.0.attn_hc.base'``). Fall back to the central+ # registry and reverse those entries ourselves in that case.+ if not weight_conversions and hasattr(transformers, "conversion_mapping"):+ try:+ from transformers.conversion_mapping import (+ get_checkpoint_conversion_mapping as transformers_get_checkpoint_conversion_mapping,+ )+ except ImportError: # pragma: no cover - transformers < 5+ transformers_get_checkpoint_conversion_mapping = None++ config = getattr(model, "config", None)+ if transformers_get_checkpoint_conversion_mapping is not None and config is not None:+ # Include the text sub-model type for composite models (e.g. VLMs), where+ # the MoE / hyper-connection renames are registered on the text config.+ model_types = []+ for candidate in (+ getattr(config, "model_type", None),+ getattr(getattr(config, "text_config", None), "model_type", None),+ ):+ if candidate and candidate not in model_types:+ model_types.append(candidate)++ weight_conversions = []+ seen = set()+ for model_type in model_types:+ mapping = transformers_get_checkpoint_conversion_mapping(model_type)+ if not mapping:+ continue+ for entry in mapping:+ sig = (tuple(entry.source_patterns), tuple(entry.target_patterns))+ if sig in seen:+ continue+ seen.add(sig)+ weight_conversions.append(entry)++ if weight_conversions:for weight_conversion in weight_conversions:- reverse_conversion_mapping = weight_conversion.reverse_transform()+ try:+ reverse_conversion_mapping = weight_conversion.reverse_transform()+ except Exception: # pragma: no cover - not every transform is reversible (e.g. quantized converters)+ continuefor source_pattern in reverse_conversion_mapping.source_patterns:reverse_checkpoint_conversion_mapping[source_pattern] = reverse_conversion_mapping.target_patternsreturn reverse_checkpoint_conversion_mapping+def get_reverse_weight_transforms(model):+ """Build transformers' scope-aware reverse weight transforms for ``model``.++ Returns a ``(renamings, converters)`` tuple of reversed ``WeightTransform``+ objects, or ``None`` when transforms are unavailable (transformers too old,+ or nothing to revert).++ This mirrors ``transformers.core_model_loading.revert_weight_conversion`` so+ AutoRound reverts parameter names exactly the way transformers itself does+ when saving, honouring each transform's ``scope_prefix`` / ``base_model_prefix``+ and ``^`` anchors. Two sources are used, in order:++ * ``model._weight_conversions`` -- attached only to ``from_pretrained`` models;+ the exact, already-scoped transforms that were used at load time.+ * ``get_model_conversion_mapping(model, add_legacy=False)`` -- for models built+ ``from_config`` (e.g. AutoRound's meta / disk-stream skeleton), rebuilt from+ the model's real module tree so every transform is correctly scoped. As+ transformers does in this case, ``PrefixChange`` transforms are dropped:+ because the model was not loaded from a checkpoint we cannot know whether a+ prefix was present, and re-adding it corrupts keys -- doubling+ ``language_model`` (``model.language_model.language_model.layers.*``) or+ nesting the sibling vision tower (``model.language_model.visual.*``).++ The flattened ``{source: target}`` regex path in+ :func:`revert_checkpoint_conversion_mapping`, by contrast, drops both the+ ``^`` anchor and the per-sub-model scope, which is precisely what triggers the+ prefix-doubling / vision-nesting corruption above.+ """+ try:+ from transformers.core_model_loading import PrefixChange, WeightConverter, WeightRenaming+ except Exception: # pragma: no cover - transformers < 5 has no such primitives+ return None++ weight_conversions = getattr(model, "_weight_conversions", None)++ if not weight_conversions:+ # Model not created via ``from_pretrained`` -> rebuild scoped transforms+ # from the real module tree and drop ``PrefixChange`` (see docstring).+ try:+ from transformers.conversion_mapping import get_model_conversion_mapping++ weight_conversions = get_model_conversion_mapping(model, add_legacy=False)+ except Exception: # pragma: no cover - no module tree / older transformers+ return None+ weight_conversions = [c for c in weight_conversions if not isinstance(c, PrefixChange)]++ if not weight_conversions:+ return None++ try:+ # Mirror transformers.core_model_loading.revert_weight_conversion: reverse+ # the conversion order first, then reverse each individual transform.+ reversed_conversions = [conversion.reverse_transform() for conversion in list(weight_conversions)[::-1]]+ except Exception: # pragma: no cover - not every transform is reversible+ return None++ renamings = [entry for entry in reversed_conversions if isinstance(entry, WeightRenaming)]+ converters = [entry for entry in reversed_conversions if isinstance(entry, WeightConverter)]+ if not renamings and not converters:+ return None+ return renamings, converters+++def revert_name_with_weight_transforms(name: str, transforms) -> str:+ """Revert ``name`` to its original checkpoint form using scope-aware transforms.++ ``transforms`` is the ``(renamings, converters)`` tuple returned by+ :func:`get_reverse_weight_transforms`. Applies transformers'+ ``rename_source_key`` (reverse mode) so the reverse honours each transform's+ ``scope_prefix`` / ``base_model_prefix`` and ``^`` anchors, avoiding the+ prefix-leak / double-prefix corruption of the flattened regex fallback. On+ any failure it returns ``name`` unchanged so saving never crashes.+ """+ if "," in name:+ return ",".join(revert_name_with_weight_transforms(part, transforms) for part in name.split(","))++ renamings, converters = transforms+ try:+ from transformers.core_model_loading import rename_source_key++ renamed_key, _ = rename_source_key(name, renamings, converters, reverse=True)+ return renamed_key+ except Exception: # pragma: no cover - defensive: never break saving on rename+ return name++def revert_checkpoint_conversion_mapping(name: str, key_mapping: dict[str, str]) -> str:if "," in name:return ",".join(revert_checkpoint_conversion_mapping(part, key_mapping) for part in name.split(","))@@ -1238,6 +1371,16 @@ def revert_checkpoint_conversion_mapping(name: str, key_mapping: dict[str, str])# Skip stripping the capture group if the target backreferences it,# otherwise re.subn raises "invalid group reference".if not re.search(r"\\g?<?\d+>?", target_pattern):+ # A capture group that matches a numeric index (e.g.+ # ``ngram_embedding.shard_(\d+).weight``) must NOT be stripped:+ # dropping it and applying the rule would rewrite every+ # ``shard_0, shard_1, ... shard_N`` onto the *same* reverted name,+ # silently collapsing hundreds of distinct tensors into one (all+ # but the first are then dropped by the saver's dedup guard).+ # Such a rule cannot be expressed without a backreference, so skip+ # it entirely and keep the original (indexed) name intact.+ if re.search(r"\([^)]*(?:\\d|\[0-9\])[^)]*\)", source_pattern):+ continuesource_pattern = re.sub(r"\(.*\)", "", source_pattern)# Weight-conversion reverse mappings may expose bare tensor namesdiff --git a/auto_round/utils/disk_stream_util.py b/auto_round/utils/disk_stream_util.pyindex f8de6614d..05fefe638 100644--- a/auto_round/utils/disk_stream_util.py+++ b/auto_round/utils/disk_stream_util.py@@ -492,6 +492,128 @@ def _wildcard_concat_converters_for(model_type):return tuple(converters)+@lru_cache(maxsize=None)+def _wildcard_split_converters_for(model_type):+ """Save-side inverse of :func:`_wildcard_concat_converters_for`.++ Returns the same wildcard shard-concat converters, but also surfaces the+ ``Concatenate`` op's ``num_shards_attribute`` so the merged model-side+ parameter can be split back into the exact number of checkpoint shards at+ save time.++ Returns a tuple of ``(target_suffix, source_suffix, concat_dim,+ num_shards_attribute)`` tuples; ``source_suffix`` still contains the ``*``+ shard wildcard and ``num_shards_attribute`` may be ``None`` when the op does+ not declare one (the caller then infers the count another way).+ """+ if not model_type:+ return ()+ try:+ from transformers.conversion_mapping import get_checkpoint_conversion_mapping+ from transformers.core_model_loading import Concatenate, WeightConverter+ except ImportError: # pragma: no cover - transformers < 5+ return ()++ mapping = get_checkpoint_conversion_mapping(model_type)+ if not mapping:+ return ()++ converters = []+ for entry in mapping:+ if not isinstance(entry, WeightConverter):+ continue+ if len(entry.source_patterns) != 1 or len(entry.target_patterns) != 1:+ continue+ if len(entry.operations) != 1 or not isinstance(entry.operations[0], Concatenate):+ continue+ source = entry.source_patterns[0].rstrip("$")+ target = entry.target_patterns[0].rstrip("$")+ if "*" not in source or "*" in target:+ continue+ op = entry.operations[0]+ converters.append((target, source, getattr(op, "dim", 0), getattr(op, "num_shards_attribute", None)))+ return tuple(converters)+++def _config_model_types(config):+ """Collect ``model_type`` values from a config and its ``text_config`` (if any)."""+ model_types = []+ for candidate in (config, getattr(config, "text_config", None)):+ model_type = getattr(candidate, "model_type", None)+ if model_type and model_type not in model_types:+ model_types.append(model_type)+ return model_types+++def _resolve_num_shards(config, num_shards_attribute):+ """Read the shard count declared by a ``Concatenate`` converter from the config."""+ if not num_shards_attribute:+ return None+ for candidate in (config, getattr(config, "text_config", None)):+ value = getattr(candidate, num_shards_attribute, None)+ if isinstance(value, int) and value > 0:+ return value+ return None+++def split_merged_concat_tensor(config, full_name: str, tensor: "torch.Tensor"):+ """Split a merged model-side concat parameter back into its checkpoint shards.++ Inverse of the load-time :func:`_assemble_sharded_tensor`. Qwen3-Next "Flash"+ (``qwen4_exp_text``) merges the per-layer PLE n-gram embedding shards+ (``...ngram_embedding.shard_<i>.weight``) into a single+ ``...ngram_embedding.weight`` on load; at save time (immediate ShardWriter+ path, which does not go through ``save_pretrained``) that merge must be+ reversed so the checkpoint keeps its original numbered shards -- otherwise a+ single, unusable ``[sum_rows, dim]`` blob is written and reload fails.++ Args:+ config: The model config (used for ``model_type`` and the shard count).+ full_name: The model-side parameter name (e.g. ``....ngram_embedding.weight``).+ tensor: The merged tensor.++ Returns:+ A list of ``(shard_name, shard_tensor)`` pairs, or ``None`` when+ *full_name* is not a wildcard-concat target (nothing to split).+ """+ if config is None:+ return None+ for model_type in _config_model_types(config):+ for target_suffix, source_suffix, dim, num_shards_attribute in _wildcard_split_converters_for(model_type):+ if not full_name.endswith(target_suffix):+ continue+ prefix = full_name[: -len(target_suffix)]+ # Require a clean path boundary so a suffix does not match mid-token.+ if prefix and not prefix.endswith("."):+ continue+ num_shards = _resolve_num_shards(config, num_shards_attribute)+ if not num_shards:+ # Fall back to an even split by the merged size if the row count is+ # divisible by a plausible shard count declared elsewhere; without a+ # count we cannot safely reconstruct shards, so leave the tensor as-is.+ logger.warning(+ "Cannot split merged concat tensor %r: shard count "+ "(config.%s) is unavailable; leaving it merged.",+ full_name,+ num_shards_attribute,+ )+ return None+ if tensor.shape[dim] % num_shards != 0:+ logger.warning(+ "Cannot evenly split merged concat tensor %r (dim %d size %d) "+ "into %d shards; leaving it merged.",+ full_name,+ dim,+ tensor.shape[dim],+ num_shards,+ )+ return None+ chunks = torch.chunk(tensor, num_shards, dim=dim)+ source_prefix = prefix + source_suffix # still contains the '*' wildcard+ return [(source_prefix.replace("*", str(i)), chunk) for i, chunk in enumerate(chunks)]+ return None++def _assemble_sharded_tensor(index, full_name: str, device: str):"""Reconstruct a fused model-side param stored on disk as numbered shards.@@ -833,7 +955,7 @@ def materialize_module(module: nn.Module, module_name: str, index: SafetensorsInraise RuntimeError(f"Failed to materialize fused MoE parameter '{module_name}.{name}' from the checkpoint "f"(value shape {tuple(value.shape)}). This usually means the model's fused-expert layout "- "is not yet recognised by the meta-skeleton loader. Set AR_DISABLE_AUTO_META_LOAD=1 to load "+ "is not yet recognised by the meta-skeleton loader. Set AR_DISABLE_META_LOAD=1 to load ""the whole model on CPU instead (uses more RAM).") from exc_fused_cache.clear()diff --git a/auto_round/utils/model.py b/auto_round/utils/model.pyindex 010b3669e..35fd1fe5b 100644--- a/auto_round/utils/model.py+++ b/auto_round/utils/model.py@@ -353,6 +353,89 @@ def _is_fp8_model(model_path, trust_remote_code=True):return quant_method == "fp8" and model_type in _FP8_SUPPORTED_MODEL_TYPES+def _maybe_truncate_debug_layers(config) -> bool:+ """Debug helper: truncate the number of decoder layers on a config in place.++ Controlled by the ``AR_DEBUG_LAYER_NUM`` env var (exposed on the CLI as+ ``--num_hidden_layers``). When set to a positive integer N, only the first N+ decoder layers are kept so that very large models can be loaded quickly with+ a fraction of the memory for isolating and debugging issues.++ Handles both flat configs (``num_hidden_layers``) and the nested sub-configs+ used by multimodal models (e.g. ``text_config`` on a Qwen*-VL config). Every+ reachable ``PretrainedConfig`` that exposes ``num_hidden_layers`` is truncated.++ Returns ``True`` if the config was modified.+ """+ debug_layer_num = envs.AR_DEBUG_LAYER_NUM+ if debug_layer_num is None or config is None:+ return False++ # Collect the config itself plus any nested sub-config it carries. Multimodal+ # models keep the decoder layer count on a sub-config (text_config /+ # thinker_config / ...), so scan the attribute dict generically instead of+ # hard-coding one name.+ targets = [config]+ seen = {id(config)}+ for value in list(getattr(config, "__dict__", {}).values()):+ if hasattr(value, "num_hidden_layers") and id(value) not in seen:+ targets.append(value)+ seen.add(id(value))++ changed = False+ for cfg in targets:+ current = getattr(cfg, "num_hidden_layers", None)+ if isinstance(current, int) and debug_layer_num < current:+ cfg.num_hidden_layers = debug_layer_num+ changed = True++ if changed:+ logger.warning_once(+ f"AR_DEBUG_LAYER_NUM={debug_layer_num} is set: loading only the first {debug_layer_num} "+ "decoder layer(s) for debugging. The resulting model is partial and must not be used for a "+ "real quantization run."+ )+ return changed+++def install_debug_layer_config_patch() -> None:+ """Make ``AR_DEBUG_LAYER_NUM`` apply to *every* model-loading path.++ The various loaders (``llm_load_model`` / ``mllm_load_model`` /+ ``diffusion_load_model`` / the meta-skeleton builder) each let ``transformers``+ resolve the config internally, so there is no single AutoRound call site to+ intercept. They do all funnel through ``PretrainedConfig.from_dict`` (used by+ ``AutoConfig.from_pretrained``, every concrete config's ``from_pretrained``,+ and therefore every ``model.from_pretrained``), so patch that one classmethod+ to truncate the decoder-layer count on the freshly built config.++ No-op unless ``AR_DEBUG_LAYER_NUM`` is set. Idempotent.+ """+ if envs.AR_DEBUG_LAYER_NUM is None:+ return++ from transformers import PretrainedConfig++ original = PretrainedConfig.__dict__.get("from_dict")+ if original is None or getattr(original.__func__, "_ar_debug_patched", False):+ return+ original_func = original.__func__++ def _patched_from_dict(cls, config_dict, **kwargs):+ result = original_func(cls, config_dict, **kwargs)+ # from_dict returns either the config or a (config, unused_kwargs) tuple.+ config = result[0] if isinstance(result, tuple) else result+ try:+ _maybe_truncate_debug_layers(config)+ except Exception as exc: # pragma: no cover - best-effort debug path+ logger.debug(f"AR_DEBUG_LAYER_NUM truncation skipped for {cls}: {exc}")+ return result++ _patched_from_dict._ar_debug_patched = True+ PretrainedConfig.from_dict = classmethod(_patched_from_dict)+ logger.debug("Installed AR_DEBUG_LAYER_NUM config patch on PretrainedConfig.from_dict")++def llm_load_model(pretrained_model_name_or_path: str,platform: str = "hf",@@ -406,6 +489,10 @@ def llm_load_model(}load_kwargs.update(kwargs)+ # Debug helper: honor AR_DEBUG_LAYER_NUM (load only the first N decoder+ # layers) on every load path via a transformers config patch.+ install_debug_layer_config_patch()+if version.parse(transformers.__version__) >= version.parse("4.56.0"):is_fp8 = _is_fp8_model(pretrained_model_name_or_path, trust_remote_code=trust_remote_code)if is_fp8 and "quantization_config" not in load_kwargs:diff --git a/docs/moe.md b/docs/moe.mdnew file mode 100644index 000000000..344ed11a5--- /dev/null+++ b/docs/moe.md@@ -0,0 +1,13 @@+torch 2.14 is used (<2.14 will cause torch compile exception), H200 with 140G or A100 with 80g+x denote exceptions++| Model | AR Version | Device | ram | vram | time cost | comment |+|---------------------------------------------------|------------|----------|-------|-----------|-----------|-------------------------------------------------------------|+| Qwen/Qwen3.6-35B-A3B | 0.15 | A100 | 25GB | 25GB | ~200m | |+| Qwen/Qwen3.6-35B-A3B | 0.16 | A100 | 30G | 28G | ~60m | |+| nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 | 0.15 | A100 | >124G | ❌ | ❌ | '_ExpertContainer' object has no attribute 'gate_proj' |+| nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 | 0.16 | A100 | 21G | 40G | ~50m | vllm inference issue, probably not related to AutoRound |+| Qwen3.8-Flash-Next 180B | 0.15 | H200 | >256G | ❌ | ❌ | group_size issue |+| Qwen3.8-Flash-Next 180B | 0.16 | H200 | ~120G | ~100G | 9h | 95GB ngrams are on cpu, full attention layer is much slower |+| zai-org/GLM-5.3-Flash-BF16 320B | 0.15 | H200 X 2 | 50 | (100,120) | ~8H | |+| zai-org/GLM-5.3-Flash-BF16 320B | 0.16 | H200 X 2 | 40 | (110,120) | ~2.5h | vllm inference issue, probably not related to AutoRound |diff --git a/test/unit/common/utils/test_utils.py b/test/unit/common/utils/test_utils.pyindex 795356083..ea94ca031 100644--- a/test/unit/common/utils/test_utils.py+++ b/test/unit/common/utils/test_utils.py@@ -7,6 +7,7 @@ import torchimport auto_round.utils.device as auto_round_utilsfrom auto_round.utils.common import (compress_layer_names,+ get_reverse_checkpoint_conversion_mapping,preserve_original_visual_block_name,revert_checkpoint_conversion_mapping,)@@ -110,6 +111,40 @@ def test_revert_checkpoint_conversion_mapping_handles_backreference_patterns():assert converted == "model.layers.0.self_attn.q_proj.weight"+def test_get_reverse_checkpoint_conversion_mapping_falls_back_to_central_registry():+ # Models built from config (AutoRound's meta / disk-stream skeleton) are not+ # created through ``from_pretrained`` and therefore carry no+ # ``_weight_conversions``. The reverse map must still be recovered from+ # transformers' central per-family registry (transformers >= 5.x), otherwise+ # module-side names such as ``attn_hc.base`` are written to the checkpoint+ # verbatim and break reload (e.g. vLLM ``KeyError: 'layers.0.attn_hc.base'``).+ pytest.importorskip("transformers.conversion_mapping")+ from transformers.conversion_mapping import get_checkpoint_conversion_mapping as _tf_mapping++ if not _tf_mapping("glm5_next"):+ pytest.skip("installed transformers has no glm5_next conversion mapping")++ model = SimpleNamespace(config=SimpleNamespace(model_type="glm5_next"))+ reverse_mapping = get_reverse_checkpoint_conversion_mapping(model)++ # attn_hc.base -> hc_attn_base is the exact rename that broke vLLM deployment.+ assert (+ revert_checkpoint_conversion_mapping("model.language_model.layers.0.attn_hc.base", reverse_mapping)+ == "model.language_model.layers.0.hc_attn_base"+ )+ assert (+ revert_checkpoint_conversion_mapping("model.language_model.layers.3.ffn_hc.scale", reverse_mapping)+ == "model.language_model.layers.3.hc_ffn_scale"+ )+ # The forget-gate submodule renames must reverse too, to avoid the next KeyError.+ assert (+ revert_checkpoint_conversion_mapping(+ "model.language_model.layers.0.self_attn.forget_gate.A_log", reverse_mapping+ )+ == "model.language_model.layers.0.self_attn.A_log"+ )++def test_preserve_original_visual_block_name():# Single visual block nameassert preserve_original_visual_block_name("model.visual.blocks", "visual.blocks") == "model.visual.blocks"