Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

fix alg_ext torch compile issue (#1540)

Wenhua Cheng9d54ddcc07a6
diff --git a/README.md b/README.md
index ba7dfecf8..81d172a57 100644
--- a/README.md
+++ b/README.md
@@ -37,6 +37,8 @@ See our papers [SignRoundV1](https://arxiv.org/pdf/2309.05516) and [SignRoundV2]
## 🆕 What's New
+* [2026/03] MTP layer quantization has been supported in [PR](https://github.com/intel/auto-round/pull/1526)
+
* [2025/12] The **SignRoundV2** paper is available. Turn on `enable_alg_ext` and use the **AutoScheme** API for mixed-precision quantization to reproduce the results: [*Paper*](http://arxiv.org/abs/2512.04746), [*Notes for evaluating LLaMA models*](./docs/alg_202508.md).
* [2025/11] AutoRound has landed in **LLM-Compressor**: [*Usage*](https://github.com/vllm-project/llm-compressor/tree/main/examples/autoround/README.md), [*vLLM blog*](https://blog.vllm.ai/2025/12/09/intel-autoround-llmc.html), [*RedHat blog*](https://developers.redhat.com/articles/2025/12/09/advancing-low-bit-quantization-llms-autoround-x-llm-compressor), [*X post*](https://x.com/vllm_project/status/1998710451312771532), [*Intel blog*](https://community.intel.com/t5/Blogs/Products-and-Solutions/HPC/Advancing-Low-Bit-Quantization-for-LLMs-AutoRound-x-LLM/post/1729336), [*Linkedin*](https://www.linkedin.com/posts/vllm-project_advancing-lowbit-quantization-for-llms-activity-7404478053768441856-ru8f/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAapNW8BLnAdCAr57GOwSCJXjf76ZvOEOAg), [*微信*](https://mp.weixin.qq.com/s/l5WA-1_4ipffQN6GOH2Iqg), [*知乎*](https://zhuanlan.zhihu.com/p/1982167638315664412).
diff --git a/auto_round/compressors/base.py b/auto_round/compressors/base.py
index f2cfd8089..9d7184e40 100644
--- a/auto_round/compressors/base.py
+++ b/auto_round/compressors/base.py
@@ -14,7 +14,6 @@
import copy
import os
-import re
import sys
import time
import traceback
@@ -99,7 +98,6 @@ from auto_round.utils import (
llm_load_model,
memory_monitor,
mv_module_from_gpu,
- safe_device_move_with_meta_handling,
set_amax_for_all_moe_layers,
set_module,
to_device,
@@ -513,7 +511,20 @@ class BaseCompressor(object):
self.enable_torch_compile = enable_torch_compile
self._adjust_torch_compile(enable_torch_compile)
- self.block_forward = compile_func(block_forward, self.device) if self.enable_torch_compile else block_forward
+ if (
+ (self.act_bits < 16 and (not self.act_dynamic or self.data_type == "nvfp")) # have hooks
+ or self.enable_alg_ext # Use imatrix
+ or not self.disable_opt_rtn # Use imatrix
+ ):
+ self.block_forward = block_forward
+ else:
+ # TODO FIXME
+ # This function could not be compiled, causing a large accuracy drop when `enable_alg_ext` is used.
+ # To avoid issues, remove it in all scenarios except WOQ.
+ self.block_forward = (
+ compile_func(block_forward, self.device) if self.enable_torch_compile else block_forward
+ )
+
self._check_configs()
torch.set_printoptions(precision=3, sci_mode=True)
diff --git a/auto_round/utils/device.py b/auto_round/utils/device.py
index d605268a2..5d615cf59 100644
--- a/auto_round/utils/device.py
+++ b/auto_round/utils/device.py
@@ -1636,7 +1636,10 @@ class MemoryMonitor:
"""Log memory usage summary."""
summary = self.get_summary()
logger_method = getattr(logger, level.lower(), logger.info)
- logger_method(f"{msg} {summary}")
+ if len(msg):
+ logger_method(f"{msg} {summary}")
+ else:
+ logger_method(f"{summary}")
return summary