PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Enable torch.compile by default and update related documentation (#2070)
Xin Hebe862d3744a4
diff --git a/README.md b/README.mdindex ab0d90fab..984237364 100644--- a/README.md+++ b/README.md@@ -37,6 +37,8 @@ See our papers [SignRoundV1](https://arxiv.org/pdf/2309.05516) and [SignRoundV2]## 🆕 What's New+* [2026/07] `torch.compile` is enabled by default except on Windows to accelerate quantization. Minor numerical differences compared with the non-compiled path are expected due to compiler optimizations. To disable it, pass `enable_torch_compile=False` to the Python API or use `--disable_torch_compile` on the CLI.+* [2026/06] AutoScheme has been refined to improve accuracy for gguf format. See [AutoScheme Accuracy](./docs/auto_scheme_acc.md) for details. This enhancement incurs additional tuning cost.* [2026/06] AutoRound is now fully integrated into vLLM-Omni: [*vLLM blog*](https://vllm.ai/blog/2026-06-02-vllm-omni-autoround)@@ -258,7 +260,7 @@ ar.quantize_and_save(output_dir="./qmodel", format="auto_round")- **`seqlen` (int)**: Data length of the sequence for tuning (default is `2048`).##### Device/Speed Configuration-- **`enable_torch_compile` (bool)**: If no exception is raised, typically we recommend setting it to True for faster quantization with lower resource.+- **`enable_torch_compile` (bool)**: Enabling `torch.compile` may speed up quantization, but compilation overhead can change peak memory and may increase it for some models or quantization schemes. It is enabled by default except on Windows, where it is disabled because TorchInductor requires the MSVC `cl.exe` compiler. On Windows, pass `enable_torch_compile=True` in Python or `--enable_torch_compile` on the CLI to force enable it. On other platforms, use `enable_torch_compile=False` or `--disable_torch_compile` to opt out.- **`low_gpu_mem_usage` (bool)**: Whether to offload intermediate features to CPU at the cost of ~30%-100% more tuning time (default is `False`).- **`low_cpu_mem_usage` (bool)**: [Experimental Feature]Whether to enable saving immediately to reduce ram usage (default is `True`).- **`device_map` (str|dict|int)**: The device to be used for tuning, e.g., `auto`, `cpu`, `cuda`, `0,1,2` (default is `0`). When using `auto`, it will try to use all available GPUs.diff --git a/README_CN.md b/README_CN.mdindex 67a935029..677299ef1 100644--- a/README_CN.md+++ b/README_CN.md@@ -35,6 +35,7 @@ AutoRound 是专为大语言模型(LLMs)和视觉-语言模型(VLMs)设## 🆕 最新进展+* [2026/07] 除 Windows 外默认启用 `torch.compile`,以加速量化。由于编译器优化,与未编译路径相比可能会出现细微的数值差异,这是符合预期的。如需关闭,可在 Python API 中传入 `enable_torch_compile=False`,或在命令行中使用 `--disable_torch_compile`。* [2026/06] AutoScheme对gguf格式优化,精度有所提升。具体结果请参考[文档](docs/auto_scheme_acc.md)。不过这些精度提升是以更高的调优时间和计算开销为代价的。@@ -252,7 +253,7 @@ ar.quantize_and_save(output_dir="./qmodel", format="auto_round")##### 设备 / 速度配置-- **`enable_torch_compile`**(bool):通常建议设为 `True` 来提升量化速度、降低资源消耗,但是有极小概率会触发异常,建议使用最新的 tiron 版本。+- **`enable_torch_compile`**(bool):启用 `torch.compile` 可能提升量化速度,但编译开销会改变峰值内存,某些模型或量化方案的峰值内存可能增加。除 Windows 外默认开启;Windows 上默认关闭,因为 TorchInductor 需要 MSVC 的 `cl.exe` 编译器。在 Windows 上可通过 Python 参数 `enable_torch_compile=True` 或命令行参数 `--enable_torch_compile` 强制开启;其他平台可使用 `enable_torch_compile=False` 或 `--disable_torch_compile` 关闭。- **`low_gpu_mem_usage`**(bool):若要节省显存,可以设为 `True` 。它会将中间特征卸载到 CPU,但会增加 30%-100% 的时间(默认 `False`)。- **`low_cpu_mem_usage`**(bool):[实验性功能] 若要减少内存占用,可以设为 `True` 来启用即时保存(默认 `False`)。- **`device_map`**(str | dict | int):计算设备指定,如 `auto`、`cpu`、`cuda`、`0,1,2`(默认 `0`)。使用 `auto` 时会尝试利用所有可用 GPU。diff --git a/auto_round/algorithms/block_runner.py b/auto_round/algorithms/block_runner.pyindex 0e88ff3b4..60f9b87ab 100644--- a/auto_round/algorithms/block_runner.py+++ b/auto_round/algorithms/block_runner.py@@ -111,7 +111,7 @@ class BlockForwardRunner:is_diffusion: bool = False,shared_cache_keys: tuple = (),output_config: list[str] | None = None,- enable_torch_compile: bool = False,+ enable_torch_compile: bool = True,) -> None:self.batch_dim = batch_dimself.batch_size = batch_size@@ -133,7 +133,7 @@ class BlockForwardRunner:# ── Factory ──────────────────────────────────────────────────────────────@classmethod- def from_orchestrator(cls, orchestrator: "BaseOrchestrator", enable_torch_compile=False) -> "BlockForwardRunner":+ def from_orchestrator(cls, orchestrator: "BaseOrchestrator", enable_torch_compile=True) -> "BlockForwardRunner":"""Create from an orchestrator instance (called once at orchestrator init)."""model_ctx = getattr(orchestrator, "model_context", None)is_diffusion = getattr(model_ctx, "is_diffusion", False) if model_ctx else Falsediff --git a/auto_round/algorithms/composer.py b/auto_round/algorithms/composer.pyindex ce2d27d97..015224ec6 100644--- a/auto_round/algorithms/composer.py+++ b/auto_round/algorithms/composer.py@@ -83,6 +83,13 @@ class BlockContext:# ---------------------------------------------------------------------------# AlgorithmComposer# ---------------------------------------------------------------------------+def _can_compile_block_forward(block_quantizer, rotation_configs, user_enabled: bool) -> bool:+ """Return whether every component participating in block replay supports compilation."""+ if not user_enabled or not block_quantizer.can_compile_block_forward():+ return False+ return all(getattr(config, "can_compile_block_forward", lambda: True)() for config in rotation_configs)++class AlgorithmComposer:"""An ordered composition of pre-processors + one block quantizer, built froma list of algorithm config objects and an optional compressor.@@ -181,11 +188,19 @@ class AlgorithmComposer:can_compile_block_forward = Falseself.block_quantizer = block_quantizers[0]if orchestrator is not None:- if self.block_quantizer is not None and self.block_quantizer.can_compile_block_forward():- user_torch_compile = getattr(- getattr(orchestrator, "compress_context", None), "enable_torch_compile", False- )- can_compile_block_forward = bool(user_torch_compile)+ user_torch_compile = bool(+ getattr(getattr(orchestrator, "compress_context", None), "enable_torch_compile", False)+ )+ rotation_configs = getattr(orchestrator, "rotation_configs", ())+ can_compile_block_forward = _can_compile_block_forward(+ self.block_quantizer, rotation_configs, user_torch_compile+ )+ if (+ user_torch_compile+ and not can_compile_block_forward+ and any(not getattr(config, "can_compile_block_forward", lambda: True)() for config in rotation_configs)+ ):+ logger.info("Block-forward torch.compile is disabled because an enabled rotation is incompatible.")# Bind compressor-level infrastructure (set before _build_quantizer is called).self.block_forward = (diff --git a/auto_round/algorithms/transforms/base.py b/auto_round/algorithms/transforms/base.pyindex d7093dbc5..46fcbb370 100644--- a/auto_round/algorithms/transforms/base.py+++ b/auto_round/algorithms/transforms/base.py@@ -67,6 +67,10 @@ class BaseRotationConfig:#: Human-readable algorithm name, must be unique across all subclasses.algorithm: str = "base"+ def can_compile_block_forward(self) -> bool:+ """Whether block replay can be compiled after applying this rotation."""+ return True+# ---------------------------------------------------------------------------# Algorithm basediff --git a/auto_round/algorithms/transforms/hadamard/config.py b/auto_round/algorithms/transforms/hadamard/config.pyindex 9e46ce0ff..4947f1ff4 100644--- a/auto_round/algorithms/transforms/hadamard/config.py+++ b/auto_round/algorithms/transforms/hadamard/config.py@@ -106,6 +106,10 @@ class RotationConfig(BaseModel, BaseRotationConfig):"""super().__init__(**data)+ def can_compile_block_forward(self) -> bool:+ """Hadamard may install online hooks that are not Dynamo compatible."""+ return False+@field_validator("backend")@classmethoddef _validate_backend(cls, v: str) -> str:diff --git a/auto_round/auto_scheme/delta_loss.py b/auto_round/auto_scheme/delta_loss.pyindex 20a2427f9..33e39be9b 100644--- a/auto_round/auto_scheme/delta_loss.py+++ b/auto_round/auto_scheme/delta_loss.py@@ -94,7 +94,7 @@ class AutoSchemeWrapperLinear(WrapperLinear):device="cpu",enable_round_tuning=True,need_weight_grad=False,- enable_torch_compile=False,+ enable_torch_compile=True,**kwargs,):"""Wrap ``orig_layer`` to accumulate a ``mix_score`` (weight + activation loss) during@@ -208,7 +208,7 @@ class AutoSchemeWrapperLinearIMatrix(WrapperLinear):device="cpu",enable_round_tuning=True,need_weight_grad=False,- enable_torch_compile=False,+ enable_torch_compile=True,**kwargs,):"""Wrap ``orig_layer`` and eagerly run the imatrix-aware quant search to build ``qdq_w``."""@@ -386,7 +386,7 @@ class AutoSchemeWrapperLinearForGGUFKImatrix(AutoSchemeWrapperLinear):device="cpu",enable_round_tuning=True,need_weight_grad=False,- enable_torch_compile=False,+ enable_torch_compile=True,**kwargs,):"""Wrap ``orig_layer`` and eagerly run the imatrix-weighted GGUF K-quant search to@@ -875,7 +875,7 @@ def get_score_for_scheme(pbar=None,shared_layers=None,need_weight_grad=False,- enable_torch_compile=False,+ enable_torch_compile=True,low_gpu_mem_usage=True,major_device="cpu",batch_size=1,@@ -1901,7 +1901,7 @@ def _gen_layer_config(dataset: str = "pile-10k",tokenizer=None,device_map=None,- enable_torch_compile=False,+ enable_torch_compile=True,model_name=None,major_device="cpu",device_list=None,@@ -2744,7 +2744,7 @@ def gen_layer_config(dataset: str = "pile-10k",tokenizer=None,device_map=None,- enable_torch_compile=False,+ enable_torch_compile=True,low_gpu_mem_usage=True,min_avg_bit_scheme=None,processor=None,diff --git a/auto_round/auto_scheme/gen_auto_scheme.py b/auto_round/auto_scheme/gen_auto_scheme.pyindex f3235c9da..e86e1e52b 100644--- a/auto_round/auto_scheme/gen_auto_scheme.py+++ b/auto_round/auto_scheme/gen_auto_scheme.py@@ -96,7 +96,7 @@ class GenScheme:dataset: str = "pile-10k",device_map: Union[str, torch.device, int, dict, None] = None,tokenizer=None,- enable_torch_compile=False,+ enable_torch_compile=True,processor=None,):self.auto_scheme = auto_schemediff --git a/auto_round/autoround.py b/auto_round/autoround.pyindex 3bb20e274..76d68e264 100644--- a/auto_round/autoround.py+++ b/auto_round/autoround.py@@ -113,7 +113,8 @@ class AutoRound:sym (bool): Whether to use symmetric weight quantization.layer_config (dict): Per-layer quantization configuration.nsamples (int): Number of calibration samples.- enable_torch_compile (bool): Whether to enable torch.compile for quant blocks/layers.+ enable_torch_compile (bool): Whether to enable torch.compile for quant blocks/layers. Defaults to True except+ on Windows, where it defaults to False."""SKIP_ARGS = ("local_args", "kwargs", "cls", "model_cls", "dynamic_compressor", "alg_configs")@@ -146,7 +147,7 @@ class AutoRound:gradient_accumulate_steps: int = 1,low_gpu_mem_usage: bool = False,device_map: Union[str, torch.device, int, dict] = 0,- enable_torch_compile: bool = False,+ enable_torch_compile: Optional[bool] = None,seed: int = 42,enable_adam: bool = False,enable_alg_ext: bool = False,@@ -171,7 +172,8 @@ class AutoRound:gradient_accumulate_steps (int, optional): Gradient accumulation steps. Defaults to 1.low_gpu_mem_usage (bool, optional): Lower GPU memory mode. Defaults to False.device_map (str | dict, optional): Device map for each module. Defaults to 0.- enable_torch_compile (bool, optional): Enable torch.compile for low cost in quantization. Defaults to False.+ enable_torch_compile (bool, optional): Enable torch.compile for low cost in quantization. Defaults to True+ except on Windows, where it defaults to False. Pass True to force it.seed (int, optional): Random seed. Defaults to 42.enable_adam (bool, optional): Enable Adam-based optimizer. Defaults to False.enable_alg_ext (bool, optional): Enable algorithm extension (primarily for INT2)diff --git a/auto_round/cli/main.py b/auto_round/cli/main.pyindex 67d38c7ad..50871dfb7 100644--- a/auto_round/cli/main.py+++ b/auto_round/cli/main.py@@ -280,11 +280,8 @@ def tune(args):device_str, use_auto_mapping = get_device_and_parallelism(args.device_map)- if args.enable_torch_compile:- logger.info(- "`torch.compile` is enabled to reduce tuning costs. "- "If it causes issues, you can disable it by removing `--enable_torch_compile` argument."- )+ if args.enable_torch_compile is False:+ logger.info("`torch.compile` is explicitly disabled with `--disable_torch_compile`.")model_name = args.modelif model_name[-1] == "/":@@ -305,7 +302,7 @@ def tune(args):f"{fmt} is not supported for lm-head quantization, please change to {auto_round_formats}")- enable_torch_compile = True if "--enable_torch_compile" in sys.argv else False+ enable_torch_compile = args.enable_torch_compilescheme = args.scheme.upper()from auto_round.schemes import PRESET_SCHEMES@@ -367,10 +364,10 @@ def tune(args):if args.avg_bits is not None:if args.options is None:raise ValueError("please set --options for auto scheme")- if enable_torch_compile:+ if enable_torch_compile is False:logger.warning(- "`enable_torch_compile=True` with AutoScheme may cause compile errors "- "on some models. If so, try removing `--enable_torch_compile`."+ "`torch.compile` is disabled with AutoScheme. "+ "Enabling it (the default) is strongly recommended to save VRAM.")scheme = AutoScheme(options=args.options,diff --git a/auto_round/cli/parser.py b/auto_round/cli/parser.pyindex ce68bf283..61000a737 100644--- a/auto_round/cli/parser.py+++ b/auto_round/cli/parser.py@@ -150,7 +150,20 @@ def build_quantize_parser(*, prog: str = "auto_round quantize") -> argparse.Arguhelp="Deprecated compatibility flag. Low CPU memory mode is enabled by default.",)rt.add_argument("--disable_low_cpu_mem_usage", action="store_true", help="Disable low CPU memory mode.")- rt.add_argument("--enable_torch_compile", action="store_true", help="Enable torch.compile during quantization.")+ torch_compile_group = rt.add_mutually_exclusive_group()+ torch_compile_group.add_argument(+ "--enable_torch_compile",+ dest="enable_torch_compile",+ action="store_true",+ help="Enable torch.compile during quantization (force enable on Windows).",+ )+ torch_compile_group.add_argument(+ "--disable_torch_compile",+ dest="enable_torch_compile",+ action="store_false",+ help="Disable torch.compile during quantization.",+ )+ rt.set_defaults(enable_torch_compile=None)rt.add_argument("--disable_trust_remote_code", action="store_true", help="Disable trust_remote_code when loading models.")diff --git a/auto_round/compressors/base.py b/auto_round/compressors/base.pyindex 0efa0dd20..ad0c116bc 100644--- a/auto_round/compressors/base.py+++ b/auto_round/compressors/base.py@@ -200,7 +200,7 @@ class BaseOrchestrator(object):scheme: Union[str, dict, QuantizationScheme, AutoScheme] = "W4A16",low_gpu_mem_usage: bool = False,device_map: Union[str, torch.device, int, dict] = 0,- enable_torch_compile: bool = False,+ enable_torch_compile: Optional[bool] = None,seed: int = 42,low_cpu_mem_usage: bool = True,layer_config: Optional[dict] = None,@@ -347,6 +347,19 @@ class BaseOrchestrator(object):self.nblocks = nblocks+ if enable_torch_compile is None:+ enable_torch_compile = sys.platform != "win32"+ if not enable_torch_compile:+ logger.warning_once(+ "`torch.compile` is disabled by default on Windows because TorchInductor requires the MSVC "+ "`cl.exe` compiler, which may not be available. Pass `enable_torch_compile=True` or use "+ "`--enable_torch_compile` to force enable it."+ )+ elif enable_torch_compile and sys.platform == "win32":+ logger.warning_once(+ "Forcing `torch.compile` on Windows. TorchInductor may fail if the MSVC `cl.exe` compiler "+ "is not installed or not available on PATH."+ )self.enable_torch_compile = enable_torch_compile# Whether to pack the layer immediately after tuning@@ -797,7 +810,7 @@ class BaseOrchestrator(object):def diffusion(self) -> bool:return self.model_context.is_diffusion- def _get_torch_compile_guard_state(self) -> tuple[bool, bool, int]:+ def _get_torch_compile_guard_state(self) -> tuple[bool, bool]:"""Return raw dtype state used by torch.compile guard rules."""# Determine fp8 / nvfp4 intent from raw config before scheme resolution.cfg = self.quantize_config@@ -807,7 +820,10 @@ class BaseOrchestrator(object):raw_scheme_upper = raw_scheme.upper()is_raw_nv_fp = "nv_fp" in raw_dt or "nv_fp" in raw_adt or "NVFP" in raw_scheme_upper- is_valid_act_static = cfg.act_dynamic is False and (getattr(cfg, "act_bits", 16) or 16) <= 8+ has_static_global_scale = "static_gs" in raw_adt or "NVFP4" in raw_scheme_upper+ is_valid_act_static = (cfg.act_dynamic is False or has_static_global_scale) and (+ getattr(cfg, "act_bits", 16) or 16+ ) <= 8return is_raw_nv_fp, is_valid_act_static@@ -825,23 +841,18 @@ class BaseOrchestrator(object):):logger.info("%s",- "'enable_torch_compile' is set to `False` by default. "- "Enabling it can reduce tuning cost by 20%, but it might throw an exception.",+ "'enable_torch_compile' is disabled. Enabling it can reduce tuning cost by about 20%.",)def _apply_torch_compile_constraints(self, enable_torch_compile: bool) -> None:"""Apply torch.compile disabling rules for the current compressor state."""self.enable_torch_compile = enable_torch_compile- is_raw_nv_fp, is_valid_act_static = self._get_torch_compile_guard_state()+ _, is_valid_act_static = self._get_torch_compile_guard_state()# On HPU, we rely on torch.compile to speed up the model execution.if self.enable_torch_compile and is_valid_act_static:self.enable_torch_compile = Falselogger.warning_once("reset enable_torch_compile to `False` as activation is static")- # TODO: fix https://github.com/intel/auto-round/issues/1109- if self.enable_torch_compile and is_raw_nv_fp:- self.enable_torch_compile = False- logger.warning_once("reset enable_torch_compile to `False` as nvfp4 is enabled")def _precheck_torch_compile(self, enable_torch_compile: bool) -> None:"""Apply early torch.compile adjustments before scheme resolution.diff --git a/auto_round/compressors/entry.py b/auto_round/compressors/entry.pyindex 9cdff97af..776a3049a 100644--- a/auto_round/compressors/entry.py+++ b/auto_round/compressors/entry.py@@ -335,7 +335,7 @@ class AutoRound(object):low_gpu_mem_usage: bool = False,device_map: Union[str, torch.device, int, dict] = 0,iters: int = None,- enable_torch_compile: bool = False,+ enable_torch_compile: Optional[bool] = None,seed: int = 42,low_cpu_mem_usage: bool = True,layer_config=None,@@ -392,6 +392,7 @@ class AutoRound(object):layer_config=layer_config,tokenizer=tokenizer,device_map=device_map,+ enable_torch_compile=enable_torch_compile,**compressor_kwargs,**base_kwargs,**mllm_kwargs,@@ -682,7 +683,7 @@ class AutoRoundCompatible:gradient_accumulate_steps: int = 1,low_gpu_mem_usage: bool = False,device_map: Union[str, torch.device, int, dict] = 0,- enable_torch_compile: bool = False,+ enable_torch_compile: Optional[bool] = None,seed: int = 42,low_cpu_mem_usage: bool = True,algorithm: str = None,@@ -722,6 +723,7 @@ class AutoRoundCompatible:tokenizer=tokenizer,device_map=device_map,low_cpu_mem_usage=low_cpu_mem_usage,+ enable_torch_compile=enable_torch_compile,**compressor_only_kwargs,**kwargs,)diff --git a/auto_round/compressors/model_free.py b/auto_round/compressors/model_free.pyindex 7f9c1f06c..1fb4ed9cb 100644--- a/auto_round/compressors/model_free.py+++ b/auto_round/compressors/model_free.py@@ -95,6 +95,7 @@ import multiprocessing as mpimport osimport reimport shutil+import sysimport timefrom concurrent.futures import ProcessPoolExecutor, as_completedfrom dataclasses import asdict, fields@@ -107,7 +108,7 @@ from auto_round.compressors.utils import is_mx_fpfrom auto_round.logger import loggerfrom auto_round.schemes import PRESET_SCHEMES, QuantizationScheme, preset_name_to_schemefrom auto_round.utils.common import AUDIO_MM_KEYS, VISION_MM_KEYS, compress_layer_names, to_standard_regex-from auto_round.utils.device import clear_memory, memory_monitor+from auto_round.utils.device import clear_memory, compile_func, memory_monitorfrom auto_round.utils.missing_tensors import quantize_weight_rtn, split_fused_expert_tensors# ---------------------------------------------------------------------------@@ -567,6 +568,7 @@ def _quantize_single_tensor(tensor: torch.Tensor,matcher: "_PatternMatcher",device: str = "cpu",+ quantize_func: Callable = quantize_weight_rtn,) -> tuple[str, dict[str, torch.Tensor], str | None, str | None]:"""Quantize one eligible weight tensor and return packed outputs.@@ -620,7 +622,7 @@ def _quantize_single_tensor(# ---- Integer WOQ path ----try:- qweight, qzeros, scales = quantize_weight_rtn(+ qweight, qzeros, scales = quantize_func(weight=tensor,bits=bits,group_size=group_size,@@ -919,6 +921,7 @@ def _process_shard(matcher: "_PatternMatcher | None" = None,fp8_block_size: list | None = None,model_type: str | None = None,+ enable_torch_compile: bool = False,) -> tuple[dict[str, torch.Tensor], list[str], list[str]]:"""Quantize eligible weights in a single safetensors shard.@@ -941,6 +944,7 @@ def _process_shard(output_tensors: dict[str, torch.Tensor] = {}quantized_layers: list[str] = []+ quantize_func = compile_func(quantize_weight_rtn, device) if enable_torch_compile else quantize_weight_rtnif shard_path.endswith(".bin"):# PyTorch pickle checkpoint — load with weights_only where supported.@@ -1014,6 +1018,7 @@ def _process_shard(tensor,matcher,device,+ quantize_func,)output_tensors.update(out_dict)if q_layer:@@ -1535,6 +1540,7 @@ def _process_single_shard_task(model_type: str | None,quant_output_dir: str,total_shards: int,+ enable_torch_compile: bool = False,) -> tuple[int, str, str | None, str | None, list[str] | None, list[str] | None, list[str] | None]:"""Process one shard in an isolated subprocess task.@@ -1560,6 +1566,7 @@ def _process_single_shard_task(device=device,fp8_block_size=fp8_block_size,model_type=model_type,+ enable_torch_compile=enable_torch_compile,)out_shard_name = f"model-{shard_idx + 1:05d}-of-{total_shards:05d}.safetensors"@@ -1952,6 +1959,7 @@ class _ModelFreeCompressorCore:device: str = "cpu",quant_lm_head: bool = False,quant_nontext_module: bool = False,+ enable_torch_compile: Optional[bool] = None,) -> None:# --- raw inputs ---self.model_name_or_path = model_name_or_path@@ -1963,6 +1971,7 @@ class _ModelFreeCompressorCore:self.device = deviceself.quant_lm_head = quant_lm_headself.quant_nontext_module = quant_nontext_module+ self.enable_torch_compile = sys.platform != "win32" if enable_torch_compile is None else enable_torch_compile# --- derived state populated during run() ---self.scheme_obj: QuantizationScheme | None = None@@ -2248,6 +2257,7 @@ class _ModelFreeCompressorCore:ignore_patterns=self.ignore_patterns,fp8_block_size=self.fp8_block_size,model_type=self.model_type,+ enable_torch_compile=self.enable_torch_compile,quant_output_dir=self._quant_output_dir,total_shards=len(self.shard_names),)@@ -2444,6 +2454,7 @@ class _ModelFreeCompressorCore:f" Diffusion model: {self.is_diffusion_model}\n"f" Quant lm_head: {self.quant_lm_head}\n"f" Quant nontext module: {self.quant_nontext_module}\n"+ f" Torch compile: {self.enable_torch_compile}\n"f" Device: {self.device}")@@ -2515,6 +2526,7 @@ class ModelFreeCompressor(_ModelFreeCompressorCore):tokenizer: Any = None,device_map: Any = None,low_cpu_mem_usage: bool = True,+ enable_torch_compile: Optional[bool] = None,**kwargs,) -> None:import copy@@ -2548,6 +2560,7 @@ class ModelFreeCompressor(_ModelFreeCompressorCore):device=device,quant_lm_head=quant_lm_head,quant_nontext_module=quant_nontext_module,+ enable_torch_compile=enable_torch_compile,)# Compressor-role state (mirrors BaseCompressor attributes used bydiff --git a/auto_round/compressors/orchestrator.py b/auto_round/compressors/orchestrator.pyindex f5f92b905..54619f415 100644--- a/auto_round/compressors/orchestrator.py+++ b/auto_round/compressors/orchestrator.py@@ -77,7 +77,7 @@ class CompressionOrchestrator(BaseOrchestrator):dataset: Union[str, list, tuple, torch.utils.data.DataLoader] = "NeelNanda/pile-10k",low_gpu_mem_usage: bool = False,device_map: Union[str, torch.device, int, dict] = 0,- enable_torch_compile: bool = False,+ enable_torch_compile: Optional[bool] = None,seed: int = 42,low_cpu_mem_usage: bool = True,**kwargs,diff --git a/auto_round/context/compress.py b/auto_round/context/compress.pyindex b8dfc3070..25704864c 100644--- a/auto_round/context/compress.py+++ b/auto_round/context/compress.py@@ -33,7 +33,7 @@ class CompressContext(BaseContext):self,low_cpu_mem_usage: bool = True,low_gpu_mem_usage: bool = False,- enable_torch_compile: bool = False,+ enable_torch_compile: bool = True,is_immediate_packing: bool = False,is_immediate_saving: bool = False,formats: Union[list, str] = None,diff --git a/auto_round/data_type/nvfp.py b/auto_round/data_type/nvfp.pyindex 7a3a2df52..84f356c42 100644--- a/auto_round/data_type/nvfp.py+++ b/auto_round/data_type/nvfp.py@@ -58,7 +58,7 @@ def calculate_gparam(tensor, group_size=16, device="cpu"):if isinstance(tensor, (float, int)):tensor_amax = torch.tensor(tensor, device=device, dtype=torch.float32).abs()elif isinstance(tensor, torch.Tensor):- tensor_amax = tensor.abs().max().to(torch.float32)+ tensor_amax = tensor.to(torch.float32).abs().max()global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_amax)return global_scale@@ -103,7 +103,7 @@ def nv_fp4(tensor, bits=4, group_size=16, v=0, global_scale=None, max_scale=1.0,init_scale = 1.0 if init_scale is None else init_scaletensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)if global_scale is None:- tensor_max = tensor.abs().max().to(torch.float32)+ tensor_max = tensor.to(torch.float32).abs().max()global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_max)global_scale = global_scale.to(device=tensor.device, dtype=torch.float32)if isinstance(max_scale, torch.Tensor):@@ -122,14 +122,14 @@ def nv_fp4_with_static_gs(tensor, bits=4, group_size=16, v=0, tensor_max=None, *orig_dtype = tensor.dtypetensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)if tensor_max is None:- tensor_max = tensor.abs().max().to(torch.float32)+ tensor_max = tensor.to(torch.float32).abs().max()else:if not isinstance(tensor_max, torch.Tensor):tensor_max = torch.tensor(tensor_max, device=tensor.device, dtype=torch.float32)else:tensor_max = tensor_max.to(device=tensor.device, dtype=torch.float32)if tensor_max.numel() != 1:- tensor_max = tensor_max.abs().max()+ tensor_max = tensor_max.to(torch.float32).abs().max()global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_max)global_scale = global_scale.to(tensor.device)@@ -235,12 +235,12 @@ def fp4_v2_with_global_scale(tensor, bits=4, group_size=16, v=0, tensor_max=Noneorig_dtype = tensor.dtypetensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)if tensor_max is None:- tensor_max = tensor.abs().max().to(torch.float32)+ tensor_max = tensor.to(torch.float32).abs().max()elif tensor_max is not None:if not isinstance(tensor_max, torch.Tensor):tensor_max = torch.tensor(tensor_max, device=tensor.device, dtype=torch.float32)if tensor_max.numel() != 1:- tensor_max = tensor.abs().max().to(torch.float32)+ tensor_max = tensor.to(torch.float32).abs().max()global_scale = FLOAT8_UE5M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_max)qdq_res, scale = ref_fp4_quant(tensor, global_scale, group_size, v)qdq_res = revert_tensor_by_pad(qdq_res, orig_shape=orig_shape, pad_len=pad_len)diff --git a/auto_round/wrapper.py b/auto_round/wrapper.pyindex 5a25b25d9..6e8928855 100644--- a/auto_round/wrapper.py+++ b/auto_round/wrapper.py@@ -82,7 +82,7 @@ class WrapperLinear(torch.nn.Module):enable_norm_bias_tuning=False,device="cpu",enable_round_tuning=True,- enable_torch_compile=False,+ enable_torch_compile=True,disable_opt_rtn=True,**kwargs,):@@ -562,7 +562,7 @@ class WrapperLinear(torch.nn.Module):class WrapperWALayer(torch.nn.Module):- def __init__(self, orig_layer, enable_torch_compile=False, device="cpu"):+ def __init__(self, orig_layer, enable_torch_compile=True, device="cpu"):super(WrapperWALayer, self).__init__()self.orig_layer = orig_layerself.enable_torch_compile = enable_torch_compile@@ -770,7 +770,7 @@ def wrapper_block(block,enable_minmax_tuning,enable_norm_bias_tuning,- enable_torch_compile=False,+ enable_torch_compile=True,device="cpu",wrapper_cls=WrapperLinear,**kwargs,diff --git a/docs/environments.md b/docs/environments.mdindex c27591934..5934cef78 100644--- a/docs/environments.md+++ b/docs/environments.md@@ -100,7 +100,7 @@ export AR_SEARCH_SCALE_RATIO=0.75```### AR_DYNAMO_CACHE_SIZE_LIMIT-- **Description**: Minimum value to which `torch._dynamo`'s `cache_size_limit`, `accumulated_cache_size_limit`, and `recompile_limit` are bumped when `enable_torch_compile=True`. The same compiled quant function is reused across every linear layer in a transformer block (q/k/v/o_proj, gate/up/down_proj, ...) but each layer has a different weight shape, so per-layer static recompiles quickly exceed dynamo's default limit (8) and trigger a noisy fallback to eager. Raising the limit keeps static-shape compilation (best perf) and just allows more cache entries.+- **Description**: Minimum value to which `torch._dynamo`'s `cache_size_limit`, `accumulated_cache_size_limit`, and `recompile_limit` are bumped when `torch.compile` is enabled (the default except on Windows). The same compiled quant function is reused across every linear layer in a transformer block (q/k/v/o_proj, gate/up/down_proj, ...) but each layer has a different weight shape, so per-layer static recompiles quickly exceed dynamo's default limit (8) and trigger a noisy fallback to eager. Raising the limit keeps static-shape compilation (best perf) and just allows more cache entries.- **Default**: `16`- **Valid Values**: positive integer- **Usage**: Increase if your model has more than 16 distinct linear-weight shapes per block (rare).diff --git a/docs/environments_CN.md b/docs/environments_CN.mdindex 935accd39..9dd02d251 100644--- a/docs/environments_CN.md+++ b/docs/environments_CN.md@@ -100,7 +100,7 @@ export AR_SEARCH_SCALE_RATIO=0.75```### AR_DYNAMO_CACHE_SIZE_LIMIT-- **描述**:在启用 `enable_torch_compile=True` 时,将 `torch._dynamo` 的 `cache_size_limit`、`accumulated_cache_size_limit` 与 `recompile_limit` 提升到的最小值。同一个被编译的量化函数会被 transformer block 内的所有 linear 层(q/k/v/o_proj、gate/up/down_proj 等)复用,但每层权重 shape 不同,按层的静态重编译会很快超过 dynamo 默认上限(8),导致打印告警并退回 eager。提高该上限可以保留静态 shape 编译(性能最佳),仅增加缓存条目数。+- **描述**:在开启 `torch.compile`(除 Windows 外默认开启)时,将 `torch._dynamo` 的 `cache_size_limit`、`accumulated_cache_size_limit` 与 `recompile_limit` 提升到的最小值。同一个被编译的量化函数会被 transformer block 内的所有 linear 层(q/k/v/o_proj、gate/up/down_proj 等)复用,但每层权重 shape 不同,按层的静态重编译会很快超过 dynamo 默认上限(8),导致打印告警并退回 eager。提高该上限可保留静态 shape 编译(性能最佳),仅增加缓存条目数。- **默认值**:`16`- **有效值**:正整数- **用途**:当模型单个 block 内 linear 权重 shape 种类超过 16 时(较少见)可调大。diff --git a/docs/step_by_step.md b/docs/step_by_step.mdindex e040dc854..1c9f38706 100644--- a/docs/step_by_step.md+++ b/docs/step_by_step.md@@ -238,7 +238,7 @@ model_name_or_path = "Qwen/Qwen3-0.6B"ar = AutoRound(model_name_or_path,scheme="W4A16",- # enable_torch_compile=True,+ # torch.compile is enabled by default except on Windows. On Windows, set True to force enable it.)output_dir = "./tmp_autoround"@@ -849,7 +849,7 @@ autoround.save_quantized(format="auto_awq", output_dir="tmp_autoround")- **Reduced GPU Memory Usage:**- - set `enable_torch_compile` to True+ - keep `torch.compile` enabled by default on non-Windows platforms; on Windows, pass `enable_torch_compile=True` to opt in- enable `low_gpu_mem_usage`(more tuning cost)@@ -869,7 +869,7 @@ autoround.save_quantized(format="auto_awq", output_dir="tmp_autoround")quantize_and_save API, as long as only one export format is specified.- **Speedup the tuning:**- - set `enable_torch_compile` to True+ - keep `torch.compile` enabled by default on non-Windows platforms; on Windows, pass `enable_torch_compile=True` to opt in- use `auto-round-light` configuration@@ -879,6 +879,10 @@ autoround.save_quantized(format="auto_awq", output_dir="tmp_autoround")- or combine them+ `torch.compile` is disabled by default on Windows because TorchInductor requires the MSVC `cl.exe` compiler. Windows+ users can pass `enable_torch_compile=True` to the Python API or use `--enable_torch_compile` to force enable it. On+ other platforms, pass `enable_torch_compile=False` or use `--disable_torch_compile` to opt out.+- **Enable quantized lm-head:**diff --git a/docs/step_by_step_CN.md b/docs/step_by_step_CN.mdindex 660077ef6..1d9dda39f 100644--- a/docs/step_by_step_CN.md+++ b/docs/step_by_step_CN.md@@ -229,7 +229,7 @@ model_name_or_path = "Qwen/Qwen3-0.6B"ar = AutoRound(model_name_or_path,scheme="W4A16",- # 可选开启torch编译加速:enable_torch_compile=True,+ # 除 Windows 外默认启用 torch.compile;Windows 用户可设置 True 强制开启。)output_dir = "./tmp_autoround"@@ -836,7 +836,7 @@ autoround.save_quantized(format="auto_awq", output_dir="tmp_autoround")### 超参数调整#### 降低 GPU 显存占用以下方法可单独或组合使用,其中部分方式会增加训练耗时或带来轻微的精度损失:-- 将 `enable_torch_compile` 设为 True(开启 PyTorch 编译加速,不损失精度)+- 非 Windows 平台保持默认开启 `torch.compile`;Windows 上如需开启,请显式设置 `enable_torch_compile=True`(开启 PyTorch 编译加速,不损失精度)- 开启 `low_gpu_mem_usage`(低显存模式,**增加训练耗时**)- 设置 `--bs 1 --gradient_accumulate_steps 8`(批次1+梯度累积8步,**增加训练耗时**)- 将 `bs` 降至 4(**可能会有轻微的精度损失**)@@ -848,11 +848,15 @@ autoround.save_quantized(format="auto_awq", output_dir="tmp_autoround")#### 提升训练速度以下方法可单独或组合使用,其中部分方式可能带来精度损失:-- 将 `enable_torch_compile` 设为 True(无精度损失)+- 非 Windows 平台保持默认开启 `torch.compile`;Windows 上如需开启,请显式设置 `enable_torch_compile=True`(无精度损失)- 使用 `auto-round-light` (小模型/ 2-bits 场景可能有明显精度损失)- 将 `seqlen` 降至 512(**部分场景可能出现大幅精度损失**)- 将 `bs` 降至 4(**仅有轻微精度损失**)+Windows 上默认关闭 `torch.compile`,因为 TorchInductor 需要 MSVC 的 `cl.exe` 编译器。Windows 用户可在+Python API 中传入 `enable_torch_compile=True`,或使用命令行参数 `--enable_torch_compile` 强制开启。其他+平台如需关闭,可传入 `enable_torch_compile=False` 或使用 `--disable_torch_compile`。+#### 开启 lm-head 层量化该配置目前**仅支持 AutoRound 原生格式的推理**,命令行启用方式如下:```bashdiff --git a/docs/tips_and_tricks.md b/docs/tips_and_tricks.mdindex 3f9177fc2..14015eca4 100644--- a/docs/tips_and_tricks.md+++ b/docs/tips_and_tricks.md@@ -169,8 +169,12 @@ use `--model_dtype "fp16"`.### **7. Better Use `torch.compile`**-**Suggestion: Use `torch.compile` if possible, but be aware that it may throw exceptions for some models, especially-with lower versions of PyTorch**. Starting from PyTorch 2.6, we have set using `torch.compile` as the default.+**`torch.compile` is enabled by default except on Windows. Windows users can pass `enable_torch_compile=True` or use+`--enable_torch_compile` to force enable it. On other platforms, pass `enable_torch_compile=False` or use+`--disable_torch_compile` to opt out.**++It can reduce tuning cost by about 20%, but may throw exceptions on some models, especially with lower versions of+PyTorch.**Reasoning:** `torch.compile` is useful to reduce tuning cost. Thanks Liu Yi for kind discussion.diff --git a/test/fixtures.py b/test/fixtures.pyindex 258b05456..6842f8c14 100644--- a/test/fixtures.py+++ b/test/fixtures.py@@ -388,12 +388,14 @@ def tiny_qwen3_omni_moe_model_path():shutil.rmtree(tiny_model_path, ignore_errors=True)-# Mock torch.cuda.get_device_capability to always return (9, 0) like H100+# Mock FP8 capability checks without letting the fake capability affect Inductor code generation.@pytest.fixture()def mock_fp8_capable_device():from unittest.mock import patch- with patch("torch.cuda.get_device_capability", return_value=(9, 0)):+ with patch("torch.cuda.get_device_capability", return_value=(9, 0)), patch(+ "torch.compile", side_effect=lambda function, *args, **kwargs: function+ ):yielddiff --git a/test/test_cpu/algorithms/test_awq.py b/test/test_cpu/algorithms/test_awq.pyindex 89a976146..32966bd07 100644--- a/test/test_cpu/algorithms/test_awq.py+++ b/test/test_cpu/algorithms/test_awq.py@@ -368,6 +368,7 @@ class TestAWQWeightClip:nsamples=2,seqlen=8,batch_size=2,+ enable_torch_compile=False, # disable torch.compile to ensure CI pass, cannot reproduce in local env.)model, layer_config = ar.quantize()diff --git a/test/test_cpu/conftest.py b/test/test_cpu/conftest.pynew file mode 100644index 000000000..9cd601db0--- /dev/null+++ b/test/test_cpu/conftest.py@@ -0,0 +1,8 @@+import pytest+import torch++[email protected](autouse=True)+def disable_torch_compile(monkeypatch):+ """Skip torch.compile overhead in CPU tests."""+ monkeypatch.setattr(torch, "compile", lambda function, *args, **kwargs: function)diff --git a/test/test_cpu/core/test_init.py b/test/test_cpu/core/test_init.pyindex 6b89434cd..7ad883df4 100644--- a/test/test_cpu/core/test_init.py+++ b/test/test_cpu/core/test_init.py@@ -1,13 +1,85 @@-import argparse+import inspect+import loggingfrom auto_round import AutoRound+from auto_round.auto_scheme import AutoScheme+from auto_round.cli.parser import build_quantize_parser+from auto_round.compressors.base import BaseOrchestratorfrom auto_round.compressors.entry import AutoRound as NewAutoRound+from auto_round.compressors.entry import AutoRoundCompatible+++def test_torch_compile_platform_default_is_deferred():+ assert inspect.signature(AutoRound.__new__).parameters["enable_torch_compile"].default is None+ assert inspect.signature(NewAutoRound.__new__).parameters["enable_torch_compile"].default is None+ assert inspect.signature(AutoRoundCompatible.__new__).parameters["enable_torch_compile"].default is None+ assert inspect.signature(BaseOrchestrator.__init__).parameters["enable_torch_compile"].default is None+++def test_cli_torch_compile_flags():+ parser = build_quantize_parser()+ assert parser.parse_args(["--model", "test-model"]).enable_torch_compile is None+ assert parser.parse_args(["--model", "test-model", "--enable_torch_compile"]).enable_torch_compile is True+ assert parser.parse_args(["--model", "test-model", "--disable_torch_compile"]).enable_torch_compile is False+++def test_auto_scheme_inherits_torch_compile_setting():+ auto_scheme = AutoScheme(avg_bits=4.0, options=["W4A16"])+ assert auto_scheme.enable_torch_compile is None+++def test_torch_compile_runtime_defaults(tiny_opt_model_path):+ ar = AutoRound(model=tiny_opt_model_path, scheme="W4A16", iters=0, nsamples=1)+ assert ar.enable_torch_compile++ ar = AutoRound(+ model=tiny_opt_model_path,+ scheme="W4A16",+ iters=0,+ nsamples=1,+ enable_torch_compile=False,+ )+ assert not ar.enable_torch_compile++ ar = AutoRound(model=tiny_opt_model_path, scheme="NVFP4", iters=0, nsamples=1)+ assert ar.enable_torch_compile+++def test_torch_compile_windows_defaults(monkeypatch, caplog, tiny_opt_model_path):+ monkeypatch.setattr("auto_round.compressors.base.sys.platform", "win32")++ with caplog.at_level(logging.WARNING):+ ar = AutoRound(model=tiny_opt_model_path, scheme="W4A16", iters=0, nsamples=1)+ assert not ar.enable_torch_compile+ assert "disabled by default on Windows" in caplog.text+ assert "cl.exe" in caplog.text++ caplog.clear()+ with caplog.at_level(logging.WARNING):+ ar = AutoRound(+ model=tiny_opt_model_path,+ scheme="W4A16",+ iters=0,+ nsamples=1,+ enable_torch_compile=True,+ )+ assert ar.enable_torch_compile+ assert "Forcing `torch.compile` on Windows" in caplog.text++ ar = AutoRound(+ model=tiny_opt_model_path,+ scheme="W4A16",+ iters=0,+ nsamples=1,+ enable_torch_compile=False,+ )+ assert not ar.enable_torch_compiledef test_argparse_check(tiny_opt_model_path):return # TODO wenhuachar = AutoRound(model=tiny_opt_model_path, scheme="NVFP4", enable_torch_compile=True)- assert not ar.enable_torch_compile, "NVFP4 cannot work with torch.compile."+ assert ar.enable_torch_compile, "NVFP4 should preserve the torch.compile setting."ar = AutoRound(model=tiny_opt_model_path, scheme="FP8_STATIC", enable_torch_compile=True)assert not ar.enable_torch_compile, "FP8_STATIC cannot work with torch.compile."diff --git a/test/test_cpu/core/test_pipeline_fail_fast.py b/test/test_cpu/core/test_pipeline_fail_fast.pyindex f4e59bc1d..59972e557 100644--- a/test/test_cpu/core/test_pipeline_fail_fast.py+++ b/test/test_cpu/core/test_pipeline_fail_fast.py@@ -3,7 +3,7 @@import pytestfrom auto_round import AWQConfig, OptimizedRTNConfig, RotationConfig, RTNConfig, SignRoundConfig, SpinQuantConfig-from auto_round.algorithms.composer import AlgorithmComposer+from auto_round.algorithms.composer import AlgorithmComposer, _can_compile_block_forwardfrom auto_round.algorithms.config_resolver import (get_algorithm_class,resolve_shared_config_values,@@ -29,6 +29,16 @@ class NoWeightClipConfig(RTNConfig):pass+class CompileCompatibleRotation:+ def can_compile_block_forward(self):+ return True+++class CompileCompatibleQuantizer:+ def can_compile_block_forward(self):+ return True++def test_split_awq_plus_rtn():pre, block = split_quantization_configs([AWQConfig(), RTNConfig()])assert len(pre) == 1 and type(pre[0]).__name__ == "AWQConfig"@@ -51,6 +61,15 @@ def test_pipeline_multiple_block_quantizers_rejected():AlgorithmComposer([RTNConfig(), SignRoundConfig()])+def test_hadamard_disables_only_block_forward_compile():+ quantizer = CompileCompatibleQuantizer()+ hadamard = RotationConfig()++ assert not _can_compile_block_forward(quantizer, [hadamard], user_enabled=True)+ assert _can_compile_block_forward(quantizer, [CompileCompatibleRotation()], user_enabled=True)+ assert not _can_compile_block_forward(quantizer, [], user_enabled=False)++def test_registry_builtin_aliases_and_unknown():assert isinstance(_r.resolve_alg_config("RTN"), RTNConfig)assert isinstance(_r.resolve_alg_config("awq"), AWQConfig)diff --git a/test/test_cpu/export/test_export.py b/test/test_cpu/export/test_export.pyindex 3b6c411eb..baa1394f9 100644--- a/test/test_cpu/export/test_export.py+++ b/test/test_cpu/export/test_export.py@@ -13,7 +13,7 @@ from auto_round.export.export_to_autoround import export as autoround_exportfrom auto_round.export.export_to_autoround import export_to_fp8 as autoround_fp8_exportfrom auto_round.export.export_to_awq import export as awq_export-from ...helpers import forbid_threaded_packing, get_model_path, opt_name_or_path, transformers_version+from ...helpers import forbid_threaded_packing, get_model_pathdef _get_folder_size(path: str) -> float:@@ -28,19 +28,17 @@ def _get_folder_size(path: str) -> float:class TestAutoRound:-- @classmethod- def setup_class(self):- self.model_name = opt_name_or_path- self.model = AutoModelForCausalLM.from_pretrained(self.model_name, torch_dtype="auto", trust_remote_code=True)- self.tokenizer = AutoTokenizer.from_pretrained(self.model_name, trust_remote_code=True)-@classmethoddef teardown_class(self):shutil.rmtree("runs", ignore_errors=True)@pytest.fixture(autouse=True)- def _save_dir(self, tmp_path):+ def _test_setup(self, tiny_opt_model_path, tmp_path):+ self.model_name = tiny_opt_model_path+ self.model = AutoModelForCausalLM.from_pretrained(+ tiny_opt_model_path, torch_dtype="auto", trust_remote_code=True+ )+ self.tokenizer = AutoTokenizer.from_pretrained(tiny_opt_model_path, trust_remote_code=True)self.save_dir = str(tmp_path / "saved")yieldshutil.rmtree(self.save_dir, ignore_errors=True)@@ -225,10 +223,10 @@ class TestAutoRound:quantized_model_path = self.save_dir_, quantized_model_path = autoround.quantize_and_save(output_dir=quantized_model_path, format="auto_round")f = safe_open(os.path.join(quantized_model_path, "model.safetensors"), framework="pt")- assert "model.decoder.layers.8.self_attn.k_proj.input_scale" in f.keys()- assert "model.decoder.layers.8.self_attn.k_proj.weight_scale" in f.keys()- assert f.get_tensor("model.decoder.layers.5.self_attn.v_proj.input_scale").shape == torch.Size([1])- assert f.get_tensor("model.decoder.layers.5.self_attn.v_proj.weight").dtype == torch.float8_e4m3fn+ assert "model.decoder.layers.0.self_attn.k_proj.input_scale" in f.keys()+ assert "model.decoder.layers.0.self_attn.k_proj.weight_scale" in f.keys()+ assert f.get_tensor("model.decoder.layers.0.self_attn.v_proj.input_scale").shape == torch.Size([1])+ assert f.get_tensor("model.decoder.layers.0.self_attn.v_proj.weight").dtype == torch.float8_e4m3fnif static_kv_dtype is None:with torch.no_grad():import transformers@@ -258,13 +256,13 @@ class TestAutoRound:assert output is not None, "Output should not be None"if static_kv_dtype == "fp8":- assert "model.decoder.layers.8.self_attn.k_scale" in f.keys()- assert "model.decoder.layers.8.self_attn.v_scale" in f.keys()- assert f.get_tensor("model.decoder.layers.5.self_attn.v_scale").shape == torch.Size([1])- assert f.get_tensor("model.decoder.layers.5.self_attn.k_scale").shape == torch.Size([1])+ assert "model.decoder.layers.0.self_attn.k_scale" in f.keys()+ assert "model.decoder.layers.0.self_attn.v_scale" in f.keys()+ assert f.get_tensor("model.decoder.layers.0.self_attn.v_scale").shape == torch.Size([1])+ assert f.get_tensor("model.decoder.layers.0.self_attn.k_scale").shape == torch.Size([1])assert (- f.get_tensor("model.decoder.layers.5.self_attn.k_scale").dtype == torch.float32- or f.get_tensor("model.decoder.layers.5.self_attn.k_scale").dtype == torch.bfloat16+ f.get_tensor("model.decoder.layers.0.self_attn.k_scale").dtype == torch.float32+ or f.get_tensor("model.decoder.layers.0.self_attn.k_scale").dtype == torch.bfloat16)model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", trust_remote_code=True)@@ -286,10 +284,10 @@ class TestAutoRound:_, quantized_model_path = autoround.quantize_and_save(output_dir=quantized_model_path, format="auto_round")f = safe_open(os.path.join(quantized_model_path, "model.safetensors"), framework="pt")- assert "model.decoder.layers.8.self_attn.k_proj.input_scale" in f.keys()- assert "model.decoder.layers.8.self_attn.k_proj.weight_scale" in f.keys()- assert f.get_tensor("model.decoder.layers.5.self_attn.v_proj.input_scale").shape == torch.Size([1])- assert f.get_tensor("model.decoder.layers.5.self_attn.v_proj.weight").dtype == torch.float8_e4m3fn+ assert "model.decoder.layers.0.self_attn.k_proj.input_scale" in f.keys()+ assert "model.decoder.layers.0.self_attn.k_proj.weight_scale" in f.keys()+ assert f.get_tensor("model.decoder.layers.0.self_attn.v_proj.input_scale").shape == torch.Size([1])+ assert f.get_tensor("model.decoder.layers.0.self_attn.v_proj.weight").dtype == torch.float8_e4m3fndef test_static_fp8_attn(self):import os@@ -310,13 +308,13 @@ class TestAutoRound:quantized_model_path = self.save_dir_, quantized_model_path = autoround.quantize_and_save(output_dir=quantized_model_path, format="auto_round")f = safe_open(os.path.join(quantized_model_path, "model.safetensors"), framework="pt")- assert "model.decoder.layers.8.self_attn.k_proj.input_scale" in f.keys()- assert "model.decoder.layers.8.self_attn.k_proj.weight_scale" in f.keys()- assert f.get_tensor("model.decoder.layers.5.self_attn.v_proj.input_scale").shape == torch.Size([1])- assert f.get_tensor("model.decoder.layers.5.self_attn.v_proj.weight").dtype == torch.float8_e4m3fn+ assert "model.decoder.layers.0.self_attn.k_proj.input_scale" in f.keys()+ assert "model.decoder.layers.0.self_attn.k_proj.weight_scale" in f.keys()+ assert f.get_tensor("model.decoder.layers.0.self_attn.v_proj.input_scale").shape == torch.Size([1])+ assert f.get_tensor("model.decoder.layers.0.self_attn.v_proj.weight").dtype == torch.float8_e4m3fncheck_attrs = ["k_scale", "v_scale", "q_scale"]for attr in check_attrs:- weight_name = f"model.decoder.layers.8.self_attn.{attr}"+ weight_name = f"model.decoder.layers.0.self_attn.{attr}"assert weight_name in f.keys()assert f.get_tensor(weight_name).shape == torch.Size([1])assert f.get_tensor(weight_name).dtype == torch.float32 or f.get_tensor(weight_name).dtype == torch.bfloat16@@ -524,8 +522,8 @@ class TestAutoRound:quantized_model_path = self.save_dir_, quantized_model_path = autoround.quantize_and_save(output_dir=quantized_model_path, format="llm_compressor")with safe_open(os.path.join(quantized_model_path, "model.safetensors"), framework="pt") as f:- assert "model.decoder.layers.8.self_attn.k_proj.weight_scale" in f.keys()- assert f.get_tensor("model.decoder.layers.5.self_attn.v_proj.weight").dtype == torch.int8+ assert "model.decoder.layers.0.self_attn.k_proj.weight_scale" in f.keys()+ assert f.get_tensor("model.decoder.layers.0.self_attn.v_proj.weight").dtype == torch.int8shutil.rmtree(quantized_model_path, ignore_errors=True)@pytest.mark.parametrize(@@ -553,10 +551,10 @@ class TestAutoRound:_, quantized_model_path = autoround.quantize_and_save(output_dir=quantized_model_path, format="llm_compressor")with safe_open(os.path.join(quantized_model_path, "model.safetensors"), framework="pt") as f:# weights must be packed as int32 (compressed-tensors stores both int4 and int8 as torch.int32)- weight = f.get_tensor("model.decoder.layers.5.self_attn.v_proj.weight_packed")+ weight = f.get_tensor("model.decoder.layers.0.self_attn.v_proj.weight_packed")assert weight.dtype == torch.int32, f"Expected int32 weight for {scheme}, got {weight.dtype}"# weight_scale must be present and be a float tensor- scale_key = "model.decoder.layers.8.self_attn.k_proj.weight_scale"+ scale_key = "model.decoder.layers.0.self_attn.k_proj.weight_scale"assert scale_key in f.keys(), f"Missing {scale_key} for {scheme} export"scale = f.get_tensor(scale_key)assert scale.dtype in (diff --git a/test/test_cpu/quantization/test_model_free.py b/test/test_cpu/quantization/test_model_free.pyindex 520c05d0b..610779882 100644--- a/test/test_cpu/quantization/test_model_free.py+++ b/test/test_cpu/quantization/test_model_free.py@@ -16,6 +16,7 @@import jsonimport os+from unittest.mock import Mockimport pytest@@ -260,6 +261,17 @@ class TestParseLayerConfig:class TestProcessShard:+ @pytest.mark.parametrize("enabled, expected_calls", [(True, 1), (False, 0)])+ def test_torch_compile_setting(self, tmp_path, monkeypatch, enabled, expected_calls):+ shard_path = str(tmp_path / "shard.safetensors")+ save_file({"layer.fc1.weight": torch.randn(64, 128)}, shard_path)+ compile_mock = Mock(side_effect=lambda func, _device: func)+ monkeypatch.setattr("auto_round.compressors.model_free.compile_func", compile_mock)++ _process_shard(shard_path, _DEFAULT_SCHEME, {}, [], enable_torch_compile=enabled)++ assert compile_mock.call_count == expected_calls+def test_quantizes_eligible_weights(self, tmp_path):shard_path = str(tmp_path / "shard.safetensors")save_file({"layer.fc1.weight": torch.randn(64, 128), "layer.fc1.bias": torch.randn(64)}, shard_path)diff --git a/test/test_cpu/quantization/test_model_free_parity.py b/test/test_cpu/quantization/test_model_free_parity.pyindex 787c7a627..6f0943f28 100644--- a/test/test_cpu/quantization/test_model_free_parity.py+++ b/test/test_cpu/quantization/test_model_free_parity.py@@ -141,6 +141,7 @@ def test_parity_model_free_vs_disable_opt_rtn(tmp_path, tiny_opt_model_path, schdisable_opt_rtn=True,model_free=True,device_map=device,+ enable_torch_compile=False, # disable torch.compile to ensure model is loaded in full precision)assert getattr(ar_a, "model_free", False) is True# Model must NOT be loaded into memory in the model-free path.@@ -159,6 +160,7 @@ def test_parity_model_free_vs_disable_opt_rtn(tmp_path, tiny_opt_model_path, schdisable_model_free=True, # opt out of auto-routingdevice_map=device,amp=False, # disable_amp to ensure model is loaded in full precision+ enable_torch_compile=False, # disable torch.compile to ensure model is loaded in full precision)assert getattr(ar_b, "model_free", False) is False# Confirm the regular path actually loaded the model on the requesteddiff --git a/test/test_cpu/quantization/test_nvfp4_quant_linear.py b/test/test_cpu/quantization/test_nvfp4_quant_linear.pyindex 80a289b70..9dc84fe15 100644--- a/test/test_cpu/quantization/test_nvfp4_quant_linear.py+++ b/test/test_cpu/quantization/test_nvfp4_quant_linear.py@@ -26,6 +26,15 @@ def fixed_seed():# (Optional) cleanup or reset after test+def test_calculate_gparam_with_float8_input():+ tensor = torch.tensor([[-2.0, 0.0, 1.0]], dtype=torch.float32).to(torch.float8_e4m3fn)++ global_scale = calculate_gparam(tensor)++ assert global_scale.dtype == torch.float32+ assert torch.isfinite(global_scale)++@pytest.mark.parametrize("scheme", [AutoRoundExportFormat.NVFP4.value])@torch.inference_mode()def test_nvfp4_quantlinear_from_original_and_forward(scheme):diff --git a/test/test_cuda/integrations/test_sglang.py b/test/test_cuda/integrations/test_sglang.pyindex 34465a74e..457612a4f 100644--- a/test/test_cuda/integrations/test_sglang.py+++ b/test/test_cuda/integrations/test_sglang.py@@ -1,13 +1,12 @@import dataclasses-import gcimport json+import multiprocessingimport multiprocessing.resource_trackerimport shutil-import sys+import tracebackfrom pathlib import Pathimport pytest-import sglang as sglimport torchfrom auto_round import AutoRound@@ -29,6 +28,34 @@ def _patched_stop(self, *args, _orig=_original_stop, **kwargs):multiprocessing.resource_tracker.ResourceTracker._stop = _patched_stop+def _run_sglang_worker(model_path, connection):+ try:+ import sglang as sgl+ from sglang.srt.server_args import ServerArgs++ fields = {field.name for field in dataclasses.fields(ServerArgs)}+ engine_kwargs = {"model_path": model_path, "mem_fraction_static": 0.5}+ if "disable_piecewise_cuda_graph" in fields:+ engine_kwargs["disable_piecewise_cuda_graph"] = True+ if "cuda_graph_bs" in fields:+ engine_kwargs["cuda_graph_bs"] = [1]+ elif "disable_cuda_graph" in fields:+ engine_kwargs["disable_cuda_graph"] = True+ if "cuda_graph_bs_decode" in fields:+ engine_kwargs["cuda_graph_bs_decode"] = [1]++ llm = sgl.Engine(**engine_kwargs)+ try:+ outputs = llm.generate(["Hello, my name is"], {"temperature": 0.6, "top_p": 0.95})+ connection.send((True, outputs[0]["text"]))+ finally:+ llm.shutdown()+ except BaseException:+ connection.send((False, traceback.format_exc()))+ finally:+ connection.close()++class TestAutoRound:model_name = qwen_name_or_path@@ -66,36 +93,26 @@ class TestAutoRound:except Exception:pass- def _server_arg_fields():- try:- from sglang.srt.server_args import ServerArgs-- return {f.name for f in dataclasses.fields(ServerArgs)}- except Exception:- return set()-- engine_kwargs = {"model_path": str(model_path), "mem_fraction_static": 0.5}- fields = _server_arg_fields()- if "disable_piecewise_cuda_graph" in fields:- engine_kwargs["disable_piecewise_cuda_graph"] = True- if "cuda_graph_bs" in fields:- engine_kwargs["cuda_graph_bs"] = [1]- elif "disable_cuda_graph" in fields:- engine_kwargs["disable_cuda_graph"] = True- if "cuda_graph_bs_decode" in fields:- engine_kwargs["cuda_graph_bs_decode"] = [1]-- llm = sgl.Engine(**engine_kwargs)+ context = multiprocessing.get_context("spawn")+ parent_connection, child_connection = context.Pipe(duplex=False)+ process = context.Process(target=_run_sglang_worker, args=(str(model_path), child_connection))+ process.start()+ child_connection.close()try:- prompts = ["Hello, my name is"]- sampling_params = {"temperature": 0.6, "top_p": 0.95}- outputs = llm.generate(prompts, sampling_params)- return outputs[0]["text"]+ if not parent_connection.poll(600):+ process.terminate()+ pytest.fail("SGLang inference timed out after 600 seconds")+ succeeded, result = parent_connection.recv()finally:- llm.shutdown()- del llm- gc.collect()- torch.cuda.empty_cache()+ parent_connection.close()+ process.join(timeout=30)+ if process.is_alive():+ process.kill()+ process.join()++ if not succeeded:+ pytest.fail(f"SGLang worker failed:\n{result}")+ return resultdef test_ar_format_sglang(self, dataloader):autoround = AutoRound(