PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
NeUQI grid search for optimized RTN (--enable_neuqi: joint asym scale/zp search, two-stage sym search, frozen-init anchor) (#2341)
avtc44b48d60875e
diff --git a/README.md b/README.mdindex 5dd3c4535..ed040ce09 100644--- a/README.md+++ b/README.md@@ -41,6 +41,8 @@ See our papers [SignRoundV1](https://arxiv.org/pdf/2309.05516) and [SignRoundV2]* [2026/09] We now support 5/6/7-bit WOQ models in vLLM and Transformers on CUDA devices, thanks to Humming Kernel.+* [2026/09] The **NeUQI** grid search ([arXiv 2505.17595](https://arxiv.org/abs/2505.17595)) is available for calibration-free quantization: `--enable_neuqi` with `--iters 0` runs a joint (scale, zero-point) search for asymmetric layers and a two-stage scale search for symmetric layers, and anchors the tuning grid when `iters > 0`. [*Results*](./docs/neuqi_acc.md)+* [2026/08] We experimentally support **algorithm composition** (e.g., `--algs awq,signround` or `--algs hadamard,awq,signround`) to improve accuracy [*Overview*](./docs/algorithm_combinations.md). We welcome any practical, deployable algorithms that are ready for real-world use. Feel free to submit a PR or leave a comment in Issues.* [2026/06] AutoScheme has been refined to improve accuracy for gguf format. See [AutoScheme Accuracy](./docs/auto_scheme_acc.md) for details. This enhancement incurs additional tuning cost.@@ -78,7 +80,7 @@ Support **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** for maximum compatibility. DeAutomatically configure in minutes, with about 1.1X-1.5X the model’s BF16 RAM size as overhead. Accuracy [results](./docs/auto_scheme_acc.md) and [user guide](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#autoscheme).✅ **Optimized Round-to-Nearest Mode**-Use `--iters 0` for fast quantization with some accuracy drop for 4 bits. Details are shown in [opt_rtn mode](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#opt-rtn-mode)+Use `--iters 0` for fast quantization with some accuracy drop for 4 bits; an optional NeUQI grid search (`--enable_neuqi`) improves the zero-shot scales. Details are shown in [opt_rtn mode](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#opt-rtn-mode)✅ **Affordable Quantization Cost**Quantize 7B models in about 10 minutes on a single GPU. Details are shown in [quantization costs](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#quantization-costs)diff --git a/README_CN.md b/README_CN.mdindex cf1863231..e1ca93044 100644--- a/README_CN.md+++ b/README_CN.md@@ -39,6 +39,8 @@ AutoRound 是专为大语言模型(LLMs)和视觉-语言模型(VLMs)设* [2026/09] 现在支持在 CUDA 设备上通过 vLLM 和 Transformers 使用 5/6/7-bit WOQ 模型,感谢 Humming Kernel 的支持。+* [2026/09] 面向免标定量化的 **NeUQI** 网格搜索([arXiv 2505.17595](https://arxiv.org/abs/2505.17595))现已可用:`--enable_neuqi` 配合 `--iters 0` 时,非对称层执行联合 (scale, zero-point) 搜索、对称层执行两阶段 scale 搜索;`iters > 0` 时搜索结果将作为调优网格的锚点:[*结果*](./docs/neuqi_acc.md)。+* [2026/08] 我们实验性地支持**算法组合**(例如 `--algs awq,signround` 或 `--algs hadamard,awq,signround`),以提升精度:[*总览*](./docs/algorithm_combinations_CN.md). 我们欢迎能真正落地的任何算法组合,欢迎提交 PR 或在 Issues 中留言。* [2026/08] 感谢 Humming Kernel,AutoScheme WOQ 在 vLLM 上的部署已实验性恢复:[*vLLM PR*](https://github.com/vllm-project/vllm/pull/52890),[*示例模型*](https://huggingface.co/Intel/Qwen3.8-27B-bpw2.8-AutoRound)。注意:共享层需按 vLLM 的融合模式进行配置。diff --git a/auto_round/algorithms/quantization/base.py b/auto_round/algorithms/quantization/base.pyindex 70b3299e4..f9cfaebd4 100644--- a/auto_round/algorithms/quantization/base.py+++ b/auto_round/algorithms/quantization/base.py@@ -231,6 +231,7 @@ class BaseQuantizer(BaseAlgorithm):enable_round_tuning=False,enable_torch_compile=self.compress_context.enable_torch_compile,disable_opt_rtn=disable_opt_rtn,+ enable_neuqi=getattr(self.config, "enable_neuqi", False),iters=0,)layer = layer.unwrapper({})@@ -247,6 +248,8 @@ class BaseQuantizer(BaseAlgorithm):enable_norm_bias_tuning=False,enable_round_tuning=False,enable_torch_compile=self.compress_context.enable_torch_compile,+ disable_opt_rtn=disable_opt_rtn,+ enable_neuqi=getattr(self.config, "enable_neuqi", False),iters=0,)layer = layer.unwrapper({})diff --git a/auto_round/algorithms/quantization/rtn/config.py b/auto_round/algorithms/quantization/rtn/config.pyindex b7b90ef39..ac4fc376c 100644--- a/auto_round/algorithms/quantization/rtn/config.py+++ b/auto_round/algorithms/quantization/rtn/config.py@@ -41,12 +41,24 @@ class RTNConfig(QuantizationConfig):const=False,help="Force optimized RTN path.",)+ registry.add_argument(+ "--enable_neuqi",+ field="enable_neuqi",+ action="store_true",+ help=(+ "Enable the NeUQI grid search (arXiv 2505.17595): the asymmetric zero-shot "+ "optimized-RTN path runs a joint (scale, integer zero-point) search instead of plain "+ "min/max, symmetric layers run the two-stage scale search, and SignRound tuning "+ "anchors its grid to the search result (frozen init)."+ ),+ )def __init__(self,*,disable_opt_rtn: bool = None,enable_opt_rtn: bool = None,+ enable_neuqi: bool = False,**kwargs,) -> None:"""Initialize an RTN configuration.@@ -56,6 +68,14 @@ class RTNConfig(QuantizationConfig):``None`` keeps the default heuristic, True forces plainRTN, and False forces the optimized implementation.enable_opt_rtn: Convenience alias for ``disable_opt_rtn=False``.+ enable_neuqi: Opt into the NeUQI grid search (arXiv 2505.17595):+ the asymmetric zero-shot optimized-RTN path runs a joint+ (scale, integer zero-point) search instead of plain min/max,+ symmetric layers run the two-stage scale search, and SignRound+ tuning (``iters > 0``) anchors its grid to the search result+ (frozen init). Requires the optimized path (``disable_opt_rtn``+ not forced True). The search grid sizes are tunable via+ ``AR_NEUQI_COARSE`` and ``AR_NEUQI_FINE``.**kwargs: Common quantization arguments forwarded toQuantizationConfig, such as bits, group_size, sym,data_type, and activation quantization fields.@@ -77,6 +97,16 @@ class RTNConfig(QuantizationConfig):disable_opt_rtn = Falseself.disable_opt_rtn = disable_opt_rtn+ if enable_neuqi and disable_opt_rtn:+ # validated AFTER the W8A16/W8A8 heuristic above so the auto-disable+ # cannot silently neutralize the search opt-in+ raise ValueError(+ "--enable_neuqi requires the optimized-RTN path: it replaces the plain min/max "+ "initialization with a search. Drop --disable_opt_rtn. Note "+ "that W8A16/W8A8 schemes auto-disable the optimized path for efficiency."+ )+ self.enable_neuqi = enable_neuqi+class OptimizedRTNConfig(RTNConfig):need_calib = Truediff --git a/auto_round/algorithms/quantization/rtn/quantizer.py b/auto_round/algorithms/quantization/rtn/quantizer.pyindex e4c6466e2..38584dee6 100644--- a/auto_round/algorithms/quantization/rtn/quantizer.py+++ b/auto_round/algorithms/quantization/rtn/quantizer.py@@ -12,7 +12,10 @@# See the License for the specific language governing permissions and# limitations under the License.+import re+import torch+import torch.nn as nnfrom auto_round.algorithms.quantization.base import BaseQuantizerfrom auto_round.algorithms.quantization.rtn.config import OptimizedRTNConfig, RTNConfig@@ -22,6 +25,11 @@ from auto_round.utils import (SUPPORTED_LAYER_TYPES,check_to_quantized,)+from auto_round.utils.device_manager import device_manager++_EXPERT_BATCH_MAX_ELEMS = 2**28 # ~1 GiB fp32 stacked weights per batched call++_EXPERT_RE = re.compile(r"^(?P<parent>.*)\.experts\.\d+\.(?P<proj>[^.]+)$")@register_pipeline_member(RTNConfig)@@ -132,9 +140,200 @@ class OptimizedRTNQuantizer(RTNQuantizer):input_ids: Raw token IDs from the tokenizer (unused in RTN).**kwargs: Reserved for forward-compatibility with future parameters."""- # Normalize imatrix and quantize layers+ # Normalize imatrix (cheap elementwise divides), then quantize the+ # block's target modules - same-shape expert projections batched into+ # single stacked search calls when the search is active+ targets = []for name, m in block.named_modules():if hasattr(m, "imatrix"):m.imatrix /= m.imatrix_cntif hasattr(m, "global_name") and check_to_quantized(m):- self.quantize_layer_outside_block(m)+ targets.append(m)+ self._quantize_targets(targets)++ def _split_expert_batches(self, targets: list):+ """Partition same-shape expert projections into batchable groups.++ Experts of one layer share weight shapes (e.g. 192 x ``[1536, 4096]``+ gate/up/down projections). The per-group search is row-independent, so+ a whole group can be quantized in one call by stacking weights along+ the output dim - same results as per-module calls, one search's worth+ of per-module overhead instead of 192.+ """+ grouped = {}+ singles = []+ for m in targets:+ match = _EXPERT_RE.match(getattr(m, "global_name", "") or "")+ if (+ match is None+ or type(m) is not nn.Linear+ or not isinstance(getattr(m, "group_size", None), int)+ or getattr(m, "group_size", None) <= 0+ or getattr(m, "super_bits", None) is not None+ or getattr(m, "data_type", "int") != "int"+ or m.weight.shape[1] % m.group_size != 0 # not row-divisible: keep per-module+ or getattr(m, "act_bits", 16) <= 8 # act-quant layers need the per-module wrapper path+ ):+ singles.append(m)+ continue+ key = (+ match["parent"],+ match["proj"],+ tuple(m.weight.shape),+ m.bits,+ m.group_size,+ bool(getattr(m, "sym", False)),+ )+ grouped.setdefault(key, []).append(m)+ batches = []+ for g in grouped.values():+ if len(g) >= 2:+ batches.append(g)+ else:+ singles.extend(g) # singleton groups must stay per-module, never dropped+ return batches, singles++ def _expert_search_active(self) -> bool:+ """Whether the optimized-RTN NeUQI search runs for expert modules.++ Mirrors the MoE heuristic in ``_quantize_layer_via_rtn``: experts skip+ the search unless explicitly forced on (``enable_opt_rtn``), so batching+ must not change that decision - only batch when the search would run.+ Both symmetry classes search under ``enable_neuqi`` (asym: joint+ (scale, zp); sym: two-stage scale), so both batch.+ """+ cfg = self.config+ if bool(getattr(cfg, "disable_opt_rtn", False)):+ return False+ if getattr(cfg, "orig_disable_opt_rtn", None) is None and getattr(self.model_context, "is_moe_model", False):+ return False+ return bool(getattr(cfg, "enable_neuqi", False))++ def _quantize_expert_batch(self, mods: list, device) -> list:+ """Quantize same-shape expert projections in one stacked search call.++ Returns the modules NOT quantized here (empty list = success); the+ caller falls back to per-module calls for them. Reproduces the+ per-module wrapper outputs: quantize-dequantized weights written in+ place, plus ``scale``/``zp``/``q_scale_thresh``/``data_type``+ attributes matching the unwrapper's conventions. On a mid-batch+ failure (e.g. OOM) already-written modules keep their results and+ only the remainder is returned -- per-module fallback must never+ re-quantize an already quantized-dequantized weight.+ """+ m0 = mods[0]+ bits, g = m0.bits, m0.group_size+ sym = bool(getattr(m0, "sym", False))+ # mirror the wrapper's threshold rule: fp32-scale modules use a tighter+ # q_scale_thresh than the fp16 default+ scale_dtype = getattr(m0, "scale_dtype", torch.float16)+ q_scale_thresh = 1e-8 if scale_dtype == torch.float32 else 1e-5+ resolved_dtype = "opt_rtn_int_sym_neuqi" if sym else "opt_rtn_int_asym"+ try:+ if sym:+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_sym_neuqi+ else:+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_asym+ except ImportError:+ return list(mods)++ max_elems = _EXPERT_BATCH_MAX_ELEMS+ per_call = max(1, max_elems // m0.weight.numel())+ # divisibility is guaranteed by _split_expert_batches; re-checked here+ # so a direct call can never half-write a chunk before bailing out+ if any(m.weight.shape[1] % g != 0 for m in mods):+ return list(mods)+ written = 0 # modules fully quantized so far (chunk writes are atomic)+ try:+ for s in range(0, len(mods), per_call):+ chunk = mods[s : s + per_call]+ dev = torch.device(device)+ weights = torch.cat([m.weight.data.reshape(m.weight.shape[0], -1) for m in chunk], dim=0).to(dev)+ imat_rows = []+ for m in chunk:+ if hasattr(m, "imatrix"):+ imat_rows.append(m.imatrix.to(dev).unsqueeze(0).expand(m.weight.shape[0], -1))+ if imat_rows and len(imat_rows) < len(chunk): # mixed: fill gaps uniformly+ it = iter(imat_rows)+ imat_rows = [+ next(it) if hasattr(m, "imatrix") else torch.ones(m.weight.shape, device=dev) for m in chunk+ ]+ # uniform weighting needs no materialized tensor: qw=None is+ # bit-identical to a full ones imatrix and skips ~module-size+ # allocations per expert+ imat = torch.cat(imat_rows, dim=0) if imat_rows else None+ if dev.type == "cuda":+ torch.cuda.synchronize(dev)++ if sym:+ # returns follow the quant_tensor_opt_rtn_sym conventions: the+ # zero point is the scalar nmax+ qdq, scale, zp = quant_tensor_opt_rtn_sym_neuqi(+ weights,+ bits=bits,+ group_size=g,+ v=0.0,+ q_scale_thresh=q_scale_thresh,+ imatrix=imat,+ scale_dtype=scale_dtype,+ )+ else:+ qdq, scale, zp = quant_tensor_opt_rtn_asym(+ weights,+ bits=bits,+ group_size=g,+ v=0.0,+ q_scale_thresh=q_scale_thresh,+ imatrix=imat,+ scale_dtype=scale_dtype,+ )++ if dev.type == "cuda":+ torch.cuda.synchronize(dev)+ row_w = row_g = 0+ with torch.no_grad():+ for m in chunk:+ out = m.weight.shape[0]+ n_groups_m = out * (m.weight.shape[1] // g)+ m.weight.data.copy_(qdq[row_w : row_w + out].reshape(m.weight.shape).to(m.weight.device))+ m.scale = scale[row_g : row_g + n_groups_m].reshape(out, -1).to("cpu")+ if sym:+ m.zp = zp+ else:+ m.zp = zp[row_g : row_g + n_groups_m].reshape(out, -1).to("cpu")+ m.q_scale_thresh = q_scale_thresh+ m.data_type = resolved_dtype+ m.weight.grad = None+ row_w += out+ row_g += n_groups_m+ written = s + len(chunk)+ if dev.type == "cuda":+ torch.cuda.synchronize(dev)+ except torch.OutOfMemoryError:+ # fall back to per-module calls for the unwritten remainder only:+ # smaller transients, identical results, no double quantization+ pass+ return mods[written:]++ def _quantize_targets(self, targets: list) -> None:+ """Quantize a block's target modules; same-shape expert groups in one+ stacked search call each (row-independent search: results identical to+ per-module calls)."""+ if not targets:+ return+ batches = []+ if self._expert_search_active():+ batches, targets = self._split_expert_batches(targets)+ if batches:+ n_batched = sum(len(b) for b in batches)+ logger.info(+ "[OptRTN] expert batching: %d expert modules in %d batched groups.",+ n_batched,+ len(batches),+ )+ device = device_manager.device+ for b in batches:+ for m in self._quantize_expert_batch(b, device):+ self.quantize_layer_outside_block(m) # unwritten remainder: per-module fallback+ for m in targets:+ self.quantize_layer_outside_block(m)diff --git a/auto_round/algorithms/quantization/sign_round/config.py b/auto_round/algorithms/quantization/sign_round/config.pyindex 03de8728e..a362a34e1 100644--- a/auto_round/algorithms/quantization/sign_round/config.py+++ b/auto_round/algorithms/quantization/sign_round/config.py@@ -120,6 +120,16 @@ class SignRoundConfig(QuantizationConfig):action="boolean_optional",help="Enable last-block LM cross-entropy (LFQ) loss for the final transformer block (experimental).",)+ registry.add_argument(+ "--enable_neuqi",+ field="enable_neuqi",+ action="store_true",+ help=(+ "Enable the NeUQI grid search (arXiv 2505.17595): the tuning grid is anchored to "+ "the searched (scale, zero-point) result once at layer init and frozen during "+ "tuning (joint search for asym layers, two-stage scale search for sym layers)."+ ),+ )def __init__(self,@@ -140,6 +150,7 @@ class SignRoundConfig(QuantizationConfig):optimizer: str | None = None, # TODO later wenhuach delete thisenable_adam: bool = False, # TODO later wenhuach delete thisenable_lfq: bool = False,+ enable_neuqi: bool = False,**kwargs,) -> None:"""Initialize a SignRound configuration.@@ -170,11 +181,18 @@ class SignRoundConfig(QuantizationConfig):quantized output of previous blocks during calibration.optimizer: Optional optimizer name override.enable_adam: Whether to use the Adam-based SignRound variant.+ enable_neuqi: Opt into the NeUQI grid search (arXiv 2505.17595):+ the joint (scale, integer zero-point) search runs once at+ layer init and anchors the tuning grid to the winner, frozen+ during tuning (the two-stage symmetric scale search for sym+ layers). The search grid sizes are tunable via+ ``AR_NEUQI_COARSE`` and ``AR_NEUQI_FINE``.**kwargs: Common quantization arguments forwarded toQuantizationConfig, such as bits, group_size, sym,data_type, and activation quantization fields."""super().__init__(**kwargs)+ self.enable_neuqi = enable_neuqiself.gradient_accumulate_steps = gradient_accumulate_stepsself.iters = itersif self.iters < 0:diff --git a/auto_round/algorithms/quantization/sign_round/quantizer.py b/auto_round/algorithms/quantization/sign_round/quantizer.pyindex 116e5b2e9..e5a7e0257 100644--- a/auto_round/algorithms/quantization/sign_round/quantizer.py+++ b/auto_round/algorithms/quantization/sign_round/quantizer.py@@ -374,6 +374,7 @@ class SignRoundQuantizer(BaseQuantizer):self.enable_norm_bias_tuning,enable_torch_compile=self.compress_context.enable_torch_compile,device=device,+ enable_neuqi=getattr(self.config, "enable_neuqi", False),)round_params = []@@ -666,6 +667,7 @@ class SignRoundQuantizer(BaseQuantizer):enable_minmax_tuning=self.enable_minmax_tuning,enable_torch_compile=self.compress_context.enable_torch_compile,device=device,+ enable_neuqi=getattr(self.config, "enable_neuqi", False),).to(device)round_params = []minmax_params = []diff --git a/auto_round/autoround.py b/auto_round/autoround.pyindex 22baaf226..4826d1b0c 100644--- a/auto_round/autoround.py+++ b/auto_round/autoround.py@@ -187,6 +187,12 @@ def _select_rtn_compressor_base_cls(quant_config: "RTNConfig", scheme, format, bbits = resolved_attrs.get("bits")if sym is not None and sym is False:enable_imatrix = False+ # enable_neuqi: the joint (scale, zero-point) search+ # consumes the activation imatrix as a per-element weighting,+ # so it must be collected even on the asymmetric path where+ # the plain min/max initialization ignores it+ if getattr(quant_config, "enable_neuqi", False):+ enable_imatrix = Trueelif data_type == "int" and (bits is None or bits < 8):enable_imatrix = Trueelif is_weight_scheme(scheme):diff --git a/auto_round/data_type/__init__.py b/auto_round/data_type/__init__.pyindex 58c746470..f51cb80ed 100644--- a/auto_round/data_type/__init__.py+++ b/auto_round/data_type/__init__.py@@ -24,3 +24,4 @@ from auto_round.data_type.utils import ()import auto_round.data_type.nvfpimport auto_round.data_type.gguf+import auto_round.data_type.neuqidiff --git a/auto_round/data_type/neuqi.py b/auto_round/data_type/neuqi.pynew file mode 100644index 000000000..d83afeb6d--- /dev/null+++ b/auto_round/data_type/neuqi.py@@ -0,0 +1,1439 @@+# Copyright (c) 2026 Intel Corporation+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+"""Near-optimal uniform (scale, zero-point) search for asymmetric integer quantization.++This is a clean-room reimplementation of the integer-zero-point variant of the NeUQI+objective (arXiv 2505.17595, "NeUQI: Near-Optimal Uniform Quantization Parameter+Initialization for Low-Bit LLMs"). The reference implementation carries no license,+so no code is copied from it; this module implements the published optimization from+scratch. Only the integer-zero-point variant is implemented because the paper's+floating-point zero point is not consumable by stock int4 kernels (GPTQ/AWQ/Marlin+assume an integral zero point).++For each quantization group the following problem is solved:++ min_{s > 0, z in {0, .., 2**bits - 1}}+ sum_i qw_i * (s * (clamp(round(w_i / s) + z, 0, 2**bits - 1) - z) - w_i)**2++where ``qw`` is an optional per-element weighting (e.g. the activation ``imatrix``).+For a fixed scale the loss decomposes over the integer zero points, so the optimal+``z`` is found exactly by enumerating all ``2**bits`` candidates (16 for 4-bit).+The scale is searched on a coarse log grid bounded by the min-max scale (scales+beyond the min-max range are never optimal) followed by a fine additive refinement+around the best coarse entry.++The search replaces the plain min/max initialization of the asymmetric optimized-RTN+path and is a strict generalization of the symmetric ``search_scales`` grid (which+fixes ``z = 0``).++The symmetric counterpart (:func:`neuqi_search_scale_sym`) fixes the zero point at+0 and searches only the scale on the same two-stage grid; it replaces the uniform+``search_scales`` grid of ``opt_rtn_int_sym`` when engaged via+``enable_neuqi`` with ``sym=True``.++Performance: the per-candidate zero-point sweep is a pure elementwise + reduction chain,+so on CUDA it is fused into a single generated kernel via ``torch.compile``+(``AR_NEUQI_BACKEND=auto``, the default), and all coarse/fine candidates of a pass+are additionally folded into one such kernel per group chunk (the batched sweep)+so the search costs a handful of launches instead of candidates x chunks. On CUDA+the batched chunk is preferentially served by a hand-written Triton kernel+(``auto_round_extension.triton.neuqi_sweep``) that loads each weight element into+registers once and computes every (candidate, zero-point) loss from registers,+removing the L2 re-read amplification of the compiled expression. Measured on an+RTX 3090 (2M groups, g=64, bits=4, 64+32 grid): 12.3 s eager -> 0.59 s fused+per-candidate -> 0.49 s compiled-batched -> 0.22 s Triton (~57x). The fused sweeps+evaluate exactly the same losses on exactly the same candidate grid; only the fp32+summation order inside the reduction may differ, so selections can flip solely on+near-ties: on random, heavy-tailed and imatrix-weighted inputs the flips stay below+~0.1% of groups with worst-case relative loss differences of ~5e-5 (measured on an+RTX 3090; ~1e-6 on CPU), with the Triton kernel matching that tie profile exactly.+The eager chunked sweep remains the reference path (CPU default) and the ultimate+fallback; every stage latches down permanently on failure (Triton -> compiled+batched -> compiled per-candidate -> eager).+"""++import math+import threading+from functools import lru_cache++import torch++from auto_round import envs+from auto_round.data_type.register import register_dtype+from auto_round.data_type.utils import reshape_pad_tensor_by_group_size, revert_tensor_by_pad, round_ste+from auto_round.logger import logger++# Upper bound on the number of elements of the temporary [chunk, group, zero_point]+# loss tensor, used to cap peak memory regardless of the input size. The bound is+# honored by both the eager and the fused sweep, so a dynamo recompile-limit fallback+# to eager execution degrades speed only, never correctness or memory.+_MAX_TMP_ELEMS = 2**25+++def _align_qw(qw, data):+ """Move per-element loss weights onto the data device when they disagree.++ Parallel fan-out workers relocate a module to their GPU for the search;+ a plain-tensor ``imatrix`` attribute does not travel with ``module.to()``,+ leaving qw on the block's home device. Cross-device inputs would trip the+ kernel same-device asserts mid-run (and, under a compiled caller, dynamo's+ fake-tensor device propagation), so align once here.+ """+ if qw is not None and qw.device != data.device:+ qw = qw.to(data.device)+ return qw+++@lru_cache(maxsize=1)+def _log_search_engaged(coarse_n: int, fine_n: int) -> None:+ logger.info("[NeUQI] joint (scale, zero-point) search active (coarse=%d, fine=%d)", coarse_n, fine_n)+++def _coarse_grid(lo: float, n: int, device: torch.device, dtype: torch.dtype, hi: float = 1.0) -> torch.Tensor:+ """Log-spaced coarse scale fractions, always built eagerly.++ torch.logspace lowers to an fp64 ``libdevice.pow`` inside compiled+ regions, and inductor has been observed to fuse that into the batched+ sweep's index-select math, where Triton rejects fp64 in the integer+ index guards ("unexpected type fp64" -- BPT worker crash, first MoE+ block under ``--enable_torch_compile``). Building the tiny [n] grid+ eagerly keeps it a plain tensor input to any traced region.++ Deliberately NOT cached: the expert search fans out from a thread pool+ (one thread per device), and a cached tensor would be created on the+ first caller's stream and read from the others' streams with no event+ ordering -- a cross-stream read-before-write window that surfaced as a+ device-side assert on the streaming parent (hidden under+ CUDA_LAUNCH_BLOCKING=1). Recreating 64 floats per call is free and+ keeps creation on the calling thread's own stream.+ """+ return torch.logspace(math.log10(lo), math.log10(hi), n, device=device, dtype=dtype)+++def _zp_expr_last(data, qw, scale, zp_grid, maxq):+ """Zero-point sweep with the group dimension last: ``[C, Z, g]``, sum over ``dim=-1``.++ Reduction over the contiguous last dimension is the canonical fused shape for+ the Triton/CUDA codegen: the scheduler can keep the whole pointwise chain+ (round -> clamp -> dequant -> squared error -> group sum -> zero-point min) in+ registers and stream the group axis with coalesced loads. The equivalent+ middle-dim reduction (``[C, g, Z]``) measured no faster than eager on CUDA+ (RTX 3090) because the expanded buffer must be materialized or transposed to+ reduce a non-contiguous axis; it is kept as ``_zp_expr_mid`` where it is the+ faster layout (CPU, including eager execution).++ Same operands and operand order per logical (group, zero-point) element as the+ reference sweep, so losses are identical up to the fp32 summation order over+ the group (last-ulp ties only).++ Args:+ data: [chunk, g] float32 group data.+ qw: [chunk, g] float32 per-element weights or ``None``.+ scale: [chunk, 1] float32 scale candidate.+ zp_grid: [n_zp] float32 integral zero-point candidates ``0 .. maxq``.+ maxq: maximum integer value (``2**bits - 1``).++ Returns:+ (min_loss [chunk], argmin_zp [chunk]) with the first-minimum tie rule of+ ``Tensor.min`` (same winner as the eager sweep on equal losses).+ """+ # Stabilize rounding: values far outside the grid saturate identically after+ # clamping, so clamping r first never changes clamp(r + z, 0, maxq) for z >= 0.+ # Broadcasts give [C, Z, g]: data/qw over the middle (stride-0) zero-point+ # axis, scale shared across the group, reduction over the last (contiguous) dim.+ d = data.unsqueeze(-2) # [C, 1, g]+ sc = scale.unsqueeze(-1) # [C, 1, 1]+ zp = zp_grid.view(1, -1, 1) # [1, Z, 1]+ r = torch.round(d / sc).clamp_(min=-maxq - 1, max=2 * maxq + 1)+ q = (r + zp).clamp_(0, maxq)+ deq = sc * (q - zp)+ diff = deq - d+ loss = diff * diff+ if qw is not None:+ loss = loss * qw.unsqueeze(-2)+ loss = loss.sum(dim=-1) # [C, Z]+ return loss.min(dim=-1)+++def _zp_expr_mid(data, qw, scale, zp_grid, maxq):+ """Zero-point sweep with the zero-point dimension last: ``[C, g, Z]``, sum over ``dim=1``.++ Byte-identical operand order to the historical chunked sweep. This is the+ faster layout for eager execution and for the compiled CPU backend (the+ zero-point broadcast rides the contiguous last axis, which TensorIterator+ vectorizes well); it loses to ``_zp_expr_last`` under the Triton/CUDA+ scheduler for the reasons given above.++ Same args and returns as ``_zp_expr_last``.+ """+ # Stabilize rounding: values far outside the grid saturate identically after+ # clamping, so clamping r first never changes clamp(r + z, 0, maxq) for z >= 0.+ r = torch.round(data / scale).clamp_(min=-maxq - 1, max=2 * maxq + 1)+ q = (r.unsqueeze(-1) + zp_grid).clamp_(0, maxq)+ deq = scale.unsqueeze(-1) * (q - zp_grid)+ diff = deq - data.unsqueeze(-1)+ loss = diff * diff+ if qw is not None:+ loss = loss * qw.unsqueeze(-1)+ loss = loss.sum(dim=1) # [chunk, n_zp]+ return loss.min(dim=-1)+++def _zp_expr_batched(data, qw, scales, zp_grid, maxq):+ """All-candidates zero-point sweep in one call: ``[C, K, Z, g]``, group last.++ Evaluates every per-group scale candidate (``scales``, [C, K]) against every+ integer zero point in a single fused kernel: the group-axis reduction and the+ min over zero points are fused in-kernel, so the kernel outputs only the+ [C, K] best losses and [C, K] winning zero points -- one launch covers what+ the per-candidate sweep needs K launches and K bookkeeping rounds for. This+ removes the Python-dispatch bound of the per-candidate loop (the residual+ cost after single-candidate fusion) at the price of a larger symbolic+ intermediate, which is why eager execution of this expression is never used+ (see ``_eval_batched_chunk``).++ Same per-(group, candidate, zero-point) operand order as ``_zp_expr_last``;+ the caller applies the first-minimum-over-candidates rule with ``Tensor.min``,+ matching the sequential sweep's strict-improvement tie semantics.++ Args:+ data: [chunk, g] float32 group data.+ qw: [chunk, g] float32 per-element weights or ``None``.+ scales: [chunk, K] float32 per-group scale candidates.+ zp_grid: [n_zp] float32 integral zero-point candidates ``0 .. maxq``.+ maxq: maximum integer value (``2**bits - 1``).++ Returns:+ (best_loss [chunk, K], best_zp [chunk, K]) -- per-candidate minimum over+ zero points, first-minimum tie rule.+ """+ d = data.unsqueeze(-2).unsqueeze(-2) # [C, 1, 1, g]+ sc = scales.unsqueeze(-1).unsqueeze(-1) # [C, K, 1, 1]+ zp = zp_grid.view(1, 1, -1, 1) # [1, 1, Z, 1]+ r = torch.round(d / sc).clamp_(min=-maxq - 1, max=2 * maxq + 1)+ q = (r + zp).clamp_(0, maxq)+ deq = sc * (q - zp)+ diff = deq - d+ loss = diff * diff+ if qw is not None:+ loss = loss * qw.unsqueeze(-2).unsqueeze(-2)+ loss = loss.sum(dim=-1) # [C, K, Z]+ return loss.min(dim=-1) # [C, K], [C, K]+++def _zp_expr_for(device_type: str):+ """Pick the sweep layout for a device.++ The last-dim-group reduction on CUDA (canonical Triton fusion shape),+ the zero-point-last layout elsewhere (faster under TensorIterator and the+ compiled CPU backend).+ """+ return _zp_expr_last if device_type == "cuda" else _zp_expr_mid+++# torch.compile'd zero-point sweeps, keyed by the expression (layout) -- lazily+# created on first CUDA use / forced backend.+_fused_zp_fns = {}+# Set when a fused invocation raised (e.g. no host C++ compiler for Inductor);+# the process then permanently uses the eager sweep.+_fused_zp_broken = False+# Set when the all-candidates batched sweep raised; the process then permanently+# uses the per-candidate fused sweep.+_batched_broken = False+++def _zp_wants_compile(device_type: str) -> bool:+ """Whether a fused torch.compile sweep should be used, per ``AR_NEUQI_BACKEND``."""+ if envs.AR_NEUQI_BACKEND == "eager":+ return False+ backend = envs.AR_NEUQI_BACKEND+ if backend in ("compile", "triton"):+ return True+ if backend == "auto":+ return device_type == "cuda"+ return False # "eager" (and any unrecognized value): reference sweep+++# Extension Triton sweep (auto_round_extension.triton.neuqi_sweep), lazily+# resolved once; ``None`` when the extension/triton is unavailable on the host.+_triton_sweep = None+_triton_checked = False+_triton_broken = False+_neuqi_shared_sweep = None+_neuqi_shared_checked = False+_neuqi_shared_broken = False+++def _triton_sweep_fn():+ """Resolve the extension Triton sweep once; ``None`` when unavailable."""+ global _triton_sweep, _triton_checked+ if not _triton_checked:+ _triton_checked = True+ try:+ from auto_round_extension.triton.neuqi_sweep import neuqi_sweep_triton++ _triton_sweep = neuqi_sweep_triton+ logger.debug("[NeUQI] Triton zero-point sweep engaged (auto_round_extension.triton.neuqi_sweep)")+ except Exception as e:+ logger.info("[NeUQI] Triton sweep unavailable (%s); using the torch.compile sweeps", e)+ _triton_sweep = None+ return _triton_sweep+++def _neuqi_shared_sweep_fn():+ """Resolve the extension shared-multiplier coarse kernel once."""+ global _neuqi_shared_sweep, _neuqi_shared_checked+ if not _neuqi_shared_checked:+ _neuqi_shared_checked = True+ try:+ from auto_round_extension.triton.neuqi_sweep import neuqi_shared_sweep_triton++ _neuqi_shared_sweep = neuqi_shared_sweep_triton+ logger.debug(+ "[NeUQI] Triton shared-multiplier coarse sweep engaged (auto_round_extension.triton.neuqi_sweep)"+ )+ except Exception as e:+ _neuqi_shared_sweep = None+ logger.info("[NeUQI] shared coarse sweep unavailable (%s); using the batched path", e)+ return _neuqi_shared_sweep+++def _neuqi_shared_attempt(dn, qw, fracs, invf, maxq):+ """Try the shared-multiplier coarse kernel; ``None`` (and a latch) on failure."""+ global _neuqi_shared_broken+ if _neuqi_shared_broken or not dn.is_cuda:+ return None+ if not _zp_wants_triton(dn.device.type) or not _zp_batch_wanted(dn.device.type):+ return None+ fn = _neuqi_shared_sweep_fn()+ if fn is None:+ return None+ try:+ return fn(dn, qw, fracs, invf, maxq)+ except Exception as e:+ _neuqi_shared_broken = True+ torch.cuda.empty_cache() # a failed launch may hold pooled memory+ logger.warning("[NeUQI] shared coarse sweep failed (%s); using the batched path", e)+ return None+++def _zp_coarse_pass_shared(data, qw, s0, coarse, maxq, launch=None):+ """Coarse pass of the joint search in normalized space (shared-multiplier grid).++ Computes dn = w / s0 once, evaluates every group against the SHARED frac+ grid through ``launch`` (default: the gated Triton attempt), and+ unnormalizes each winner's loss by s0^2 so the fine stage can compare+ across stages. Returns (best_loss, best_frac, best_zp) or ``None`` when+ the kernel is unavailable -- the caller then serves the identical+ candidate grid through the batched / per-candidate paths.++ ``launch`` is injectable so the driver math (normalization, s0^2+ unnormalization, frac/zp mapping) stays unit-testable on CPU.+ """+ n_groups = data.shape[0]+ if launch is None:+ # pre-gate before any tensor math: CPU / non-triton backends must not+ # pay for the normalized copy+ if (+ _neuqi_shared_broken+ or not data.is_cuda+ or not _zp_wants_triton(data.device.type)+ or not _zp_batch_wanted(data.device.type)+ ):+ return None+ launch = _neuqi_shared_attempt+ dn = data / s0+ invf = 1.0 / coarse+ best_loss = torch.full((n_groups,), float("inf"), device=data.device, dtype=torch.float32)+ best_frac = torch.ones(n_groups, device=data.device, dtype=torch.float32)+ best_zp = torch.zeros(n_groups, device=data.device, dtype=torch.float32)+ chunk = 262144+ for start in range(0, n_groups, chunk):+ stop = min(start + chunk, n_groups)+ out = launch(+ dn[start:stop].contiguous(),+ qw[start:stop].contiguous() if qw is not None else None,+ coarse,+ invf,+ maxq,+ )+ if out is None:+ return None+ loss_n, k_idx, zp = out+ loss = loss_n * s0[start:stop].squeeze(-1) ** 2+ frac = coarse.index_select(0, k_idx)+ best_loss[start:stop] = loss+ best_frac[start:stop] = frac+ best_zp[start:stop] = zp.to(torch.float32)+ return best_loss, best_frac, best_zp+++_sweep_warm_lock = threading.Lock()+_sweep_warmed_devices: set = set() # device indices whose Triton sweep was warmed once+++def ensure_sweep_warmup(device) -> None:+ """Serialized, synchronized first launch of the extension Triton sweep.++ The first Triton launch JIT-compiles the kernel, initializes the launcher+ and writes the on-disk cache; doing that while unrelated GPU work is in+ flight (e.g. inductor compile workers active during SignRound wrapper+ init, or several fan-out threads first-hitting the sweep at once) has been+ observed to leave the sweep producing corrupt selections that surface as+ out-of-bounds gather indices downstream (device-side asserts, vanishing+ under CUDA_LAUNCH_BLOCKING=1). Warming once per device -- under a lock,+ followed by a full device synchronize -- makes every subsequent launch a+ cached, race-free launch. Failures latch the existing Triton-broken+ fallback instead of raising.+ """+ if device.type != "cuda":+ return+ global _triton_broken, _neuqi_shared_broken+ with _sweep_warm_lock:+ if device.index in _sweep_warmed_devices:+ return+ _sweep_warmed_devices.add(device.index)+ engaged = []+ if _zp_wants_triton(device.type):+ fn = _triton_sweep_fn()+ if fn is not None:+ try:+ data = torch.randn(4, 128, device=device, dtype=torch.float32)+ scales = torch.rand(4, 2, device=device, dtype=torch.float32) * 0.01 + 0.001+ with torch.cuda.device(device): # Triton launches on the CURRENT device+ loss, _zp = fn(data, torch.ones_like(data), scales, 15)+ torch.cuda.synchronize(device)+ if not torch.isfinite(loss).all():+ raise RuntimeError("warmup produced non-finite losses")+ logger.debug("[NeUQI] Triton sweep warmed on %s", device)+ engaged.append("zero-point sweep")+ except Exception as e: # noqa: BLE001 warmup failure must never kill a run+ _triton_broken = True+ logger.warning(+ "[NeUQI] Triton sweep warmup failed on %s (%s); using torch.compile sweeps", device, e+ )+ if _zp_wants_triton(device.type) and _zp_batch_wanted(device.type):+ fn = _neuqi_shared_sweep_fn()+ if fn is not None:+ try:+ dn = torch.randn(4, 128, device=device, dtype=torch.float32)+ fracs = torch.rand(8, device=device, dtype=torch.float32) * 0.5 + 0.5+ with torch.cuda.device(device):+ loss, _k, _z = fn(dn, torch.ones_like(dn), fracs, 1.0 / fracs, 15)+ torch.cuda.synchronize(device)+ if not torch.isfinite(loss).all():+ raise RuntimeError("warmup produced non-finite losses")+ logger.debug("[NeUQI] shared coarse sweep warmed on %s", device)+ engaged.append("shared coarse sweep")+ except Exception as e: # noqa: BLE001 warmup failure must never kill a run+ _neuqi_shared_broken = True+ logger.warning(+ "[NeUQI] shared coarse sweep warmup failed on %s (%s); using the batched path", device, e+ )++ if engaged:+ logger.info(+ "[NeUQI] Triton backend engaged for the joint search on %s (kernels: %s)", device, ", ".join(engaged)+ )+++def _zp_wants_triton(device_type: str) -> bool:+ """Whether the extension Triton sweep should be tried, per ``AR_NEUQI_BACKEND``."""+ if envs.AR_NEUQI_BACKEND == "triton":+ return True # explicit user intent outranks the wrapper-init pin+ if envs.AR_NEUQI_BACKEND in ("compile", "eager"):+ return False+ backend = envs.AR_NEUQI_BACKEND+ if backend == "triton":+ return True+ return backend == "auto" and device_type == "cuda"+++def _get_fused_zp_fn(expr):+ """Lazily compile one sweep layout; callers fall back to eager on any failure."""+ fn = _fused_zp_fns.get(expr)+ if fn is None:+ from auto_round.utils.device import _bump_dynamo_cache_limit++ # Shape variants (group size x bits x qw-present x chunk tail) must never+ # trip dynamo's recompile limit: exceeding it silently runs the frame in+ # eager, which is memory-safe here (chunking bounds the expansion) but slow.+ _bump_dynamo_cache_limit(64)+ fn = torch.compile(expr, dynamic=False)+ _fused_zp_fns[expr] = fn+ logger.info("[NeUQI] zero-point sweep fused via torch.compile (layout=%s)", expr.__name__)+ return fn+++def _default_grid(device_type: str):+ """Default (coarse, fine) grid for the search entries.++ The wide 256/64 grid only pays off on the accelerated sweep lanes+ (extension Triton kernels / torch.compile); the eager sweep evaluates+ ~coarse x fine candidates per group and is ~8x slower at 256/64 than at+ 64/32, while the measured quality ladder is flat between the two+ (Qwen3.8-27B W4A16 g128: mean KL 0.02570 @ 64/32 vs 0.02579 @ 256/64).+ Narrow the default whenever the backend policy does not select an+ accelerated path; explicit ``AR_NEUQI_COARSE``/``AR_NEUQI_FINE`` pins+ override this everywhere."""+ fast = _zp_wants_compile(device_type) or _zp_wants_triton(device_type)+ return (256, 64) if fast else (64, 32)+++def _eval_zp_chunk(data, qw, scale, zp_grid, maxq):+ """Dispatch one chunk of the zero-point sweep to the fused or eager path."""+ global _fused_zp_broken+ expr = _zp_expr_for(data.device.type)+ if not _fused_zp_broken and _zp_wants_compile(data.device.type):+ try:+ return _get_fused_zp_fn(expr)(data, qw, scale, zp_grid, maxq)+ except Exception as e: # pragma: no cover - depends on host toolchain+ _fused_zp_broken = True+ logger.warning("[NeUQI] fused zero-point sweep failed (%s); using the eager sweep", e)+ return expr(data, qw, scale, zp_grid, maxq)+++def _zp_batch_wanted(device_type: str) -> bool:+ """Whether the all-candidates batched sweep should run.++ Batches on CUDA when the fused backend engages there. The batched+ expression is only ever executed compiled: eager execution of+ [C, K, Z, g] would materialize the full expansion.+ """+ return device_type == "cuda" and _zp_wants_compile(device_type)+++def _eval_batched_chunk(data, qw, scales, zp_grid, maxq):+ """One chunk of the all-candidates batched sweep, or ``None`` when unavailable.++ Preference order: the extension Triton sweep (registers-resident, one data+ read per element -- the fastest evaluator on CUDA), then the compiled+ batched expression, then ``None`` so the caller falls back to the+ per-candidate sweep. Strict-improvement bookkeeping makes a partial pass+ followed by a fallback converge to the same selections (ties within the+ usual fp32 summation tolerance).+ """+ global _triton_broken, _batched_broken+ if not _triton_broken and _zp_wants_triton(data.device.type) and _zp_batch_wanted(data.device.type):+ fn = _triton_sweep_fn()+ if fn is not None:+ try:+ return fn(data, qw, scales, maxq)+ except Exception as e:+ _triton_broken = True+ if data.device.type == "cuda":+ torch.cuda.empty_cache()+ logger.warning("[NeUQI] Triton sweep failed (%s); using the torch.compile sweeps", e)+ if _batched_broken or not _zp_wants_compile(data.device.type):+ return None+ try:+ return _get_fused_zp_fn(_zp_expr_batched)(data, qw, scales, zp_grid, maxq)+ except Exception as e:+ _batched_broken = True+ if data.device.type == "cuda":+ torch.cuda.empty_cache() # a failed launch may hold pooled memory+ logger.warning("[NeUQI] batched zero-point sweep failed (%s); using the per-candidate sweep", e)+ return None+++def _best_zp_for_scale(data, qw, scale, zp_grid, maxq, chunk):+ """Evaluate all integer zero points for one per-group scale candidate.++ Args:+ data: [N, g] float32 group data.+ qw: [N, g] float32 per-element weights or ``None``.+ scale: [N, 1] float32 scale candidates.+ zp_grid: [n_zp] float32 integral zero-point candidates.+ maxq: maximum integer value (``2**bits - 1``).+ chunk: number of groups per evaluation slice (bounds the temporary).++ Returns:+ best_loss: [N] float32, loss of the best zero point per group.+ best_zp: [N] float32, integral-valued optimal zero point per group.+ """+ n_groups = data.shape[0]+ best_loss = torch.full((n_groups,), float("inf"), device=data.device, dtype=data.dtype)+ best_zp = torch.zeros(n_groups, device=data.device, dtype=data.dtype)++ for start in range(0, n_groups, chunk):+ stop = min(start + chunk, n_groups)+ min_loss, argmin_zp = _eval_zp_chunk(+ data[start:stop],+ qw[start:stop] if qw is not None else None,+ scale[start:stop],+ zp_grid,+ maxq,+ )+ best_loss[start:stop] = min_loss+ best_zp[start:stop] = argmin_zp.to(data.dtype)++ return best_loss, best_zp+++def _lower_scale_ratio(bits: int) -> float:+ """Fraction of the min-max scale below which clipping is never beneficial."""+ if bits >= 4:+ return 0.25+ if bits == 3:+ return 0.15+ return 0.05++[email protected] # loop-heavy eager search: never trace into dynamo graphs+def neuqi_search_scale_zero(data, bits, qw=None, q_scale_thresh=1e-5, coarse_n=None, fine_n=None):+ """Joint near-optimal (scale, integer zero-point) search per quantization group.++ Args:+ data: [N, g] float32 tensor of already-grouped weights.+ bits: quantization bit width.+ qw: optional [N, g] float32 per-element loss weights (e.g. imatrix).+ q_scale_thresh: minimum scale magnitude.+ coarse_n: number of coarse log-spaced scale candidates (env ``AR_NEUQI_COARSE``).+ fine_n: number of fine additive candidates around the best coarse entry+ (env ``AR_NEUQI_FINE``).++ Returns:+ scale: [N, 1] float32 per-group scales.+ zp: [N, 1] float32 per-group integral-valued zero points.+ """+ if coarse_n is None or fine_n is None:+ d_coarse, d_fine = _default_grid(data.device.type)+ if coarse_n is None:+ coarse_n = envs.AR_NEUQI_COARSE or d_coarse+ if coarse_n < 2:+ raise ValueError("AR_NEUQI_COARSE=%d is too small; need at least 2 candidates." % coarse_n)+ if fine_n is None:+ fine_n = envs.AR_NEUQI_FINE or d_fine+ if fine_n < 2:+ raise ValueError("AR_NEUQI_FINE=%d is too small; need at least 2 candidates." % fine_n)+ _log_search_engaged(coarse_n, fine_n)+ if torch.is_tensor(data):+ ensure_sweep_warmup(data.device)++ maxq = int(2**bits) - 1+ data = data.to(torch.float32)+ if qw is not None:+ qw = _align_qw(qw, data).to(torch.float32)++ zp_grid = torch.arange(0, maxq + 1, device=data.device, dtype=data.dtype)+ chunk = max(1, _MAX_TMP_ELEMS // (data.shape[1] * (maxq + 1)))++ wmin = torch.clamp(data.min(dim=-1).values, max=0).unsqueeze(-1)+ wmax = torch.clamp(data.max(dim=-1).values, min=0).unsqueeze(-1)+ s0 = ((wmax - wmin) / maxq).clamp_(min=q_scale_thresh) # [N, 1] min-max scale++ lo = _lower_scale_ratio(bits)+ coarse = _coarse_grid(lo, coarse_n, data.device, torch.float32)++ best_loss = torch.full((data.shape[0],), float("inf"), device=data.device, dtype=torch.float32)+ best_frac = torch.ones(data.shape[0], device=data.device, dtype=torch.float32)+ best_zp = torch.zeros(data.shape[0], device=data.device, dtype=torch.float32)+ n_groups = data.shape[0]+ batch_wanted = _zp_batch_wanted(data.device.type)++ # Coarse pass: shared log-spaced fractions of the per-group min-max scale.+ # Fastest path: the shared-multiplier Triton kernel evaluates the whole+ # grid in normalized space (dn = w / s0; one shared 1/frac multiply per+ # candidate, argmin over (k, z) in-kernel). Batched fast path next: every+ # candidate in one fused call per group chunk (the batched sweep); the+ # per-candidate loop is the reference and the fallback.+ shared_out = _zp_coarse_pass_shared(data, qw, s0, coarse, maxq)+ coarse_done = False+ if shared_out is not None:+ best_loss, best_frac, best_zp = shared_out+ coarse_done = True+ elif batch_wanted:+ chunk_b = max(1, _MAX_TMP_ELEMS // (coarse_n * (maxq + 1)))+ for start in range(0, n_groups, chunk_b):+ stop = min(start + chunk_b, n_groups)+ scales = s0[start:stop] * coarse.view(1, -1) # [chunk, K] shared fracs+ out = _eval_batched_chunk(+ data[start:stop], qw[start:stop] if qw is not None else None, scales, zp_grid, maxq+ )+ if out is None:+ break # latched failure / fused backend off: redo per candidate+ loss_k, zp_k = out # [chunk, K] each+ loss, k_idx = loss_k.min(dim=1) # first-min over candidates+ zp = zp_k.gather(1, k_idx.unsqueeze(1)).squeeze(1)+ frac = coarse.index_select(0, k_idx)+ improved = loss < best_loss[start:stop]+ best_loss[start:stop] = torch.where(improved, loss, best_loss[start:stop])+ best_zp[start:stop] = torch.where(improved, zp, best_zp[start:stop])+ best_frac[start:stop] = torch.where(improved, frac, best_frac[start:stop])+ else:+ coarse_done = True+ if not coarse_done:+ for idx in range(coarse_n):+ scale = s0 * coarse[idx]+ loss, zp = _best_zp_for_scale(data, qw, scale, zp_grid, maxq, chunk)+ improved = loss < best_loss+ best_loss = torch.where(improved, loss, best_loss)+ best_zp = torch.where(improved, zp, best_zp)+ best_frac = torch.where(improved, torch.full_like(best_frac, coarse[idx]), best_frac)++ # Fine pass: additive grid between the coarse neighbors bracketing each winner.+ best_idx = torch.argmin(torch.abs(coarse.unsqueeze(0) - best_frac.unsqueeze(1)), dim=1)+ frac_lo = coarse[(best_idx - 1).clamp_(min=0)]+ frac_hi = coarse[(best_idx + 1).clamp_(max=coarse_n - 1)]++ steps = torch.arange(1, fine_n + 1, device=data.device, dtype=torch.float32) / (fine_n + 1)+ fine_done = fine_n == 0+ if batch_wanted and not fine_done:+ # per-group fracs: frac_lo/hi differ per group, steps shared+ one_m_steps = 1.0 - steps+ chunk_b = max(1, _MAX_TMP_ELEMS // (fine_n * (maxq + 1)))+ for start in range(0, n_groups, chunk_b):+ stop = min(start + chunk_b, n_groups)+ fracs = frac_lo[start:stop].unsqueeze(1) * one_m_steps + frac_hi[start:stop].unsqueeze(1) * steps+ scales = s0[start:stop] * fracs # [chunk, K]+ out = _eval_batched_chunk(+ data[start:stop], qw[start:stop] if qw is not None else None, scales, zp_grid, maxq+ )+ if out is None:+ break+ loss_k, zp_k = out+ loss, k_idx = loss_k.min(dim=1)+ zp = zp_k.gather(1, k_idx.unsqueeze(1)).squeeze(1)+ frac = fracs.gather(1, k_idx.unsqueeze(1)).squeeze(1)+ improved = loss < best_loss[start:stop]+ best_loss[start:stop] = torch.where(improved, loss, best_loss[start:stop])+ best_zp[start:stop] = torch.where(improved, zp, best_zp[start:stop])+ best_frac[start:stop] = torch.where(improved, frac, best_frac[start:stop])+ else:+ fine_done = True+ if not fine_done:+ for step in steps:+ frac = frac_lo * (1.0 - step) + frac_hi * step+ scale = s0 * frac.unsqueeze(-1)+ loss, zp = _best_zp_for_scale(data, qw, scale, zp_grid, maxq, chunk)+ improved = loss < best_loss+ best_loss = torch.where(improved, loss, best_loss)+ best_zp = torch.where(improved, zp, best_zp)+ best_frac = torch.where(improved, frac, best_frac)++ scale = (best_frac.unsqueeze(-1) * s0).clamp_(min=q_scale_thresh)+ return scale, best_zp.unsqueeze(-1)+++_sym_engaged_logged: set = set()++[email protected] # logging side effect: graph-break here instead of tracing+def _log_sym_search_engaged(coarse_n: int, fine_n: int) -> None:+ """Once per unique grid: grid sweeps cycle configs in one process.++ Set-based memo instead of ``lru_cache`` so dynamo (the search runs inside+ compiled regions under ``--enable_torch_compile``) never has to reason+ about a cache wrapper; the decorator makes the logging call graph-break.+ """+ if (coarse_n, fine_n) in _sym_engaged_logged:+ return+ _sym_engaged_logged.add((coarse_n, fine_n))+ logger.info("[NeUQI] two-stage symmetric scale search active (coarse=%d, fine=%d)", coarse_n, fine_n)+++def _sym_loss_chunk(data, qw, scales, nmax):+ """Batched symmetric-candidate weighted qdq losses for one chunk of groups.++ Evaluates every per-group scale candidate in one vectorized call under BOTH+ symmetric clamp conventions and keeps the better one per (group, candidate):++ * standard: ``q = clamp(round(w / s), -nmax, nmax - 1)``, dequant ``s * q``+ * mirrored: ``q = clamp(round(w / s), -(nmax - 1), nmax)``, dequant ``-s * q``++ The mirrored family is the exact arithmetic of a NEGATIVE scale through the+ standard formula (``round(w / -s)`` flips the rounding sign, ``clamp`` bounds+ swap roles, ``-s * q`` restores magnitude). The incumbent+ ``search_scales`` reaches it implicitly by anchoring on the signed max-abs+ value, so groups whose largest-magnitude element is negative get a negative+ scale -- for skewed groups the one-level clamp asymmetry makes the mirrored+ grid notably cheaper (measured: the worst synthetic groups lose ~2x total+ weighted loss when the family is unreachable). Searching ``|s|`` plus a sign+ per group covers both families explicitly.++ Plain eager math (the reference path): with a single zero-point-free axis+ there is no (candidate, zero-point) expansion, so the chunk is a handful of+ elementwise kernels. Wide grids make the chunking fine-grained enough to+ become launch-bound; ``_sym_loss_chunk_eval`` routes these calls through a+ torch.compile-fused core while this function stays the eager reference.++ Args:+ data: [chunk, g] float32 group data.+ qw: [chunk, g] float32 per-element weights or ``None``.+ scales: [chunk, K] float32 POSITIVE per-group scale candidates.+ nmax: symmetric integer bound (``2 ** (bits - 1)``).++ Returns:+ (best_loss [chunk], best_idx [chunk], best_mirror [chunk] bool) with the+ first-minimum tie rule of ``Tensor.min`` over candidates (matching the+ asymmetric sweep's tie semantics); ``best_mirror`` marks whether the+ winning candidate used the mirrored convention (negative scale).+ """+ d = data.unsqueeze(1) # [C, 1, g]+ sc = scales.unsqueeze(-1) # [C, K, 1]+ r = torch.round(d / sc)+ q_std = r.clamp(min=-nmax, max=nmax - 1)+ e_std = sc * q_std - d+ l_std = e_std * e_std+ q_mir = r.clamp(min=-(nmax - 1), max=nmax)+ e_mir = sc * q_mir - d # dequant of the mirrored family is sc * q_mir (neg scale flipped inside q_mir)+ l_mir = e_mir * e_mir+ if qw is not None:+ w = qw.unsqueeze(1)+ l_std = l_std * w+ l_mir = l_mir * w+ l_std = l_std.sum(dim=-1) # [C, K]+ l_mir = l_mir.sum(dim=-1) # [C, K]+ mirror = l_mir < l_std+ loss = torch.where(mirror, l_mir, l_std)+ best_loss, best_idx = loss.min(dim=-1)+ best_mirror = mirror.gather(1, best_idx.unsqueeze(1)).squeeze(1)+ return best_loss, best_idx, best_mirror+++# Symmetric-search Triton backend (auto_round_extension.triton.neuqi_sweep), lazily+# resolved once; ``None`` when the extension/triton is unavailable on the host.+_sym_triton = None+_sym_triton_checked = False+_sym_triton_broken = False+_sym_shared_triton = None+_sym_shared_checked = False+_sym_shared_broken = False+_sym_search_warmed: set = set()+++def _sym_shared_triton_fn():+ """Resolve the extension shared-multiplier coarse kernel once."""+ global _sym_shared_triton, _sym_shared_checked+ if not _sym_shared_checked:+ _sym_shared_checked = True+ try:+ from auto_round_extension.triton.neuqi_sweep import sym_search_shared_triton++ _sym_shared_triton = sym_search_shared_triton+ logger.debug(+ "[NeUQI] Triton shared-multiplier sym coarse search engaged "+ "(auto_round_extension.triton.neuqi_sweep)"+ )+ except Exception as e:+ _sym_shared_triton = None+ logger.info("[NeUQI] shared sym search unavailable (%s); using the per-candidate path", e)+ return _sym_shared_triton+++def _sym_shared_wants_triton(device_type: str) -> bool:+ """Whether the shared-multiplier coarse kernel should run, per backend rules.++ AUTO resolves the COARSE stage to this dedicated kernel: every group+ shares the same candidate fracs, so normalizing once (dn = w / s0) turns+ the inner loop into one shared 1/frac multiply -- no per-(group, candidate)+ scales materialization and no division in the loop. Measured RTX 3090,+ 2M groups, g=128, 256+64 candidates: 0.24 s vs 0.57-0.79 s for the+ compiled core (eager 8.5 s); parity stays in the tie-flip class (~0.5%+ flipped groups, max fp64 rel diff 1.7e-6 vs eager). The FINE stage keeps+ its per-group candidate grid and stays on the compiled core under AUTO.+ ``compile``/``eager`` overrides keep the reference numerics end to end.+ """+ backend = str(envs.AR_NEUQI_BACKEND or "auto")+ if backend in ("compile", "eager"):+ return False+ return backend in ("auto", "triton") and device_type == "cuda"+++def _sym_shared_triton_attempt(dn, qw, fracs, invf, nmax):+ """Try the shared-multiplier coarse kernel; ``None`` (and a latch) on failure."""+ global _sym_shared_broken+ if _sym_shared_broken or not dn.is_cuda or not _sym_shared_wants_triton(dn.device.type):+ return None+ fn = _sym_shared_triton_fn()+ if fn is None:+ return None+ try:+ return fn(dn, qw, fracs, invf, nmax)+ except Exception as e:+ _sym_shared_broken = True+ torch.cuda.empty_cache() # a failed launch may hold pooled memory+ logger.warning("[NeUQI] shared sym coarse search failed (%s); using the per-candidate path", e)+ return None+++def _sym_triton_fn():+ """Resolve the extension symmetric-search Triton kernel once."""+ global _sym_triton, _sym_triton_checked+ if not _sym_triton_checked:+ _sym_triton_checked = True+ try:+ from auto_round_extension.triton.neuqi_sweep import sym_search_triton++ _sym_triton = sym_search_triton+ logger.debug("[NeUQI] Triton symmetric scale search engaged (auto_round_extension.triton.neuqi_sweep)")+ except Exception as e:+ _sym_triton = None+ logger.info("[NeUQI] Triton sym search unavailable (%s); using the compiled core", e)+ return _sym_triton+++def _sym_wants_triton(device_type: str) -> bool:+ """Whether the extension Triton sym search should be tried, per backend rules.++ AUTO resolves to the compiled core, not the Triton kernel: the sym chunk+ keeps the whole [C, g] data tile L2-resident (chunking bounds the [C, K, g]+ expansion), so the compiled core's wide-vector candidate re-reads from L2+ beat the registers-resident kernel's serialized candidate loop -- measured+ RTX 3090, 2M groups, g=128, 256+64 candidates: compiled 0.72 s vs Triton+ 1.35 s (eager 8.53 s). Parity is identical either way (same ~0.5% tie+ flips, max fp64 rel diff 1.7e-6 vs eager). The kernel stays available+ under AR_NEUQI_BACKEND=triton and is the tuning base if a shape regime+ where it wins shows up.+ """+ backend = str(envs.AR_NEUQI_BACKEND or "auto")+ return backend == "triton"+++def _sym_triton_attempt(data, qw, scales, nmax):+ """Try the Triton sym search once; ``None`` (and a permanent latch) on failure."""+ global _sym_triton_broken+ if _sym_triton_broken or not data.is_cuda or not _sym_wants_triton(data.device.type):+ return None+ fn = _sym_triton_fn()+ if fn is None:+ return None+ try:+ return fn(data, qw, scales, nmax)+ except Exception as e:+ _sym_triton_broken = True+ torch.cuda.empty_cache() # a failed launch may hold pooled memory+ logger.warning("[NeUQI] Triton sym search failed (%s); using the compiled core", e)+ return None+++def ensure_sym_search_warmup(device) -> None:+ """Serialized, synchronized first launch of the sym search kernel.++ Same rationale as :func:`ensure_sweep_warmup`: the first launch JIT-compiles+ the kernel and writes the on-disk cache; doing that concurrently with other+ GPU work has been observed to corrupt selections. Failures latch the+ Triton fallback instead of raising.+ """+ if device.type != "cuda":+ return+ global _sym_triton_broken, _sym_shared_broken+ with _sweep_warm_lock:+ if device.index in _sym_search_warmed:+ return+ _sym_search_warmed.add(device.index)+ engaged = []+ if _sym_wants_triton(device.type):+ fn = _sym_triton_fn()+ if fn is not None:+ try:+ data = torch.randn(4, 128, device=device, dtype=torch.float32)+ scales = torch.rand(4, 2, device=device, dtype=torch.float32) * 0.01 + 0.001+ loss, _idx, _mir = fn(data, torch.ones_like(data), scales, 7)+ torch.cuda.synchronize(device)+ if not torch.isfinite(loss).all():+ raise RuntimeError("warmup produced non-finite losses")+ logger.debug("[NeUQI] Triton sym search warmed on %s", device)+ engaged.append("symmetric scale sweep")+ except Exception as e: # noqa: BLE001 warmup failure must never kill a run+ _sym_triton_broken = True+ logger.warning(+ "[NeUQI] Triton sym search warmup failed on %s (%s); using the compiled core", device, e+ )+ if _sym_shared_wants_triton(device.type):+ fn = _sym_shared_triton_fn()+ if fn is not None:+ try:+ dn = torch.randn(4, 128, device=device, dtype=torch.float32)+ fracs = torch.rand(8, device=device, dtype=torch.float32) * 0.5 + 0.5+ loss, _idx, _mir = fn(dn, torch.ones_like(dn), fracs, 1.0 / fracs, 7)+ torch.cuda.synchronize(device)+ if not torch.isfinite(loss).all():+ raise RuntimeError("warmup produced non-finite losses")+ logger.debug("[NeUQI] shared sym coarse search warmed on %s", device)+ engaged.append("shared-multiplier coarse sweep")+ except Exception as e: # noqa: BLE001 warmup failure must never kill a run+ _sym_shared_broken = True+ logger.warning(+ "[NeUQI] shared sym coarse search warmup failed on %s (%s); using the per-candidate path",+ device,+ e,+ )++ if engaged:+ logger.info(+ "[NeUQI] Triton backend engaged for the symmetric search on %s (kernels: %s)",+ device,+ ", ".join(engaged),+ )+++def _sym_search_eval(data, qw, scales, nmax):+ """Best evaluator for one chunk: Triton (registers-resident) -> compiled core -> eager."""+ out = _sym_triton_attempt(data, qw, scales, nmax)+ if out is not None:+ return out+ return _sym_loss_chunk_eval(data, qw, scales, nmax)+++# Compiled core for the symmetric search chunk: each eager chunk is ~20 small+# kernel launches and the chunk size shrinks with the candidate count+# (_MAX_TMP_ELEMS // (2 * K * g)), so wide grids (the union variant at+# coarse=256 reaches K ~ 440) make big tensors pure launch overhead. One fused+# inductor kernel per chunk is the symmetric analogue of the asym sweep's+# compiled stage. The dedicated Triton kernels cover the stages where they+# measured faster: the shared-multiplier kernel serves the plain coarse pass+# under AUTO, the per-candidate kernel is AR_NEUQI_BACKEND=triton opt-in; this+# compiled core stays the AUTO choice for the fine stage (per-group grids).+_sym_chunk_compiled = None+_sym_compile_failed = False+_sym_compile_logged: set = set()++[email protected] # logging side effect: graph-break instead of tracing+def _log_sym_compile_engaged(backend: str) -> None:+ if backend in _sym_compile_logged:+ return+ _sym_compile_logged.add(backend)+ logger.info("[NeUQI] sym search chunk core compiled (backend=%s)", backend)+++def _sym_loss_chunk_eval(data, qw, scales, nmax):+ """``_sym_loss_chunk`` through a lazily compiled core, latching down to eager.++ Called only from inside ``torch.compiler.disable``-decorated search+ functions, so the explicit compile unit is the sole dynamo boundary and+ implicit tracing never reaches it. Any failure latches down permanently:+ retrying per chunk would repeat the failure cost every chunk.+ """+ global _sym_chunk_compiled, _sym_compile_failed+ backend = str(envs.AR_NEUQI_BACKEND or "auto")+ if _sym_compile_failed or backend not in ("auto", "compile", "triton"):+ return _sym_loss_chunk(data, qw, scales, nmax)+ if _sym_chunk_compiled is None:+ try:+ _sym_chunk_compiled = torch.compile(_sym_loss_chunk)+ except Exception: # pragma: no cover - compilation unavailable in this environment+ _sym_compile_failed = True+ return _sym_loss_chunk(data, qw, scales, nmax)+ _log_sym_compile_engaged(backend)+ try:+ return _sym_chunk_compiled(data, qw, scales, nmax)+ except Exception:+ _sym_chunk_compiled = None+ _sym_compile_failed = True+ logger.warning("[NeUQI] sym compiled chunk core failed; using eager for the rest of this process.")+ return _sym_loss_chunk(data, qw, scales, nmax)+++# Fraction of the no-clip max-abs scale above which coarser grids are never+# competitive: beyond the no-clip scale the step only coarsens with no clipping+# benefit, and the slack region far above it is dominated.+_SYM_UPPER_SCALE_RATIO = 2.0+++def _sym_coarse_pass_shared(data, qw, s0, coarse, nmax, launch=None):+ """Normalized-space coarse pass through the shared-multiplier kernel.++ Computes dn = w / s0 once, evaluates every group against the SHARED frac+ grid through ``launch`` (default: the gated Triton attempt), and+ unnormalizes each winner's loss by s0^2 so the fine stage can compare+ across stages. Returns (best_loss, best_frac, best_mirror) or ``None``+ when the kernel is unavailable anywhere -- the caller then serves the+ identical candidate grid through the per-candidate evaluator.++ ``launch`` is injectable so the driver math (normalization, s0^2+ unnormalization, frac mapping) stays unit-testable on CPU.+ """+ n_groups = data.shape[0]+ if launch is None:+ # pre-gate before any tensor math: CPU / non-shared backends must not+ # pay for the normalized copy+ if _sym_shared_broken or not data.is_cuda or not _sym_shared_wants_triton(data.device.type):+ return None+ launch = _sym_shared_triton_attempt+ dn = data / s0+ invf = 1.0 / coarse+ best_loss = torch.full((n_groups,), float("inf"), device=data.device, dtype=torch.float32)+ best_frac = torch.ones(n_groups, device=data.device, dtype=torch.float32)+ best_mirror = torch.zeros(n_groups, device=data.device, dtype=torch.bool)+ chunk = 262144+ for start_idx in range(0, n_groups, chunk):+ stop = min(start_idx + chunk, n_groups)+ out = launch(+ dn[start_idx:stop].contiguous(),+ qw[start_idx:stop].contiguous() if qw is not None else None,+ coarse,+ invf,+ nmax,+ )+ if out is None:+ return None+ loss, idx, mirror = out+ best_loss[start_idx:stop] = loss * s0[start_idx:stop].squeeze(-1) ** 2+ best_frac[start_idx:stop] = coarse.index_select(0, idx)+ best_mirror[start_idx:stop] = mirror+ return best_loss, best_frac, best_mirror++[email protected] # loop-heavy eager search: never trace into dynamo graphs+def _two_stage_sym_core(data, qw, bits, coarse, fine_n, q_scale_thresh):+ """Coarse+fine scale search core over a given (sorted, positive) coarse grid.++ Runs the batched coarse evaluation, then the additive fine refinement+ bracketed by the value-neighbors of each group's winning coarse entry,+ under both clamp conventions (see ``_sym_loss_chunk``). Strict-improvement+ updates only, so the result never worsens versus any evaluated candidate.++ Returns:+ (signed per-group scale [N, 1], per-group best weighted loss [N]).+ """+ nmax = int(2 ** (bits - 1))+ g = data.shape[1]+ n_groups = data.shape[0]+ s0 = (torch.abs(data).amax(dim=-1, keepdim=True) / nmax).clamp_(min=q_scale_thresh) # [N, 1]+ coarse_n = coarse.numel()++ best_loss = torch.full((n_groups,), float("inf"), device=data.device, dtype=torch.float32)+ best_frac = torch.ones(n_groups, device=data.device, dtype=torch.float32)+ best_mirror = torch.zeros(n_groups, device=data.device, dtype=torch.bool)++ # Coarse pass: shared fractions of the per-group no-clip scale. Under AUTO+ # on CUDA this runs in normalized space through the dedicated+ # shared-multiplier Triton kernel (dn = w / s0 once; every group multiplies+ # by the same 1/frac scalars -- no [C, K] scales, no division in the loop);+ # the winner loss is unnormalized by s0^2 for the cross-stage compare.+ # Anywhere else the per-candidate evaluator serves the identical grid.+ shared_out = _sym_coarse_pass_shared(data, qw, s0, coarse, nmax)+ if shared_out is not None:+ best_loss, best_frac, best_mirror = shared_out+ else:+ chunk = max(1, _MAX_TMP_ELEMS // (2 * coarse_n * g))+ for start_idx in range(0, n_groups, chunk):+ stop = min(start_idx + chunk, n_groups)+ scales = s0[start_idx:stop] * coarse.view(1, -1) # [C, K]+ loss, idx, mirror = _sym_search_eval(+ data[start_idx:stop], qw[start_idx:stop] if qw is not None else None, scales, nmax+ )+ best_loss[start_idx:stop] = loss+ best_frac[start_idx:stop] = coarse.index_select(0, idx)+ best_mirror[start_idx:stop] = mirror++ # Fine pass: additive grid between the coarse neighbors bracketing each winner.+ best_idx = torch.argmin(torch.abs(coarse.unsqueeze(0) - best_frac.unsqueeze(1)), dim=1)+ frac_lo = coarse[(best_idx - 1).clamp_(min=0)]+ frac_hi = coarse[(best_idx + 1).clamp_(max=coarse_n - 1)]+ if fine_n:+ steps = torch.arange(1, fine_n + 1, device=data.device, dtype=torch.float32) / (fine_n + 1)+ one_m_steps = 1.0 - steps+ chunk = max(1, _MAX_TMP_ELEMS // (2 * fine_n * g))+ for start_idx in range(0, n_groups, chunk):+ stop = min(start_idx + chunk, n_groups)+ fracs = frac_lo[start_idx:stop].unsqueeze(1) * one_m_steps + frac_hi[start_idx:stop].unsqueeze(1) * steps+ scales = s0[start_idx:stop] * fracs # [C, F]+ loss, k, mirror = _sym_search_eval(+ data[start_idx:stop], qw[start_idx:stop] if qw is not None else None, scales, nmax+ )+ frac = fracs.gather(1, k.unsqueeze(1)).squeeze(1)+ improved = loss < best_loss[start_idx:stop]+ best_loss[start_idx:stop] = torch.where(improved, loss, best_loss[start_idx:stop])+ best_frac[start_idx:stop] = torch.where(improved, frac, best_frac[start_idx:stop])+ best_mirror[start_idx:stop] = torch.where(improved, mirror, best_mirror[start_idx:stop])++ scale = best_frac.unsqueeze(-1) * s0+ scale = torch.where(best_mirror.unsqueeze(-1), -scale, scale)+ # magnitude floor on both signs (the incumbent keeps the sign and clamps+ # the magnitude), so a near-zero winner never produces a degenerate scale+ scale = torch.where(scale < 0, torch.clamp(scale, max=-q_scale_thresh), torch.clamp(scale, min=q_scale_thresh))+ return scale, best_loss++[email protected] # loop-heavy eager search: never trace into dynamo graphs+def neuqi_search_scale_sym(data, bits, qw=None, q_scale_thresh=1e-5, coarse_n=None, fine_n=None):+ """Two-stage (coarse log-spaced + fine additive) scale search, symmetric.++ Symmetric counterpart of :func:`neuqi_search_scale_zero`: the zero point is+ fixed at 0 and the integer range is clamped to+ ``[-2**(bits-1), 2**(bits-1) - 1]``. The grid is anchored at the per-group+ no-clip scale ``s0 = max|w| / nmax`` and searches log-spaced fractions in+ ``[_lower_scale_ratio(bits), _SYM_UPPER_SCALE_RATIO]`` followed by an+ additive refinement between the coarse neighbors bracketing each winner --+ the same two-stage scheme as the asymmetric search, applied to the+ zero-point-free slice. Both clamp conventions of the one-level-asymmetric+ symmetric grid are searched per group (negative scales, the family the+ incumbent reaches implicitly via its signed anchor; see+ ``_sym_loss_chunk``), and the returned scale carries the winning sign.++ Args:+ data: [N, g] float32 tensor of already-grouped weights.+ bits: quantization bit width.+ qw: optional [N, g] float32 per-element loss weights (e.g. imatrix).+ q_scale_thresh: minimum scale magnitude.+ coarse_n: number of coarse log-spaced candidates (env+ ``AR_NEUQI_COARSE``; default 256).+ fine_n: number of fine additive candidates (env+ ``AR_NEUQI_FINE``; default 64).++ Returns:+ scale: [N, 1] float32 per-group signed scales (negative = mirrored clamp+ convention won for that group).+ """+ if coarse_n is None or fine_n is None:+ d_coarse, d_fine = _default_grid(data.device.type)+ if coarse_n is None:+ # shared AR_NEUQI_COARSE/FINE pins apply (see envs.py); the unpinned+ # default is backend-aware (wide only on the accelerated lanes)+ coarse_n = envs.AR_NEUQI_COARSE or d_coarse+ if coarse_n < 2:+ raise ValueError("AR_NEUQI_COARSE=%d is too small; need at least 2 candidates." % coarse_n)+ if fine_n is None:+ fine_n = envs.AR_NEUQI_FINE or d_fine+ if fine_n < 2:+ raise ValueError("AR_NEUQI_FINE=%d is too small; need at least 2 candidates." % fine_n)+ _log_sym_search_engaged(coarse_n, fine_n)+ ensure_sym_search_warmup(data.device)++ data = data.to(torch.float32)+ if qw is not None:+ qw = _align_qw(qw, data).to(torch.float32)++ log_grid = _coarse_grid(_lower_scale_ratio(bits), coarse_n, data.device, torch.float32, hi=_SYM_UPPER_SCALE_RATIO)+ return _two_stage_sym_core(data, qw, bits, log_grid, fine_n, q_scale_thresh)[0]+++@register_dtype("opt_rtn_int_sym_neuqi")+def quant_tensor_opt_rtn_sym_neuqi(+ tensor, bits=4, group_size=-1, v=0, q_scale_thresh=1e-5, imatrix=None, scale_dtype=torch.float16, **kwargs+):+ """Quantize/dequantize with the two-stage symmetric scale search (NeUQI grid).++ Symmetric sibling of ``opt_rtn_int_asym``: the zero point is fixed at 0 and+ the joint search collapses to the scale axis, searched on the NeUQI+ coarse+fine grid instead of ``opt_rtn_int_sym``'s uniform grid. Returns+ follow the ``quant_tensor_opt_rtn_sym`` conventions (zero point = ``nmax``).+ ``v`` is accepted and ignored, matching the incumbent symmetric path.++ Args:+ tensor: Tensor to quantize.+ bits: Number of bits for the quantization (e.g., 2, 3, 4, 8).+ group_size: Number of elements sharing a scale.+ v: Rounding value perturbation (ignored, matching ``opt_rtn_int_sym``).+ q_scale_thresh: Minimum scale magnitude for numerical stability.+ imatrix: Optional per-column (1-D) or per-row (2-D, stacked modules)+ importance weights (activation imatrix).+ scale_dtype: dtype of the returned scale (kernels support fp16/fp32).++ Returns:+ (qdq_result, scale, zero_point) matching ``quant_tensor_opt_rtn_sym`` conventions.+ """+ from auto_round.data_type.gguf import _imatrix_handle_zero++ tensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)+ nmax = int(2 ** (bits - 1))++ qw = None+ if imatrix is not None:+ imatrix = _align_qw(imatrix, tensor)+ if imatrix.dim() == 1:+ qw = imatrix.reshape(1, -1)+ qw = reshape_pad_tensor_by_group_size(qw, group_size, val=1e-5)[0].view(1, -1)+ qw = qw.expand(tensor.numel() // qw.numel(), -1)+ qw = qw.reshape(tensor.shape)+ else:+ qw = reshape_pad_tensor_by_group_size(imatrix, group_size, val=1e-5)[0]+ if qw.shape != tensor.shape:+ raise ValueError(+ f"per-row imatrix shape {tuple(imatrix.shape)} incompatible with tensor {tuple(tensor.shape)}"+ )+ qw = _imatrix_handle_zero(qw, tensor, bits, group_size)++ scale = neuqi_search_scale_sym(tensor.to(torch.float32), bits, qw=qw, q_scale_thresh=q_scale_thresh)+ scale = scale.to(scale_dtype)+ # sign-preserving magnitude floor, mirroring quant_tensor_opt_rtn_sym+ scale = torch.where(scale < 0, torch.clamp(scale, max=-q_scale_thresh), torch.clamp(scale, min=q_scale_thresh))++ int_w = tensor.div(scale).round_().clamp_(-nmax, nmax - 1)+ qdq_result = (int_w.mul_(scale)).to(tensor.dtype)+ qdq_result = revert_tensor_by_pad(qdq_result, orig_shape=orig_shape, pad_len=pad_len)+ return qdq_result, scale, nmax+++@register_dtype("opt_rtn_int_asym")+def quant_tensor_opt_rtn_asym(+ tensor, bits=4, group_size=-1, v=0, q_scale_thresh=1e-5, imatrix=None, scale_dtype=torch.float16, **kwargs+):+ """Quantize/dequantize with a joint near-optimal (scale, integer zero-point) search.++ Asymmetric counterpart of ``opt_rtn_int_sym``: fills the previously missing+ optimized path for asymmetric int quantization (plain min/max before). Set+ the per-group scale and integer zero point by a two-stage coarse+fine grid+ search instead of the plain min/max initialization (arXiv 2505.17595).++ Args:+ tensor: Tensor to quantize.+ bits: Number of bits for quantization (e.g., 2, 3, 4, 8).+ group_size: Number of elements sharing a scale.+ v: Rounding value perturbation.+ q_scale_thresh: Minimum scale magnitude for numerical stability.+ imatrix: Optional per-column importance weights (activation imatrix).+ scale_dtype: dtype of the returned scale (kernels support fp16/fp32).++ Returns:+ (qdq_result, scale, zero_point) matching ``quant_tensor_asym`` conventions.+ """+ from auto_round.data_type.gguf import _imatrix_handle_zero++ tensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)+ maxq = int(2**bits) - 1++ qw = None+ if imatrix is not None:+ imatrix = _align_qw(imatrix, tensor)+ if imatrix.dim() == 1:+ # per-column importance shared by every row of this tensor+ qw = imatrix.reshape(1, -1)+ qw = reshape_pad_tensor_by_group_size(qw, group_size, val=1e-5)[0].view(1, -1)+ qw = qw.expand(tensor.numel() // qw.numel(), -1)+ qw = qw.reshape(tensor.shape)+ else:+ # per-row importance (stacked same-shape modules: each module keeps+ # its own column weights); must already match the tensor's shape+ qw = reshape_pad_tensor_by_group_size(imatrix, group_size, val=1e-5)[0]+ if qw.shape != tensor.shape:+ raise ValueError(+ f"per-row imatrix shape {tuple(imatrix.shape)} incompatible with tensor {tuple(tensor.shape)}"+ )+ qw = _imatrix_handle_zero(qw, tensor, bits, group_size)++ scale, zp = neuqi_search_scale_zero(tensor.to(torch.float32), bits, qw=qw, q_scale_thresh=q_scale_thresh)+ scale = torch.clamp(scale.to(scale_dtype), min=q_scale_thresh)+ zp = zp.to(scale_dtype)++ int_w = round_ste(tensor / scale + v)+ q = torch.clamp(int_w + zp, 0, maxq)+ qdq_result = (scale * (q - zp)).to(tensor.dtype)+ qdq_result = revert_tensor_by_pad(qdq_result, orig_shape=orig_shape, pad_len=pad_len)+ return qdq_result, scale, zp+++logger.debug("[NeUQI] opt_rtn_int_asym registered (clean-room, arXiv 2505.17595)")+logger.debug("[NeUQI] opt_rtn_int_sym_neuqi registered (two-stage sym scale search)")+++def neuqi_sync_devices(device):+ """Join in-flight GPU work before a search launches.++ The data-driven pipeline's input-caching pass moves activations across+ GPUs on side streams (accelerate). A not-yet-joined producer makes the+ search read stale tensors -- an intermittent device-side assert that+ disappears under CUDA_LAUNCH_BLOCKING=1. One sync per layer-init is noise+ next to the search cost.++ Syncs EVERY visible device, not just this layer's: under accelerate+ dispatch the cache-inputs forward moves activations GPU->GPU, and an+ un-joined peer copy recorded on ANOTHER device's stream can still be+ reading memory the search is about to reuse (allocator reuse -> the+ intermittent OOB-gather asserts that vanish under blocking mode).+ """+ if torch.device(device).type == "cuda":+ for dev_idx in range(torch.cuda.device_count()):+ torch.cuda.synchronize(dev_idx)+++def neuqi_anchor_margins(weight, bits, group_size, imatrix, device, sym=False):+ """Searched (scale, zp) grid for the ``enable_neuqi`` init, expressed as the+ per-group (tensor_min, tensor_max) pair the STE quant functions re-derive+ the grid from.++ Asym anchoring (quant_tensor_asym: scale=(wmax-wmin)/maxq,+ zp=round(-wmin/scale)): min=-zp*s, max=(maxq-zp)*s reproduces (s, zp)+ exactly. Sym anchoring (quant_tensor_sym: scale=max(|wmax|,|wmin|)/maxq,+ signed): min=-s*maxq, max=+s*maxq.++ Both symmetry classes anchor: asym layers through the joint+ (scale, zero-point) search, sym layers through the two-stage symmetric+ scale search. The sym search returns a SIGNED scale whose sign cancels in+ qdq reconstruction (the mirrored convention is the arithmetic of a+ negative scale through the standard formula), so anchoring the magnitude+ reproduces the winning qdq grid exactly.++ On the tuning path the grid is FROZEN: SignRound tunes the rounding values+ against it, and the range margins stay pinned at 1.0 (the search's grid IS+ the init). On the zero-shot asymmetric path the same margins make the+ plain asym quant reproduce the searched (scale, zp) exactly.++ Returns (wmin, wmax), each squeezed over the trailing group axis.+ """+ import time++ t0 = time.perf_counter()+ w = weight.to(device=device, dtype=torch.float32)+ qw = None+ if imatrix is not None:+ qw = _prepare_neuqi_qw(imatrix, w, bits, group_size).to(device)+ neuqi_sync_devices(device)++ if sym:+ # two-stage symmetric scale search; the returned scale is SIGNED+ # (negative = mirrored clamp convention won). quant_tensor_sym derives+ # scale = (2*(wmax_abs < wmin_abs) - 1) * max(wmax_abs, wmin_abs)/maxq,+ # so an exact magnitude tie (and wmax_abs >= wmin_abs generally)+ # resolves to the NEGATIVE scale. Bias the losing side down to HALF+ # magnitude so the comparison resolves to the search winner's sign in+ # ANY storage dtype (a 2^-20 bias would be erased by bf16/fp16+ # rounding when the anchor is cast to the weight dtype), while max()+ # still returns the unperturbed magnitude -- the derived scale is the+ # searched |s| exactly (the halved side only ever participates in the+ # comparison; margins are pinned at 1.0 so nothing else reads it).+ s_signed = neuqi_search_scale_sym(w, bits, qw=qw)+ nmax = int(2 ** (bits - 1))+ mag = s_signed.abs() * nmax+ mirrored = s_signed < 0+ # standard winner (s > 0) needs wmax_abs strictly smaller: scale positive+ wmin = torch.where(mirrored, -mag / 2, -mag)+ wmax = torch.where(mirrored, mag, mag / 2)+ else:+ s, zp = neuqi_search_scale_zero(w, bits, qw=qw)+ maxq = int(2**bits) - 1+ wmin = -(zp * s)+ wmax = (maxq - zp) * s+ global _ANCHOR_LOGGED+ if not _ANCHOR_LOGGED:+ _ANCHOR_LOGGED = True+ logger.info(+ "[NeUQI] init-search anchored the quantization grid (%.2fs first layer; margins pinned at 1.0)",+ time.perf_counter() - t0,+ )+ return wmin.squeeze(-1), wmax.squeeze(-1)+++def _prepare_neuqi_qw(imatrix, weight_reshape, bits, group_size):+ """Normalize an imatrix to the padded ``[N, gs]`` grouped layout.++ Mirrors the zero-shot search path: the calibration hooks attach a 1D+ ``[in_features]`` per-column importance vector -- it must be padded by+ group size, expanded to the weight's grouped rows, and zero-guarded+ before any per-group search consumes it. ``None`` never reaches here+ (uniform weights).+ """+ from auto_round.data_type.gguf import _imatrix_handle_zero++ qw = imatrix.reshape(1, -1).to(torch.float32)+ qw = reshape_pad_tensor_by_group_size(qw, group_size, val=1e-5)[0].view(1, -1)+ qw = qw.expand(weight_reshape.numel() // qw.numel(), -1).reshape(weight_reshape.shape)+ return _imatrix_handle_zero(qw, weight_reshape, bits, group_size)+++_ANCHOR_LOGGED = Falsediff --git a/auto_round/data_type/utils.py b/auto_round/data_type/utils.pyindex 7bca6fe97..676b2822e 100644--- a/auto_round/data_type/utils.py+++ b/auto_round/data_type/utils.py@@ -103,7 +103,7 @@ def revert_tensor_by_pad(data: torch.Tensor, orig_shape: tuple, pad_len: Union[idef get_quant_func(- dtype: str, bits: int, sym: bool, disable_opt_rtn=False, group_size=None, iters=200+ dtype: str, bits: int, sym: bool, disable_opt_rtn=False, group_size=None, iters=200, enable_neuqi=False) -> tuple[callable, str]:"""Retrieve the quantization function based on data type, bit width, and symmetry.@@ -117,6 +117,12 @@ def get_quant_func(sym (bool): A flag indicating whether the quantization is symmetric (True) or asymmetric (False).disable_opt_rtn(bool): whether to disable optimized rtn.group_size (tuple): The block size for weight quantization (e.g., (128, 128)).+ enable_neuqi (bool): Opt into the NeUQI grid search (arXiv 2505.17595).+ For ``sym=False`` the joint (scale, zero-point) search replaces the+ plain min/max initialization on the optimized path; for ``sym=True``+ the two-stage symmetric scale search+ (``opt_rtn_int_sym_neuqi``) replaces the uniform ``search_scales``+ grid. ``False`` (default) keeps the incumbent behavior byte-identical.Returns:function: The quantization function corresponding to the specified parameters.@@ -136,6 +142,15 @@ def get_quant_func(if not disable_opt_rtn and iters == 0:rtn_data_type = "opt_rtn_" + dtypedata_types = [rtn_data_type, pad_bits(rtn_data_type), pad_sym(rtn_data_type), pad_sym(pad_bits(rtn_data_type))]+ if not enable_neuqi and not sym:+ # the joint NeUQI search is opt-in: without it the asymmetric+ # optimized path has no search (the integer zero point breaks the+ # scale-only grid) and resolves to the plain min/max function+ data_types = [dt for dt in data_types if not dt.endswith("_asym") and not dt.endswith(f"_asym{bits}")]+ if sym and enable_neuqi:+ # two-stage symmetric scale search (NeUQI grid machinery, zero point+ # fixed at 0) -- preferred over the uniform search_scales grid+ data_types = [f"{rtn_data_type}_sym_neuqi"] + data_typesfor data_type in data_types:from auto_round.data_type import QUANT_FUNC_WITH_DTYPEdiff --git a/auto_round/envs.py b/auto_round/envs.pyindex 0a64af5a8..e7a8b706c 100644--- a/auto_round/envs.py+++ b/auto_round/envs.py@@ -210,6 +210,16 @@ environment_variables: dict[str, Callable[[], Any]] = {# Exposed on the CLI as ``--num_hidden_layers``. The resulting model is a# partial model and must not be used for a real/production quantization run."AR_DEBUG_LAYER_NUM": lambda: _get_optional_positive_int_env("AR_DEBUG_LAYER_NUM"),+ # NeUQI joint (scale, zero-point) search knobs: coarse/fine candidate+ # counts (shared by both symmetry classes), backend selection for the+ # zero-point sweep (auto|eager|compile|triton; latches down permanently+ # on failure), and the sweep/knobs are otherwise internal (layout and+ # candidate batching are device-automatic).+ # unset -> None so the search entries can apply their backend-aware+ # defaults (wide 256/64 on Triton/compile lanes, narrow 64/32 on eager)+ "AR_NEUQI_COARSE": lambda: int(v) if (v := os.getenv("AR_NEUQI_COARSE")) is not None else None,+ "AR_NEUQI_FINE": lambda: int(v) if (v := os.getenv("AR_NEUQI_FINE")) is not None else None,+ "AR_NEUQI_BACKEND": lambda: os.getenv("AR_NEUQI_BACKEND", "auto").lower(),}diff --git a/auto_round/wrapper.py b/auto_round/wrapper.pyindex 36ffbb173..66897972e 100644--- a/auto_round/wrapper.py+++ b/auto_round/wrapper.py@@ -84,6 +84,7 @@ class WrapperLinear(torch.nn.Module):enable_round_tuning=True,enable_torch_compile=True,disable_opt_rtn=True,+ enable_neuqi=False,**kwargs,):"""Initializes the WrapperLinear module.@@ -98,6 +99,7 @@ class WrapperLinear(torch.nn.Module):self.orig_layer = orig_layerself.orig_layer.iters = kwargs.pop("iters", 200)self.disable_opt_rtn = disable_opt_rtn+ self.enable_neuqi = enable_neuqiself.output_device = deviceself.device = self.orig_layer.tuning_device if hasattr(self.orig_layer, "tuning_device") else deviceself.enable_minmax_tuning = enable_minmax_tuning@@ -181,13 +183,50 @@ class WrapperLinear(torch.nn.Module):if clip_max_flat.numel() == self.weight_max.numel() and clip_min_flat.numel() == self.weight_min.numel():self.weight_max = torch.minimum(self.weight_max, clip_max_flat)self.weight_min = torch.maximum(self.weight_min, clip_min_flat)+ # NeUQI frozen init (enable_neuqi on the tuning path): anchor+ # the grid to the searched (scale, zp) instead of the raw per-group+ # min/max. Placement mirrors the AWQ clip-as-init above; unsupported+ # layouts (tuple group sizes, >=16 bits, non-int data types) keep the+ # status-quo min/max grid. The zero-shot path needs no anchor -- its+ # search runs inside the quant function dispatch itself.+ self._neuqi_frozen_margins = False+ if (+ self.enable_neuqi+ and self.enable_round_tuning+ and weight_reshape is not None+ and self.weight_min is not None+ and self.orig_layer.bits < 16+ and not isinstance(orig_layer.group_size, tuple)+ and self.orig_layer.data_type == "int"+ ):+ if getattr(orig_layer, "awq_clip_max", None) is not None:+ logger.warning_once(+ "enable_neuqi anchors the tuning grid to its own search result, which "+ "overrides the AWQ clip-as-init range on this layer; the clip is ignored."+ )+ from auto_round.data_type.neuqi import neuqi_anchor_margins++ _wmin, _wmax = neuqi_anchor_margins(+ weight_reshape,+ self.orig_layer.bits,+ orig_layer.group_size,+ getattr(orig_layer, "imatrix", None),+ self.device,+ sym=bool(getattr(orig_layer, "sym", True)),+ )+ self._neuqi_frozen_margins = True+ self.weight_min = _wmin.to(self.weight_min.dtype)+ self.weight_max = _wmax.to(self.weight_max.dtype)self._init_params("value", p_dtype, weight_reshape.shape, 0, self.enable_round_tuning and self.orig_layer.bits < 16)# Min-max scale initializationshape = get_scale_shape(orig_weight, orig_layer.group_size)- self._init_params("min_scale", p_dtype, shape, 1.0, (self.enable_minmax_tuning and self.orig_layer.bits < 16))- self._init_params("max_scale", p_dtype, shape, 1.0, (self.enable_minmax_tuning and self.orig_layer.bits < 16))+ _margins_tunable = (self.enable_minmax_tuning and self.orig_layer.bits < 16) and (+ not self._neuqi_frozen_margins+ )+ self._init_params("min_scale", p_dtype, shape, 1.0, _margins_tunable)+ self._init_params("max_scale", p_dtype, shape, 1.0, _margins_tunable)self.weight_quant_func, self.data_type = get_quant_func(orig_layer.data_type,@@ -196,6 +235,7 @@ class WrapperLinear(torch.nn.Module):self.disable_opt_rtn,orig_layer.group_size,iters=orig_layer.iters,+ enable_neuqi=self.enable_neuqi,)if self.enable_torch_compile:self.weight_quant_func = compile_func(self.weight_quant_func, self.device)diff --git a/auto_round_extension/triton/neuqi_sweep.py b/auto_round_extension/triton/neuqi_sweep.pynew file mode 100644index 000000000..251d5557b--- /dev/null+++ b/auto_round_extension/triton/neuqi_sweep.py@@ -0,0 +1,568 @@+# Copyright (c) 2026 Intel Corporation+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+"""Triton kernel for the NeUQI all-candidates zero-point sweep.++One launch evaluates every per-group scale candidate (``scales``, [C, K])+against every integer zero point for a chunk of groups. Unlike the+``torch.compile``-fused expression -- whose kernel parallelizes over the+(candidate, zero-point) lanes and therefore re-reads each weight element from+L2 once per lane -- this kernel loads each element of a [BC, g] data tile into+registers exactly once and computes all K x Z losses from registers+(amplification 1x). Measured on an RTX 3090 (2M groups, g=64, bits=4, 64+32+candidate grid, fp32): 0.216 s vs 0.488 s (compiled batched expression) and+12.3 s (eager chunked sweep). Selections match the torch paths' tie profile+(same ~0.07% flipped groups, worst fp64 relative loss difference ~5e-5): the+kernel rounds with round-half-to-even (matching ``torch.round``; libdevice+``round`` is half-away-from-zero and would change selections) and keeps the+first-minimum zero-point rule via a strict ``<`` with ascending zero points.++Contract (``neuqi_sweep_triton``):+ data: [C, g] float32, contiguous, CUDA+ qw: [C, g] float32, contiguous, CUDA, or ``None``+ scales: [C, K] float32, contiguous, CUDA+ maxq: maximum integer value (``2**bits - 1``)+ -> (loss [C, K] float32, argmin zero point [C, K] int32)++Contract (``sym_search_triton``): the symmetric scale search with the same+registers-resident structure, one data load per element and all K candidates+under BOTH clamp conventions computed from registers. Rounding and tie rules+match ``_sym_loss_chunk`` (round-half-to-even; the standard convention wins a+loss tie against the mirrored one; ascending-candidate first-minimum).++ data: [C, g] float32, contiguous, CUDA+ qw: [C, g] float32, contiguous, CUDA, or ``None``+ scales: [C, K] float32 POSITIVE per-group candidates, contiguous, CUDA+ nmax: symmetric integer bound (``2**(bits - 1)``)+ -> (best_loss [C] float32, best candidate index [C] int64,+ best_mirror [C] bool: the mirrored convention won that group)++Contract (``sym_search_shared_triton``): the coarse stage of the symmetric+search in NORMALIZED space. The driver precomputes dn = w / s0 once (s0 =+per-group no-clip scale); every group then shares the same candidate fracs+f_k, so the kernel multiplies dn by the SHARED 1/f_k scalar instead of a+per-(group, candidate) scale -- no [C, K] scales tensor at all, no division+inside the candidate loop. The loss omits the s0^2 factor (argmin within a+group is invariant); the driver unnormalizes the winner for cross-stage+comparisons. Measured RTX 3090, 2M groups, g=128, 256+64 candidates: 0.24 s+vs 0.57-0.79 s for the compiled core (parity: same ~0.5% tie-flip profile,+max fp64 rel diff 1.7e-6 vs eager).++ dn: [C, g] float32, contiguous, CUDA (w / s0)+ qw: [C, g] float32, contiguous, CUDA, or ``None``+ fracs: [K] float32 POSITIVE shared multipliers, contiguous, CUDA+ invf: [K] float32, 1 / fracs, contiguous, CUDA, same device+ nmax: symmetric integer bound (``2**(bits - 1)``)+ -> (best_loss [C] float32 NORMALIZED, best candidate index [C] int64,+ best_mirror [C] bool: the mirrored convention won that group)+"""++import torch++try:+ import triton+ import triton.language as tl++ _TRITON_IMPORT_ERROR = None+except Exception as e: # pragma: no cover - depends on host toolchain+ triton = None+ tl = None+ _TRITON_IMPORT_ERROR = e+++if triton is not None:++ @triton.jit+ def _rint_f32(x):+ """Native round-half-to-even: cvt.rni.f32.f32 (exact torch.round)."""+ return tl.inline_asm_elementwise("cvt.rni.f32.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1)++ @triton.jit+ def _neuqi_sweep_kernel(+ data_ptr,+ qw_ptr,+ scales_ptr,+ loss_ptr,+ zp_ptr,+ C,+ G,+ K,+ NZ,+ MAXQ: tl.constexpr,+ HAS_QW: tl.constexpr,+ BC: tl.constexpr,+ GP: tl.constexpr,+ NOMASK: tl.constexpr,+ ZU: tl.constexpr,+ ):+ """Registers-resident (candidate, zero-point) sweep.++ Rounding is native cvt.rni.f32.f32 (round-half-to-even, exactly+ torch.round; measured selection-identical to the previous arithmetic+ emulation). NOMASK skips the tail-group mask selects when GP == G+ (pow2 group size). ZU statically unrolls the zero-point loop when+ MAXQ + 1 <= 32 (bits <= 5) for ILP -- wider bit widths keep the+ runtime loop to bound compile time and register pressure.+ """+ pid = tl.program_id(0)+ c_off = pid * BC + tl.arange(0, BC)+ g_off = tl.arange(0, GP)+ cm = c_off < C+ gm = g_off < G+ d = tl.load(data_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)+ if HAS_QW:+ qwt = tl.load(qw_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)+ else:+ qwt = tl.zeros((BC, GP), dtype=tl.float32) + 1.0+ for k in range(0, K):+ sc = tl.load(scales_ptr + c_off * K + k, mask=cm, other=1.0) # [BC]+ x = d / sc[:, None]+ r = _rint_f32(x)+ r = tl.minimum(tl.maximum(r, -MAXQ - 1.0), 2.0 * MAXQ + 1.0)+ best = tl.zeros((BC,), dtype=tl.float32) + float("inf")+ bz = tl.zeros((BC,), dtype=tl.int32)+ if ZU:+ for z in tl.static_range(0, MAXQ + 1):+ q = tl.minimum(tl.maximum(r + z, 0.0), MAXQ)+ err = sc[:, None] * (q - z) - d+ loss2 = err * err * qwt+ if NOMASK:+ acc = tl.sum(loss2, axis=1)+ else:+ acc = tl.sum(tl.where(gm[None, :], loss2, 0.0), axis=1)+ better = acc < best # strict: ascending z keeps the first minimum+ best = tl.where(better, acc, best)+ bz = tl.where(better, z, bz)+ else:+ for z in range(0, NZ):+ q = tl.minimum(tl.maximum(r + z, 0.0), MAXQ)+ err = sc[:, None] * (q - z) - d+ loss2 = err * err * qwt+ if NOMASK:+ acc = tl.sum(loss2, axis=1)+ else:+ acc = tl.sum(tl.where(gm[None, :], loss2, 0.0), axis=1)+ better = acc < best+ best = tl.where(better, acc, best)+ bz = tl.where(better, z, bz)+ tl.store(loss_ptr + c_off * K + k, best, mask=cm)+ tl.store(zp_ptr + c_off * K + k, bz, mask=cm)+++if triton is not None:++ @triton.jit+ def _sym_search_kernel(+ data_ptr,+ qw_ptr,+ scales_ptr,+ loss_ptr,+ idx_ptr,+ mirror_ptr,+ C,+ G,+ K,+ NMAX: tl.constexpr,+ HAS_QW: tl.constexpr,+ BC: tl.constexpr,+ GP: tl.constexpr,+ ):+ pid = tl.program_id(0)+ c_off = pid * BC + tl.arange(0, BC)+ g_off = tl.arange(0, GP)+ cm = c_off < C+ gm = g_off < G+ d = tl.load(data_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)+ if HAS_QW:+ qwt = tl.load(qw_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)+ else:+ qwt = tl.zeros((BC, GP), dtype=tl.float32) + 1.0+ best = tl.zeros((BC,), dtype=tl.float32) + float("inf")+ bk = tl.zeros((BC,), dtype=tl.int32)+ bmir = tl.zeros((BC,), dtype=tl.int32)+ for k in range(0, K):+ sc = tl.load(scales_ptr + c_off * K + k, mask=cm, other=1.0) # [BC]+ x = d / sc[:, None]+ fl = tl.floor(x)+ fr = x - fl+ # round-half-to-even (matches torch.round)+ fl_odd = fl - 2.0 * tl.floor(fl * 0.5) == 1.0+ take_up = (fr > 0.5) | ((fr == 0.5) & fl_odd)+ r = fl + take_up.to(tl.float32)+ # standard convention: clamp(-nmax, nmax - 1), dequant s * q+ q_std = tl.minimum(tl.maximum(r, -NMAX), NMAX - 1.0)+ e_std = sc[:, None] * q_std - d+ l_std = tl.sum(tl.where(gm[None, :], e_std * e_std * qwt, 0.0), axis=1)+ # mirrored convention (negative-scale family): clamp(-(nmax - 1), nmax)+ q_mir = tl.minimum(tl.maximum(r, -(NMAX - 1.0)), NMAX)+ e_mir = sc[:, None] * q_mir - d+ l_mir = tl.sum(tl.where(gm[None, :], e_mir * e_mir * qwt, 0.0), axis=1)+ mir = l_mir < l_std # strict: the standard convention wins a tie+ loss = tl.where(mir, l_mir, l_std)+ better = loss < best # strict: ascending k keeps the first minimum+ best = tl.where(better, loss, best)+ bk = tl.where(better, k, bk)+ bmir = tl.where(better, mir.to(tl.int32), bmir)+ tl.store(loss_ptr + c_off, best, mask=cm)+ tl.store(idx_ptr + c_off, bk, mask=cm)+ tl.store(mirror_ptr + c_off, bmir, mask=cm)+++if triton is not None:++ @triton.jit+ def _neuqi_shared_kernel(+ dn_ptr,+ qw_ptr,+ fracs_ptr,+ invf_ptr,+ loss_ptr,+ kp_ptr,+ zp_ptr,+ C,+ G,+ K,+ MAXQ: tl.constexpr,+ HAS_QW: tl.constexpr,+ BC: tl.constexpr,+ GP: tl.constexpr,+ NOMASK: tl.constexpr,+ ZU: tl.constexpr,+ ):+ """Coarse stage of the joint (scale, zero-point) search in NORMALIZED space.++ The driver precomputes dn = w / s0 once (s0 = per-group min-max+ scale); every group then shares the same candidate fracs f_k, so the+ kernel multiplies dn by the SHARED 1/f_k scalar -- no [C, K] scales+ tensor, no division inside the candidate loop. The loss omits the+ s0^2 factor (the argmin over (k, z) within a group is invariant); the+ driver unnormalizes the winner. In-kernel argmin over k keeps the+ ascending-k first-minimum rule of torch's min(dim=1). Rounding is+ native cvt.rni.f32.f32; tie rules match the reference sweep (ascending+ z keeps the first minimum, strict <).+ """+ pid = tl.program_id(0)+ c_off = pid * BC + tl.arange(0, BC)+ g_off = tl.arange(0, GP)+ cm = c_off < C+ gm = g_off < G+ d = tl.load(dn_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)+ if HAS_QW:+ qwt = tl.load(qw_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)+ else:+ qwt = tl.zeros((BC, GP), dtype=tl.float32) + 1.0+ best = tl.zeros((BC,), dtype=tl.float32) + float("inf")+ bk = tl.zeros((BC,), dtype=tl.int32)+ bz = tl.zeros((BC,), dtype=tl.int32)+ for k in range(0, K):+ f = tl.load(fracs_ptr + k)+ invf = tl.load(invf_ptr + k)+ x = d * invf+ r = _rint_f32(x)+ r = tl.minimum(tl.maximum(r, -MAXQ - 1.0), 2.0 * MAXQ + 1.0)+ l_best = tl.zeros((BC,), dtype=tl.float32) + float("inf")+ l_bz = tl.zeros((BC,), dtype=tl.int32)+ if ZU:+ for z in tl.static_range(0, MAXQ + 1):+ q = tl.minimum(tl.maximum(r + z, 0.0), MAXQ)+ err = f * (q - z) - d+ loss2 = err * err * qwt+ if NOMASK:+ acc = tl.sum(loss2, axis=1)+ else:+ acc = tl.sum(tl.where(gm[None, :], loss2, 0.0), axis=1)+ better = acc < l_best+ l_best = tl.where(better, acc, l_best)+ l_bz = tl.where(better, z, l_bz)+ else:+ for z in range(0, MAXQ + 1):+ q = tl.minimum(tl.maximum(r + z, 0.0), MAXQ)+ err = f * (q - z) - d+ loss2 = err * err * qwt+ if NOMASK:+ acc = tl.sum(loss2, axis=1)+ else:+ acc = tl.sum(tl.where(gm[None, :], loss2, 0.0), axis=1)+ better = acc < l_best+ l_best = tl.where(better, acc, l_best)+ l_bz = tl.where(better, z, l_bz)+ better_k = l_best < best+ best = tl.where(better_k, l_best, best)+ bk = tl.where(better_k, k, bk)+ bz = tl.where(better_k, l_bz, bz)+ tl.store(loss_ptr + c_off, best, mask=cm)+ tl.store(kp_ptr + c_off, bk, mask=cm)+ tl.store(zp_ptr + c_off, bz, mask=cm)+++def neuqi_shared_sweep_triton(dn, qw, fracs, invf, maxq):+ """Coarse-stage joint (scale, zero-point) search in normalized space on CUDA.++ Returns (best NORMALIZED loss [C] float32, best candidate index [C] int64,+ best zero point [C] int32); the driver unnormalizes the loss by s0^2 and+ maps the index through the shared frac grid.+ """+ assert triton is not None, f"triton is not importable: {_TRITON_IMPORT_ERROR}"+ assert dn.is_cuda and dn.dtype == torch.float32 and dn.is_contiguous()+ assert fracs.is_contiguous() and fracs.dtype == torch.float32 and fracs.ndim == 1+ assert invf.is_contiguous() and invf.dtype == torch.float32 and invf.shape == fracs.shape+ if qw is not None:+ assert qw.dtype == torch.float32 and qw.is_contiguous() and qw.shape == dn.shape+ _dev = dn.device+ assert fracs.device == _dev, f"neuqi_shared: fracs on {fracs.device} but dn on {_dev}"+ assert invf.device == _dev, f"neuqi_shared: invf on {invf.device} but dn on {_dev}"+ if qw is not None:+ assert qw.device == _dev, f"neuqi_shared: qw on {qw.device} but dn on {_dev}"++ c, g = dn.shape+ k = fracs.numel()+ gp = _next_pow2(g)+ bc = max(1, min(64, 4096 // gp))+ loss = torch.empty((c,), device=_dev, dtype=torch.float32)+ kp = torch.empty((c,), device=_dev, dtype=torch.int32)+ zp = torch.empty((c,), device=_dev, dtype=torch.int32)+ with torch.cuda.device(_dev): # Triton launches on the CURRENT device+ _neuqi_shared_kernel[(triton.cdiv(c, bc),)](+ dn,+ qw if qw is not None else dn, # dummy pointer when unweighted (dead lanes)+ fracs,+ invf,+ loss,+ kp,+ zp,+ c,+ g,+ k,+ MAXQ=maxq,+ HAS_QW=qw is not None,+ BC=bc,+ GP=gp,+ NOMASK=gp == g,+ ZU=maxq + 1 <= 32,+ num_warps=4,+ )+ return loss, kp.long(), zp+++def _next_pow2(n):+ p = 1+ while p < n:+ p *= 2+ return p+++def neuqi_sweep_triton(data, qw, scales, maxq):+ """Run the all-candidates zero-point sweep on CUDA (see module docstring)."""+ assert triton is not None, f"triton is not importable: {_TRITON_IMPORT_ERROR}"+ assert data.is_cuda and data.dtype == torch.float32 and data.is_contiguous()+ assert scales.is_contiguous() and scales.dtype == torch.float32+ if qw is not None:+ assert qw.dtype == torch.float32 and qw.is_contiguous() and qw.shape == data.shape+ # Triton dereferences raw pointers with no device consistency check -- a+ # cross-device input surfaces as a delayed "device-side assert" far from+ # the launch. Fail here, naming the offending tensor, instead.+ _dev = data.device+ assert scales.device == _dev, f"neuqi_sweep: scales on {scales.device} but data on {_dev}"+ if qw is not None:+ assert qw.device == _dev, f"neuqi_sweep: qw on {qw.device} but data on {_dev}"++ c, g = data.shape+ k = scales.shape[1]+ nz = maxq + 1+ gp = _next_pow2(g)+ bc = max(1, min(64, 4096 // gp))+ loss = torch.empty((c, k), device=data.device, dtype=torch.float32)+ zp = torch.empty((c, k), device=data.device, dtype=torch.int32)+ # Triton launches on the CURRENT cuda device; with tensors on another GPU+ # (block homes under streaming rotation) the launcher rejects the pointer+ # ("cannot be accessed from Triton (cpu tensor?)") or misbehaves. Pin the+ # context to the data's device for the launch.+ with torch.cuda.device(_dev):+ return _launch_sweep(data, qw, scales, loss, zp, c, g, k, nz, maxq, bc, gp)+++def sym_search_triton(data, qw, scales, nmax):+ """Evaluate all per-group scale candidates under both symmetric clamp+ conventions on CUDA (see module docstring)."""+ assert triton is not None, f"triton is not importable: {_TRITON_IMPORT_ERROR}"+ assert data.is_cuda and data.dtype == torch.float32 and data.is_contiguous()+ assert scales.is_contiguous() and scales.dtype == torch.float32+ if qw is not None:+ assert qw.dtype == torch.float32 and qw.is_contiguous() and qw.shape == data.shape+ _dev = data.device+ assert scales.device == _dev, f"sym_search: scales on {scales.device} but data on {_dev}"+ if qw is not None:+ assert qw.device == _dev, f"sym_search: qw on {qw.device} but data on {_dev}"++ c, g = data.shape+ k = scales.shape[1]+ gp = _next_pow2(g)+ bc = max(1, min(64, 4096 // gp))+ loss = torch.empty((c,), device=_dev, dtype=torch.float32)+ idx = torch.empty((c,), device=_dev, dtype=torch.int32)+ mirror = torch.empty((c,), device=_dev, dtype=torch.int32)+ with torch.cuda.device(_dev): # Triton launches on the CURRENT device+ _sym_search_kernel[(triton.cdiv(c, bc),)](+ data,+ qw if qw is not None else data, # dummy pointer when unweighted+ scales,+ loss,+ idx,+ mirror,+ c,+ g,+ k,+ NMAX=nmax,+ HAS_QW=qw is not None,+ BC=bc,+ GP=gp,+ num_warps=4,+ )+ return loss, idx.long(), mirror.bool()+++if triton is not None:++ @triton.jit+ def _sym_shared_kernel(+ dn_ptr,+ qw_ptr,+ fracs_ptr,+ invf_ptr,+ loss_ptr,+ idx_ptr,+ mirror_ptr,+ C,+ G,+ K,+ NMAX: tl.constexpr,+ HAS_QW: tl.constexpr,+ BC: tl.constexpr,+ GP: tl.constexpr,+ NOMASK: tl.constexpr,+ ):+ """Coarse-stage symmetric search in normalized space (see module docstring).++ Rounding is native cvt.rni.f32.f32 (round-half-to-even, matching+ torch.round). Tie rules match _sym_loss_chunk: the standard convention+ wins a loss tie (strict <), ascending-k first-minimum (strict <).+ NOMASK skips the tail-group mask selects when GP == G (pow2 group size).+ """+ pid = tl.program_id(0)+ c_off = pid * BC + tl.arange(0, BC)+ g_off = tl.arange(0, GP)+ cm = c_off < C+ gm = g_off < G+ d = tl.load(dn_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)+ if HAS_QW:+ qwt = tl.load(qw_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)+ best = tl.zeros((BC,), dtype=tl.float32) + float("inf")+ bk = tl.zeros((BC,), dtype=tl.int32)+ bmir = tl.zeros((BC,), dtype=tl.int32)+ for k in range(0, K):+ f = tl.load(fracs_ptr + k)+ invf = tl.load(invf_ptr + k)+ x = d * invf+ r = tl.inline_asm_elementwise(+ "cvt.rni.f32.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1+ )+ q_std = tl.minimum(tl.maximum(r, -NMAX), NMAX - 1.0)+ e_std = f * q_std - d+ q_mir = tl.minimum(tl.maximum(r, -(NMAX - 1.0)), NMAX)+ e_mir = f * q_mir - d+ if HAS_QW:+ se = e_std * e_std * qwt+ sm = e_mir * e_mir * qwt+ else:+ se = e_std * e_std+ sm = e_mir * e_mir+ if NOMASK:+ l_std = tl.sum(se, axis=1)+ l_mir = tl.sum(sm, axis=1)+ else:+ l_std = tl.sum(tl.where(gm[None, :], se, 0.0), axis=1)+ l_mir = tl.sum(tl.where(gm[None, :], sm, 0.0), axis=1)+ mir = l_mir < l_std+ loss = tl.where(mir, l_mir, l_std)+ better = loss < best+ best = tl.where(better, loss, best)+ bk = tl.where(better, k, bk)+ bmir = tl.where(better, mir.to(tl.int32), bmir)+ tl.store(loss_ptr + c_off, best, mask=cm)+ tl.store(idx_ptr + c_off, bk, mask=cm)+ tl.store(mirror_ptr + c_off, bmir, mask=cm)+++def sym_search_shared_triton(dn, qw, fracs, invf, nmax):+ """Coarse-stage symmetric search in normalized space on CUDA (see module docstring)."""+ assert triton is not None, f"triton is not importable: {_TRITON_IMPORT_ERROR}"+ assert dn.is_cuda and dn.dtype == torch.float32 and dn.is_contiguous()+ assert fracs.is_contiguous() and fracs.dtype == torch.float32 and fracs.ndim == 1+ assert invf.is_contiguous() and invf.dtype == torch.float32 and invf.shape == fracs.shape+ if qw is not None:+ assert qw.dtype == torch.float32 and qw.is_contiguous() and qw.shape == dn.shape+ _dev = dn.device+ assert fracs.device == _dev, f"sym_shared: fracs on {fracs.device} but dn on {_dev}"+ assert invf.device == _dev, f"sym_shared: invf on {invf.device} but dn on {_dev}"+ if qw is not None:+ assert qw.device == _dev, f"sym_shared: qw on {qw.device} but dn on {_dev}"++ c, g = dn.shape+ k = fracs.numel()+ gp = _next_pow2(g)+ bc = max(1, min(64, 4096 // gp))+ loss = torch.empty((c,), device=_dev, dtype=torch.float32)+ idx = torch.empty((c,), device=_dev, dtype=torch.int32)+ mirror = torch.empty((c,), device=_dev, dtype=torch.int32)+ with torch.cuda.device(_dev): # Triton launches on the CURRENT device+ _sym_shared_kernel[(triton.cdiv(c, bc),)](+ dn,+ qw if qw is not None else dn, # dummy pointer when unweighted+ fracs,+ invf,+ loss,+ idx,+ mirror,+ c,+ g,+ k,+ NMAX=nmax,+ HAS_QW=qw is not None,+ BC=bc,+ GP=gp,+ NOMASK=gp == g,+ num_warps=4,+ )+ return loss, idx.long(), mirror.bool()+++def _launch_sweep(data, qw, scales, loss, zp, c, g, k, nz, maxq, bc, gp):+ _neuqi_sweep_kernel[(triton.cdiv(c, bc),)](+ data,+ qw if qw is not None else data, # dummy pointer when unweighted (dead lanes)+ scales,+ loss,+ zp,+ c,+ g,+ k,+ nz,+ MAXQ=maxq,+ HAS_QW=qw is not None,+ BC=bc,+ GP=gp,+ NOMASK=gp == g,+ ZU=maxq + 1 <= 32,+ num_warps=4,+ )+ return loss, zpdiff --git a/docs/environments.md b/docs/environments.mdindex c47fe1cb0..7a45324f1 100644--- a/docs/environments.md+++ b/docs/environments.md@@ -130,6 +130,36 @@ export AR_ENABLE_ACT_MINMAX_TUNING=1export AR_SEARCH_SCALE_RATIO=0.75```+### AR_NEUQI_COARSE+- **Description**: Number of coarse-stage candidates in the NeUQI grid search (`--enable_neuqi`, arXiv 2505.17595). The coarse stage scans a wide log-spaced grid; the fine stage refines around the winner (`AR_NEUQI_FINE`). Applies to both the asymmetric joint (scale, zero-point) search and the symmetric two-stage scale search.+- **Default**: `256` on accelerated backends (Triton/torch.compile), `64` on the eager sweep — the eager path evaluates ~coarse x fine candidates per group, so it narrows the unpinned default (measured quality is flat between the two grids)+- **Valid Values**: positive integer+- **Usage**: Lower for faster searches at the cost of grid coverage.++```bash+export AR_NEUQI_COARSE=128+```++### AR_NEUQI_FINE+- **Description**: Number of fine-stage candidates in the NeUQI grid search, refined around the coarse winner. See `AR_NEUQI_COARSE`.+- **Default**: `64` on accelerated backends (Triton/torch.compile), `32` on the eager sweep+- **Valid Values**: positive integer+- **Usage**: Lower for faster searches at the cost of final resolution.++```bash+export AR_NEUQI_FINE=32+```++### AR_NEUQI_BACKEND+- **Description**: Backend override for the NeUQI candidate sweep. `auto` picks the fastest available (extension Triton kernel, then torch.compile-fused, then eager) and latches down permanently on the first kernel failure. `eager`/`compile`/`triton` force one backend for debugging or benchmarking.+- **Default**: `auto`+- **Valid Values**: `auto`, `eager`, `compile`, `triton`+- **Usage**: Escape hatch for driver/hardware incompatibilities; `auto` is recommended.++```bash+export AR_NEUQI_BACKEND=eager+```+### AR_DYNAMO_CACHE_SIZE_LIMIT- **Description**: Minimum value to which `torch._dynamo`'s `cache_size_limit`, `accumulated_cache_size_limit`, and `recompile_limit` are bumped when `torch.compile` is enabled (the default except on Windows). The same compiled quant function is reused across every linear layer in a transformer block (q/k/v/o_proj, gate/up/down_proj, ...) but each layer has a different weight shape, so per-layer static recompiles quickly exceed dynamo's default limit (8) and trigger a noisy fallback to eager. Raising the limit keeps static-shape compilation (best perf) and just allows more cache entries.- **Default**: `16`diff --git a/docs/environments_CN.md b/docs/environments_CN.mdindex 2995f01a6..264c88fd7 100644--- a/docs/environments_CN.md+++ b/docs/environments_CN.md@@ -130,6 +130,36 @@ export AR_ENABLE_ACT_MINMAX_TUNING=1export AR_SEARCH_SCALE_RATIO=0.75```+### AR_NEUQI_COARSE+- **描述**:NeUQI 网格搜索(`--enable_neuqi`,arXiv 2505.17595)粗阶段的候选数量。粗阶段扫描较宽的对数网格,细阶段在胜者附近细化(`AR_NEUQI_FINE`)。同时作用于非对称联合 (scale, zero-point) 搜索与对称两阶段 scale 搜索。+- **默认值**:加速后端(Triton/torch.compile)为 `256`,eager 扫描为 `64` —— eager 路径每 group 需评估约 coarse x fine 个候选,因此未显式指定时使用更窄的默认网格(两种网格的实测质量持平)+- **取值**:正整数+- **用法**:调低可加快搜索,代价是网格覆盖范围变小。++```bash+export AR_NEUQI_COARSE=128+```++### AR_NEUQI_FINE+- **描述**:NeUQI 网格搜索细阶段的候选数量,在粗阶段胜者附近细化。参见 `AR_NEUQI_COARSE`。+- **默认值**:加速后端(Triton/torch.compile)为 `64`,eager 扫描为 `32`+- **取值**:正整数+- **用法**:调低可加快搜索,代价是最终分辨率下降。++```bash+export AR_NEUQI_FINE=32+```++### AR_NEUQI_BACKEND+- **描述**:NeUQI 候选扫描的后端覆盖。`auto` 选择最快的可用后端(扩展 Triton 内核 → torch.compile 融合 → eager),并在内核首次失败时永久回退。`eager`/`compile`/`triton` 强制指定后端,用于调试或基准测试。+- **默认值**:`auto`+- **取值**:`auto`、`eager`、`compile`、`triton`+- **用法**:驱动/硬件兼容性逃生口;推荐使用 `auto`。++```bash+export AR_NEUQI_BACKEND=eager+```+### AR_DYNAMO_CACHE_SIZE_LIMIT- **描述**:在开启 `torch.compile`(除 Windows 外默认开启)时,将 `torch._dynamo` 的 `cache_size_limit`、`accumulated_cache_size_limit` 与 `recompile_limit` 提升到的最小值。同一个被编译的量化函数会被 transformer block 内的所有 linear 层(q/k/v/o_proj、gate/up/down_proj 等)复用,但每层权重 shape 不同,按层的静态重编译会很快超过 dynamo 默认上限(8),导致打印告警并退回 eager。提高该上限可保留静态 shape 编译(性能最佳),仅增加缓存条目数。- **默认值**:`16`diff --git a/docs/neuqi_acc.md b/docs/neuqi_acc.mdnew file mode 100644index 000000000..09b8d03d9--- /dev/null+++ b/docs/neuqi_acc.md@@ -0,0 +1,30 @@+### NeUQI Accuracy Validation++**NeUQI** ([arXiv 2505.17595](https://arxiv.org/abs/2505.17595)) is a calibration-free grid search for the optimized-RTN path, enabled with `--enable_neuqi`: asymmetric layers run a joint (scale, integer zero-point) search, symmetric layers a two-stage signed scale search; on the zero-shot path both are weighted by the activation imatrix (collected automatically), and with `iters > 0` the anchor uses the activation imatrix when the tuning path collects one (SignRoundV2's `enable_alg_ext`), running unweighted otherwise. With `iters > 0`, the search result anchors the SignRound tuning grid (frozen init).++#### Protocol++KL divergence between the BF16 base model and the dequantized W4A16 checkpoint over the **full vocabulary**, computed in **fp32** on openwebtext (8 sequences × 8192 tokens, 65,528 positions). Both passes run through the same evaluation stack, so the only difference is the quantization; **lower is better**. `top1` is the next-token agreement rate, `top5` the Jaccard similarity of the top-5 sets. Wall times are single-GPU runs on one RTX 3090 (24 GB) for Qwen3.8-27B (W4A16, group size 128).++#### Results — iters=0 (calibration-free zero-shot)++| recipe | mean | median | p90 | p99 | top1 | top5 | wall |+|:--|--:|--:|--:|--:|--:|--:|--:|+| asym, NeUQI | **0.02579** | 0.01361 | 0.05046 | 0.21753 | 0.9234 | 0.8514 | 11m |+| sym, NeUQI | **0.02734** | 0.01468 | 0.05358 | 0.22014 | 0.9213 | 0.8463 | **9m** |+| sym, incumbent OptRTN search | 0.02852 | 0.01533 | 0.05678 | 0.23453 | 0.9191 | 0.8429 | 13m19s |++- The symmetric two-stage search **strictly dominates the incumbent search**: −4.1% mean KL, −32% wall time, and better agreement at every quantile.+- There is no incumbent asymmetric search — `--enable_neuqi` adds the optimized-init capability for asymmetric layers; the quality ladder is flat from 64/32 to 256/64, so the [backend-aware default](./environments.md) narrows the eager sweep to 64/32 at no measured cost.++#### Results — iters > 0 (SignRoundV2 tuning, `enable_alg_ext`)++| recipe | mean | median | p90 | p99 | top1 | top5 | wall |+|:--|--:|--:|--:|--:|--:|--:|--:|+| asym, SignRoundV2 + NeUQI, iters=200 (64/32 grid) | **0.01921** | 0.00905 | 0.03591 | 0.17297 | 0.9370 | 0.8727 | 2h57m21s |+| asym, SignRoundV2 + NeUQI | **0.02106** | 0.01050 | 0.04071 | 0.18038 | 0.9317 | 0.8661 | 49m07s |+| sym, SignRoundV2 + NeUQI | **0.02191** | 0.01116 | 0.04237 | 0.18473 | 0.9304 | 0.8627 | 56m52s |+| sym, SignRoundV2 without NeUQI | 0.02203 | 0.01112 | 0.04307 | 0.18239 | 0.9295 | 0.8630 | 55m40s |++- At `iters=50` the NeUQI anchor is **quality-neutral on sym** (0.02191 vs 0.02203 without NeUQI): SignRoundV2's own tuning dominates once it runs, so the anchor's role is a no-cost initialization, not an accuracy lever. (An earlier no-NeUQI baseline of 0.02626 was not SignRoundV2-equivalent — most of its gap was the quantizer version, not NeUQI.)+- Asymmetric tuning lands at 0.02106, on par with the best prior asymmetric recipe family at the same iteration budget; the dose-response continues to 0.01921 at 200 iterations. (No SignRoundV2 asym no-NeUQI baseline has been measured; the asym anchor claim is parity-with-prior, not an A/B.)diff --git a/docs/step_by_step.md b/docs/step_by_step.mdindex 628b3e24e..8a602b0b0 100755--- a/docs/step_by_step.md+++ b/docs/step_by_step.md@@ -600,6 +600,8 @@ AutoRound also supports Optimized RTN (Round-To-Nearest) mode for fast, calibratFor the GGUF format, we have optimized the RTN algorithm inspired by llamacpp. To use the original (pure) RTN algorithm instead, enable the `--disable_opt_rtn` option.+Passing `--enable_neuqi` opts the optimized path into the **NeUQI** grid search ([arXiv 2505.17595](https://arxiv.org/abs/2505.17595)): asymmetric layers run a joint (scale, integer zero-point) search and symmetric layers a two-stage signed scale search, both weighted by the activation imatrix on the zero-shot path, which is collected automatically (with `iters > 0` the anchor uses the imatrix when the tuning path collects one, running unweighted otherwise). With `iters > 0`, the search result anchors the SignRound tuning grid (frozen init). Grid sizes are tunable via `AR_NEUQI_COARSE`/`AR_NEUQI_FINE` (see [environments](./environments.md)); the unpinned default is backend-aware (wide only on the Triton/torch.compile lanes). Accuracy and wall-time results are shown in [NeUQI accuracy validation](./neuqi_acc.md).+#### CLI UsageTwo dedicated CLI entry points are provided as shortcuts:diff --git a/docs/step_by_step_CN.md b/docs/step_by_step_CN.mdindex 6dde8eccf..e9bcbb7ee 100755--- a/docs/step_by_step_CN.md+++ b/docs/step_by_step_CN.md@@ -590,6 +590,8 @@ AutoRound 还提供优化版 RTN(Round-To-Nearest,就近舍入)模式,对于 GGUF 格式,我们参考 llamacpp 的思路,优化了 RTN 算法。若需使用原始(非优化)RTN 算法,开启 `--disable_opt_rtn` 即可。+在优化路径上开启 `--enable_neuqi` 即可启用 **NeUQI** 网格搜索([arXiv 2505.17595](https://arxiv.org/abs/2505.17595)):非对称层执行联合 (scale, 整数 zero-point) 搜索,对称层执行两阶段带符号 scale 搜索,在零样本路径(`iters=0`)上二者均以激活 imatrix 加权(imatrix 会自动采集;`iters > 0` 时锚点在调优路径采集到 imatrix 时使用它,否则不加权)。当 `iters > 0` 时,搜索结果将作为 SignRound 调优网格的锚点(frozen init)。网格规模可通过 `AR_NEUQI_COARSE`/`AR_NEUQI_FINE` 调整(参见[《环境变量》](./environments_CN.md));未显式指定时的默认值与后端相关(仅在 Triton/torch.compile 路径使用宽网格)。精度与耗时结果详见[《NeUQI 精度验证》](./neuqi_acc.md)。+#### 命令行使用我们提供了两个专用的 CLI 入口作为快捷方式:diff --git a/test/unit/test_cpu/quantization/test_neuqi.py b/test/unit/test_cpu/quantization/test_neuqi.pynew file mode 100644index 000000000..b4e4b63c3--- /dev/null+++ b/test/unit/test_cpu/quantization/test_neuqi.py@@ -0,0 +1,1699 @@+# Copyright (c) 2026 Intel Corporation+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from types import SimpleNamespace++import pytest+import torch++from auto_round.data_type.int import quant_tensor_asym+from auto_round.data_type.neuqi import _log_search_engaged, neuqi_search_scale_zero, quant_tensor_opt_rtn_asym+from auto_round.data_type.register import QUANT_FUNC_WITH_DTYPE+from auto_round.data_type.utils import get_quant_func+++def _heavy_tailed_data(n_groups=64, group_size=128, seed=0):+ gen = torch.Generator().manual_seed(seed)+ data = torch.randn(n_groups, group_size, generator=gen)+ # inject a few large outliers per group to make min/max suboptimal+ data[:, :5] *= 120.0+ return data+++def _mse(x, y):+ return ((x.float() - y.float()) ** 2).mean().item()+++def _oracle_loss_fp64(data, qw, scale, zp, maxq):+ """fp64 loss of one group under fixed (scale, zp) -- the tie-break oracle."""+ r = torch.round(data.double() / scale.double()).clamp_(min=-maxq - 1, max=2 * maxq + 1)+ q = (r + zp.double()).clamp_(0, maxq)+ deq = scale.double() * (q - zp.double())+ loss = (deq - data.double()) ** 2+ if qw is not None:+ loss = loss * qw.double()+ return loss.sum(-1)+++def _assert_launch_latches(monkeypatch, N, attempt, impl_attr, checked_attr, broken_attr, gates, tail_args):+ """A raising kernel/compile launch must return None, latch its broken flag,+ and never retry (all three latch sites share this contract)."""+ calls = {"n": 0}++ def _boom(*a, **k):+ calls["n"] += 1+ raise RuntimeError("launch failed")++ monkeypatch.setattr(N, impl_attr, _boom)+ monkeypatch.setattr(N, checked_attr, True)+ for gate_name, gate_ret in gates.items():+ monkeypatch.setattr(N, gate_name, lambda *a, _r=gate_ret, **k: _r)+ fake = SimpleNamespace(is_cuda=True, device=torch.device("cuda"))+ out = getattr(N, attempt)(fake, *tail_args)+ assert out is None+ assert getattr(N, broken_attr)+ out2 = getattr(N, attempt)(fake, *tail_args)+ assert out2 is None and calls["n"] == 1 # latched: no second launch+++def _reset_neuqi_state(defaults):+ """Reset neuqi module latch state (call before AND after each test)."""+ import auto_round.data_type.neuqi as N++ for attr, default in defaults.items():+ if hasattr(N, attr):+ setattr(N, attr, default)+++class TestNeuqiAsymSearch:+ def test_search_output_contract(self):+ """Validity/integrality/shape contract of the joint search outputs."""+ import auto_round.data_type.neuqi as N++ torch.manual_seed(5)+ data = torch.randn(512, 64) * 0.05+ qw = torch.rand(512, 64) + 0.1+ scale, zp = N.neuqi_search_scale_zero(data, bits=4, qw=qw, coarse_n=4, fine_n=3)+ assert scale.shape == (512, 1) and zp.shape == (512, 1)+ assert torch.all(zp == zp.round())+ assert (zp >= 0).all() and (zp <= 15).all()+ assert torch.all(scale > 0)++ data = _heavy_tailed_data(n_groups=32)+ scale, zp = neuqi_search_scale_zero(data, bits=4)+ assert torch.all(zp == zp.round())+ assert torch.all(zp >= 0) and torch.all(zp <= 15)+ assert torch.all(scale > 0)++ gen = torch.Generator().manual_seed(2)+ for shape in [(4, 33, 128), (17, 65), (3, 5, 7, 11)]:+ tensor = torch.randn(*shape, generator=gen)+ qdq, scale, zp = quant_tensor_opt_rtn_asym(tensor.clone(), bits=4, group_size=32)+ assert qdq.shape == tensor.shape++ def test_beats_minmax_on_heavy_tailed_data(self):+ data = _heavy_tailed_data()+ bits, group_size = 4, 128++ minmax_qdq, _, _ = quant_tensor_asym(data.clone(), bits=bits, group_size=group_size)+ neuqi_qdq, scale, zp = quant_tensor_opt_rtn_asym(data.clone(), bits=bits, group_size=group_size)++ assert _mse(neuqi_qdq, data) < _mse(minmax_qdq, data)++ def test_close_to_dense_exhaustive_reference(self):+ """On a small case the search must approach a dense exhaustive grid."""+ gen = torch.Generator().manual_seed(3)+ n, g, bits = 8, 16, 2+ maxq = 2**bits - 1+ data = torch.randn(n, g, generator=gen)+ data[:, 0] *= 50.0++ scale, zp = neuqi_search_scale_zero(data, bits=bits)+ r = torch.round(data / scale)+ q = (r + zp).clamp(0, maxq)+ our_loss = (((scale * (q - zp)) - data) ** 2).sum().item()++ # brute force: dense additive scale grid per group x all integer zero points+ best = torch.full((n,), float("inf"))+ wmin = torch.clamp(data.min(-1).values, max=0)+ wmax = torch.clamp(data.max(-1).values, min=0)+ s0 = ((wmax - wmin) / maxq).clamp(min=1e-5)+ for i in range(401):+ frac = 0.05 + (1.0 - 0.05) * i / 400+ s = (s0 * frac).unsqueeze(-1)+ r = torch.round(data / s)+ for z in range(maxq + 1):+ q = (r + z).clamp(0, maxq)+ loss = ((s * (q - z) - data) ** 2).sum(-1)+ best = torch.minimum(best, loss)+ exhaustive_loss = best.sum().item()+ # within 1% of the (much denser) exhaustive grid+ assert our_loss <= exhaustive_loss * 1.01 + 1e-9++ def test_weighting_changes_solution(self):+ """A per-element weight that de-emphasizes outliers must pull the scale down."""+ gen = torch.Generator().manual_seed(1)+ data = torch.randn(16, 64, generator=gen)+ data[:, :4] *= 100.0++ qw = torch.ones_like(data)+ qw[:, :4] = 1e-4 # ignore outliers++ # grid pinned to the measured 64/32: the scale comparison below is a+ # heuristic (only loss dominance is a hard invariant) and finer grids+ # shift near-tie selections between the two runs+ scale_plain, _ = neuqi_search_scale_zero(data, bits=4, coarse_n=64, fine_n=32)+ scale_weighted, _ = neuqi_search_scale_zero(data, bits=4, qw=qw, coarse_n=64, fine_n=32)+ # ignoring outliers must never increase the scale and must shrink it somewhere+ assert torch.all(scale_weighted <= scale_plain + 1e-6)+ assert torch.any(scale_weighted < scale_plain)++ def weighted_loss(scale):+ maxq = 15+ r = torch.round(data / scale).clamp(-maxq - 1, 2 * maxq + 1)+ losses = []+ for z in range(maxq + 1):+ q = (r + z).clamp(0, maxq)+ losses.append((((scale * (q - z) - data) ** 2) * qw).sum(-1))+ return torch.stack(losses, -1).min(-1).values.sum()++ assert weighted_loss(scale_weighted) < weighted_loss(scale_plain)++ def test_constant_group_is_exact(self):+ data = torch.full((4, 128), 0.25)+ qdq, scale, zp = quant_tensor_opt_rtn_asym(data.clone(), bits=4, group_size=128)+ # the fp16 scale cast limits exactness to one quantization step (0.25 / 15)+ assert (qdq - data).abs().max() <= 0.25 / 15+ assert torch.all(scale > 0)+++class TestNeuqiAsymIntegration:+ """Asym joint search through the quant-func API: output validity across+ bit widths and imatrix weighting that steers the solution."""++ def test_output_validity_and_imatrix_weighting(self):+ data = _heavy_tailed_data(n_groups=32)+ imatrix = torch.rand(128) + 0.1++ qdq_plain, _, _ = quant_tensor_opt_rtn_asym(data.clone(), bits=4, group_size=128)+ qdq_im, _, _ = quant_tensor_opt_rtn_asym(data.clone(), bits=4, group_size=128, imatrix=imatrix)++ qw = imatrix.expand(32, 128)+ weighted = lambda q: (((q - data) ** 2) * qw).sum().item() # noqa: E731+ assert weighted(qdq_im) < weighted(qdq_plain)++ for bits in (2, 3, 4, 8):+ d = _heavy_tailed_data(n_groups=8, group_size=64, seed=bits)+ qdq, scale, zp = quant_tensor_opt_rtn_asym(d.clone(), bits=bits, group_size=64)+ maxq = 2**bits - 1+ assert torch.all(zp >= 0) and torch.all(zp <= maxq)+ assert torch.all(scale > 0)+ assert qdq.shape == d.shape+++class TestNeuqiAsymFusedSweep:+ """The torch.compile-fused zero-point sweep must reproduce the eager reference.++ ``AR_NEUQI_BACKEND=compile`` routes every chunk through the fused expression.+ Selections must match the eager sweep; when compilation is unavailable in the+ environment (e.g. no host C++ compiler for Inductor) the dispatch must fall+ back to the eager expression and still return correct results."""++ def _search_with_backend(self, monkeypatch, backend, data, bits, qw, **kwargs):+ import auto_round.data_type.neuqi as N+ from auto_round import envs++ monkeypatch.setenv("AR_NEUQI_BACKEND", backend)+ monkeypatch.setattr(N, "_fused_zp_fns", {})+ monkeypatch.setattr(N, "_fused_zp_broken", False)+ scale, zp = N.neuqi_search_scale_zero(data.clone(), bits, qw=qw.clone() if qw is not None else None, **kwargs)+ return scale, zp, N++ @pytest.mark.enable_torch_compile+ def test_fused_matches_eager(self, monkeypatch):+ """One setup, the structural combo set: bits=2 (smallest zp grid),+ bits=4 (shipping default), bits=8 (zp grid of 256, largest sweep+ tensors), and one weighted arm at the extreme -- the qw multiply is+ size-independent, so a single weighted combo covers it. Selections+ must match the eager sweep; a flip is acceptable only between+ fp64-tied candidates."""+ for bits, weighted in ((2, False), (4, False), (8, False), (8, True)):+ gen = torch.Generator().manual_seed(bits * 10 + int(weighted))+ n, g = 128, 64+ data = torch.randn(n, g, generator=gen)+ data[:, :5] *= 120.0 # heavy tails: near-tie zero points are common here+ qw = torch.rand(n, g, generator=gen) + 0.1 if weighted else None++ scale_e, zp_e, _ = self._search_with_backend(monkeypatch, "eager", data, bits, qw, coarse_n=8, fine_n=4)+ scale_f, zp_f, N = self._search_with_backend(monkeypatch, "compile", data, bits, qw, coarse_n=8, fine_n=4)+ if N._fused_zp_broken or not N._fused_zp_fns:+ pytest.skip("torch.compile/inductor unavailable in this environment")++ mismatch = (zp_e != zp_f).logical_or(scale_e != scale_f).nonzero().flatten()+ for i_ in mismatch.tolist():+ loss_e = _oracle_loss_fp64(+ data[i_], qw[i_] if qw is not None else None, scale_e[i_], zp_e[i_], 2**bits - 1+ )+ loss_f = _oracle_loss_fp64(+ data[i_], qw[i_] if qw is not None else None, scale_f[i_], zp_f[i_], 2**bits - 1+ )+ assert (loss_f - loss_e).abs().item() <= 1e-6 * loss_e.abs().item()++ def test_last_layout_matches_mid_reference(self, monkeypatch):+ """The CUDA layout (group axis last) must match the reference layout's+ selections up to fp64 ties -- it runs on CUDA, its numerics are validated+ here on CPU."""+ import auto_round.data_type.neuqi as N+ from auto_round import envs++ gen = torch.Generator().manual_seed(31)+ data = torch.randn(128, 64, generator=gen)+ data[:, :4] *= 90.0+ qw = torch.rand(128, 64, generator=gen) + 0.1++ monkeypatch.setattr(N, "_fused_zp_fns", {})+ monkeypatch.setattr(N, "_fused_zp_broken", False)+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")+ monkeypatch.setattr(N, "_zp_expr_for", lambda dt: N._zp_expr_mid)+ scale_m, zp_m = N.neuqi_search_scale_zero(data.clone(), bits=4, qw=qw, coarse_n=16, fine_n=4)++ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")+ monkeypatch.setattr(N, "_zp_expr_for", lambda dt: N._zp_expr_last)+ scale_l, zp_l = N.neuqi_search_scale_zero(data.clone(), bits=4, qw=qw, coarse_n=16, fine_n=4)++ if N._fused_zp_broken or not N._fused_zp_fns:+ pytest.skip("torch.compile/inductor unavailable in this environment")++ mismatch = (zp_m != zp_l).logical_or(scale_m != scale_l).nonzero().flatten()+ maxq = 15+ for i in mismatch.tolist():+ loss_m = _oracle_loss_fp64(data[i], qw[i], scale_m[i], zp_m[i], maxq)+ loss_l = _oracle_loss_fp64(data[i], qw[i], scale_l[i], zp_l[i], maxq)+ assert (loss_l - loss_m).abs().item() <= 1e-6 * loss_m.abs().item()++ def test_compile_failure_falls_back_to_eager(self, monkeypatch):+ """A broken host toolchain must degrade to the eager sweep, not crash."""+ import auto_round.data_type.neuqi as N+ from auto_round import envs++ data = _heavy_tailed_data(n_groups=16, seed=11)++ def _boom(*args, **kwargs):+ raise RuntimeError("simulated inductor failure")++ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")+ monkeypatch.setattr(N, "_get_fused_zp_fn", _boom)+ monkeypatch.setattr(N, "_fused_zp_fns", {})+ monkeypatch.setattr(N, "_fused_zp_broken", False)++ scale_f, zp_f = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=4, fine_n=2)+ assert N._fused_zp_broken is True # permanently latched after the first failure+ monkeypatch.setattr(N, "_fused_zp_broken", False)+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")+ scale_e, zp_e = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=4, fine_n=2)+ torch.testing.assert_close(scale_f, scale_e)+ torch.testing.assert_close(zp_f, zp_e)+++class TestNeuqiAsymBatchedSweep:+ """The all-candidates batched sweep must reproduce the per-candidate search.++ The batched sweep evaluates every coarse/fine candidate in one fused+ call per group chunk ([C, K, Z, g], min over z fused in-kernel, then first-min+ over candidates). Selections must match the per-candidate sweep; ties may+ resolve differently only within fp64-oracle tolerance."""++ def _search(self, monkeypatch, backend, batch, data, bits, qw, **kwargs):+ import auto_round.data_type.neuqi as N+ from auto_round import envs++ monkeypatch.setenv("AR_NEUQI_BACKEND", backend)+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: batch == "on")+ monkeypatch.setattr(N, "_fused_zp_fns", {})+ monkeypatch.setattr(N, "_fused_zp_broken", False)+ monkeypatch.setattr(N, "_batched_broken", False)+ scale, zp = N.neuqi_search_scale_zero(data.clone(), bits, qw=qw.clone() if qw is not None else None, **kwargs)+ return scale, zp, N++ @pytest.mark.enable_torch_compile+ def test_batched_matches_per_candidate(self, monkeypatch):+ """One setup, the structural combo set (see test_fused_matches_eager):+ batched selections must match the per-candidate sweep up to+ fp64-oracle ties."""+ for bits, weighted in ((2, False), (4, False), (8, False), (8, True)):+ gen = torch.Generator().manual_seed(bits * 7 + int(weighted))+ n, g = 128, 64+ data = torch.randn(n, g, generator=gen)+ data[:, :4] *= 100.0+ qw = torch.rand(n, g, generator=gen) + 0.1 if weighted else None++ scale_ref, zp_ref, _ = self._search(monkeypatch, "eager", "off", data, bits, qw, coarse_n=8, fine_n=3)+ scale_b, zp_b, N = self._search(monkeypatch, "compile", "on", data, bits, qw, coarse_n=8, fine_n=3)+ if N._batched_broken or N._zp_expr_batched not in N._fused_zp_fns:+ pytest.skip("torch.compile/inductor unavailable in this environment")++ mismatch = (zp_ref != zp_b).logical_or(scale_ref != scale_b).nonzero().flatten()+ maxq = 2**bits - 1+ for i_ in mismatch.tolist():+ loss_ref = _oracle_loss_fp64(+ data[i_], qw[i_] if qw is not None else None, scale_ref[i_], zp_ref[i_], maxq+ )+ loss_b = _oracle_loss_fp64(data[i_], qw[i_] if qw is not None else None, scale_b[i_], zp_b[i_], maxq)+ assert (loss_b - loss_ref).abs().item() <= 1e-6 * loss_ref.abs().item()++ def test_batch_failure_falls_back_to_per_candidate(self, monkeypatch):+ """A batched-call failure must degrade to the per-candidate sweep, not crash."""+ import auto_round.data_type.neuqi as N+ from auto_round import envs++ data = _heavy_tailed_data(n_groups=24, group_size=64, seed=17)++ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: True)+ monkeypatch.setattr(N, "_fused_zp_fns", {})+ monkeypatch.setattr(N, "_fused_zp_broken", False)+ monkeypatch.setattr(N, "_batched_broken", False)+ monkeypatch.setattr(N, "_eval_batched_chunk", lambda *a, **k: None) # simulate failure++ scale_f, zp_f = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=6, fine_n=2)++ # reference: per-candidate sweep (batching off never touches the patched fn)+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: False)+ scale_ref, zp_ref = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=6, fine_n=2)+ torch.testing.assert_close(scale_f, scale_ref)+ torch.testing.assert_close(zp_f, zp_ref)+++class TestNeuqiAsymTritonSweep:+ """The extension Triton sweep integrates behind the batched dispatch.++ Triton kernels cannot run on CPU hosts, so the plumbing (scale construction,+ first-min bookkeeping, latch fallbacks, policy) is tested with a mock that+ honors the kernel's contract; the kernel itself is exercised on CUDA by+ ``test/unit/test_cuda/quantization/test_neuqi_triton.py`` and by the+ self-contained GPU benchmark."""++ def _mock_sweep_from(self, monkeypatch, impl):+ import auto_round.data_type.neuqi as N++ monkeypatch.setattr(N, "_triton_checked", True)+ monkeypatch.setattr(N, "_triton_sweep", impl)+ monkeypatch.setattr(N, "_triton_broken", False)++ def test_triton_mock_matches_reference(self, monkeypatch):+ """Driving the batched passes through a contract-honoring mock must+ reproduce the per-candidate reference selections exactly."""+ import auto_round.data_type.neuqi as N+ from auto_round import envs++ def mock(data, qw, scales, maxq): # same contract as neuqi_sweep_triton+ zp_grid = torch.arange(0, maxq + 1, device=data.device, dtype=torch.float32)+ return N._zp_expr_batched(data, qw, scales, zp_grid, maxq)++ self._mock_sweep_from(monkeypatch, mock)+ data = _heavy_tailed_data(n_groups=64, group_size=64, seed=23)+ qw = torch.rand(64, 64) + 0.1++ monkeypatch.setenv("AR_NEUQI_BACKEND", "triton")+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: True)+ monkeypatch.setattr(N, "_fused_zp_fns", {})+ monkeypatch.setattr(N, "_fused_zp_broken", False)+ monkeypatch.setattr(N, "_batched_broken", False)+ scale_t, zp_t = N.neuqi_search_scale_zero(data.clone(), bits=4, qw=qw, coarse_n=8, fine_n=3)++ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: False)+ scale_ref, zp_ref = N.neuqi_search_scale_zero(data.clone(), bits=4, qw=qw, coarse_n=8, fine_n=3)+ torch.testing.assert_close(scale_t, scale_ref)+ torch.testing.assert_close(zp_t, zp_ref)++ def test_triton_failure_latches_to_compile_path(self, monkeypatch):+ """A raising Triton sweep must latch off and finish via the torch paths."""+ import auto_round.data_type.neuqi as N+ from auto_round import envs++ def boom(data, qw, scales, maxq):+ raise RuntimeError("simulated triton failure")++ self._mock_sweep_from(monkeypatch, boom)+ data = _heavy_tailed_data(n_groups=32, group_size=64, seed=29)++ monkeypatch.setenv("AR_NEUQI_BACKEND", "triton")+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: True)+ monkeypatch.setattr(N, "_fused_zp_fns", {})+ monkeypatch.setattr(N, "_fused_zp_broken", False)+ monkeypatch.setattr(N, "_batched_broken", False)+ scale_f, zp_f = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=6, fine_n=2)+ assert N._triton_broken is True++ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")+ scale_ref, zp_ref = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=6, fine_n=2)+ torch.testing.assert_close(scale_f, scale_ref)+ torch.testing.assert_close(zp_f, zp_ref)++ def test_resolver_logs_engagement(self, monkeypatch):+ """A successful resolution announces itself: the Triton arm is silent+ otherwise, and live runs must be able to tell which backend served."""+ import sys+ import types++ import auto_round.data_type.neuqi as N++ stub = types.ModuleType("auto_round_extension.triton.neuqi_sweep")+ stub.neuqi_sweep_triton = object()+ monkeypatch.setitem(sys.modules, "auto_round_extension.triton.neuqi_sweep", stub)+ recorded = []+ monkeypatch.setattr(N, "logger", types.SimpleNamespace(info=recorded.append, debug=recorded.append))+ monkeypatch.setattr(N, "_triton_checked", False)+ monkeypatch.setattr(N, "_triton_sweep", None)+ assert N._triton_sweep_fn() is stub.neuqi_sweep_triton+ assert any("Triton zero-point sweep engaged" in str(r) for r in recorded)+++class TestEnableNeuqiAPI:+ """enable_neuqi (RTNConfig/SignRoundConfig kwarg + --enable_neuqi CLI flag):+ opt-in gate for the NeUQI grid search on BOTH symmetry classes."""++ def test_dispatch_table(self):+ """Off: plain asym min/max + incumbent sym; opted in: joint search on asym."""+ from auto_round.data_type.int import quant_tensor_asym as plain_asym++ func, name = get_quant_func("int", 4, False, disable_opt_rtn=False, group_size=None, iters=0)+ assert "opt_rtn" not in name and name.endswith("_asym") and func is plain_asym+ func, name = get_quant_func("int", 4, False, disable_opt_rtn=False, group_size=None, iters=0, enable_neuqi=True)+ assert name == "opt_rtn_int_asym" and func is QUANT_FUNC_WITH_DTYPE["opt_rtn_int_asym"]+ _, sym_name = get_quant_func("int", 4, True, disable_opt_rtn=False, group_size=None, iters=0)+ assert sym_name == "opt_rtn_int_sym"++ def test_config_flag_semantics(self):+ """enable_neuqi defaults off and accepts True on both symmetry classes."""+ from auto_round.algorithms.quantization.rtn.config import RTNConfig++ assert RTNConfig(bits=4, group_size=32, sym=False).enable_neuqi is False+ on = RTNConfig(bits=4, group_size=32, sym=False, disable_opt_rtn=False, enable_neuqi=True)+ assert on.enable_neuqi is True+ sym_on = RTNConfig(bits=4, group_size=32, sym=True, disable_opt_rtn=False, enable_neuqi=True)+ assert sym_on.enable_neuqi is True # engages the two-stage symmetric search++ def test_neuqi_with_opt_rtn_disabled_raises(self):+ from auto_round.algorithms.quantization.rtn.config import RTNConfig++ with pytest.raises(ValueError, match="--enable_neuqi requires the optimized-RTN path"):+ RTNConfig(bits=4, group_size=32, sym=False, disable_opt_rtn=True, enable_neuqi=True)++ def test_default_is_off(self):+ from auto_round.algorithms.quantization.rtn.config import RTNConfig++ cfg = RTNConfig(bits=4, group_size=32, sym=False)+ assert cfg.enable_neuqi is False+++class TestNeuqiGridHelpers:+ """Shared coarse-grid helper invariants (both symmetry classes use _coarse_grid)."""++ def test_coarse_grid_values_and_freshness(self):+ """The eager coarse-grid helper must be numerically identical to the+ previous inline torch.logspace, and a cached grid is a cross-stream+ hazard in the expert fan-out (49aa6405: filled on the first caller's+ stream, read from other threads' streams) -- it must be recreated per+ call, never lru_cache'd."""+ import math++ import auto_round.data_type.neuqi as N++ for lo in (0.03125, 0.25):+ got = N._coarse_grid(lo, 64, torch.device("cpu"), torch.float32)+ want = torch.logspace(math.log10(lo), 0.0, 64, dtype=torch.float32)+ assert got.shape == want.shape+ assert torch.equal(got, want)++ args = (0.001, 8, torch.device("cpu"), torch.float32)+ a = N._coarse_grid(*args)+ b = N._coarse_grid(*args)+ assert a is not b, "grid must not be shared state across calls"+ assert not hasattr(N._coarse_grid, "cache_info"), "tensor-returning helper must not be lru_cache'd"+ torch.testing.assert_close(a, b) # values stay deterministic across calls++ def test_coarse_grid_is_dynamo_opaque(self):+ """torch.logspace lowers to an fp64 libdevice.pow inside compiled+ regions; inductor fused it into the batched sweep's index math where+ Triton rejects fp64 ("unexpected type fp64") -- see the BPT worker+ crash. The grid must be built eagerly and enter traced regions as a+ plain tensor input (no logspace op in the traced graph)."""+ import torch._dynamo as dynamo++ from auto_round.data_type.neuqi import _coarse_grid++ assert getattr(_coarse_grid, "_torchdynamo_disable", False) is True++ def traced(x):+ return _coarse_grid(0.25, 8, x.device, torch.float32) * x++ ex = dynamo.explain(traced)(torch.ones(8))+ op_names = [getattr(op, "__name__", str(op)) for ops in ex.ops_per_graph for op in ops]+ assert not any("logspace" in n for n in op_names), f"logspace leaked into the traced graph: {op_names}"+ assert any("mul" in n for n in op_names), "the sweep math itself should stay traced"+++class TestNeuqiBackendPolicy:+ """Env x device gate matrices for every backend selector (pure policy:+ no data, no kernels -- the selections themselves are covered by the+ sweep-parity classes)."""++ def test_env_device_matrix(self, monkeypatch):+ import auto_round.data_type.neuqi as N++ monkeypatch.setenv("AR_NEUQI_BACKEND", "auto")+ assert N._zp_wants_compile("cuda") is True+ assert N._zp_wants_compile("cpu") is False and N._zp_wants_compile("hpu") is False+ assert N._zp_batch_wanted("cuda") is True and N._zp_batch_wanted("cpu") is False+ assert N._zp_wants_triton("cuda") is True and N._zp_wants_triton("cpu") is False+ assert N._sym_shared_wants_triton("cuda") is True and N._sym_shared_wants_triton("cpu") is False+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")+ assert N._zp_wants_compile("cpu") is True+ assert N._sym_shared_wants_triton("cuda") is False+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")+ assert N._zp_wants_compile("cuda") is False+ assert N._zp_batch_wanted("cuda") is False # batched needs the fused backend+ assert N._zp_wants_triton("cuda") is False+ assert N._sym_shared_wants_triton("cuda") is False+ monkeypatch.setenv("AR_NEUQI_BACKEND", "triton")+ assert N._zp_wants_triton("cpu") is True # forced (availability still checked)+ assert N._sym_shared_wants_triton("cuda") is True+ # unrecognized values behave as the reference sweep+ monkeypatch.setenv("AR_NEUQI_BACKEND", "nonsense")+ assert N._zp_wants_compile("cuda") is False++ def test_layout_policy(self):+ import auto_round.data_type.neuqi as N++ # device-automatic: the CUDA Triton fusion shape on cuda, the+ # zero-point-last layout everywhere else+ assert N._zp_expr_for("cuda") is N._zp_expr_last+ assert N._zp_expr_for("cpu") is N._zp_expr_mid+ assert N._zp_expr_for("hpu") is N._zp_expr_mid++ def test_cpu_default_is_eager(self, monkeypatch):+ """AR_NEUQI_BACKEND defaults to auto: CPU tensors never trigger a compile."""+ import auto_round.data_type.neuqi as N+ from auto_round import envs++ monkeypatch.setenv("AR_NEUQI_BACKEND", "auto")+ monkeypatch.setattr(N, "_fused_zp_fns", {})+ monkeypatch.setattr(N, "_fused_zp_broken", False)+ N.neuqi_search_scale_zero(_heavy_tailed_data(n_groups=8, seed=13).clone(), bits=4, coarse_n=4, fine_n=2)+ assert not N._fused_zp_fns # no compile attempt on CPU under auto+++class TestNeuqiItersGuard:+ """--enable_neuqi with iters>0: the zero-shot search dispatch stays keyed+ to iters=0 (the tuning path resolves its plain init here); the search+ itself reaches SignRound as the wrapper's frozen initialization."""++ def test_iters_dispatch(self):+ from auto_round.data_type.utils import get_quant_func++ fn, name = get_quant_func("int", 4, False, disable_opt_rtn=False, group_size=128, iters=20, enable_neuqi=True)+ assert name == "int_asym" # tuning path initializes plain; the anchor owns the search+ fn, name = get_quant_func("int", 4, False, disable_opt_rtn=False, group_size=128, iters=0, enable_neuqi=True)+ assert name.startswith("opt_rtn_int_asym")+++class TestNeuqiSymSearch:+ """Two-stage symmetric scale search (zero point fixed at 0)."""++ def _skewed_data(self, n_groups=512, group_size=128, seed=0):+ gen = torch.Generator().manual_seed(seed)+ data = torch.randn(n_groups, group_size, generator=gen)+ data[:, :6] *= 60.0 # outliers make the clip search matter+ # half the groups skew negative so the mirrored clamp family is in play+ data[::2] *= -1.0+ return data++ def _sym_loss(self, data, qw, scale):+ """Literal loss of the shipped symmetric formula (signed scales allowed)."""+ iq = (data / scale).round().clamp(-8, 7)+ err = (scale * iq - data) ** 2+ if qw is not None:+ err = err * qw+ return err.sum(-1)++ def test_returns_signed_scales_with_magnitude_floor(self):+ from auto_round.data_type.neuqi import neuqi_search_scale_sym++ data = self._skewed_data()+ qw = torch.rand(*data.shape) + 0.05+ scale = neuqi_search_scale_sym(data, bits=4, qw=qw, coarse_n=32, fine_n=16)+ assert scale.shape == (data.shape[0], 1)+ assert (scale.abs() >= 1e-5).all()+ # both clamp conventions must actually win somewhere on skewed data+ assert (scale < 0).any() and (scale > 0).any()++ def test_mirrored_family_matches_literal_negative_scale(self):+ """The mirrored-clamp loss used inside the search must equal the literal+ negative-scale evaluation of the shipped formula (guards the sign of the+ mirrored error term)."""+ from auto_round.data_type.neuqi import _sym_loss_chunk++ torch.manual_seed(3)+ data = torch.randn(64, 96)+ qw = torch.rand(64, 96) + 0.1+ scales = torch.rand(64, 8) * 0.05 + 0.005 # [C, K] positive candidates+ loss, idx, mirror = _sym_loss_chunk(data, qw, scales, nmax=8)+ for c in range(data.shape[0]):+ k = idx[c].item()+ s = scales[c, k]+ signed = -s if mirror[c] else s+ ref = self._sym_loss(data[c : c + 1], qw[c : c + 1], torch.tensor([[signed]]))[0]+ assert torch.allclose(loss[c], ref, rtol=1e-4, atol=1e-3)++ def test_signed_search_beats_positive_only_bruteforce_on_skewed_groups(self):+ """Skewed groups: allowing the mirrored family must reach (near-)the signed+ brute-force optimum, strictly below the positive-only optimum."""+ from auto_round.data_type.neuqi import neuqi_search_scale_sym++ data = self._skewed_data(128, 96, seed=7)+ qw = torch.rand(*data.shape) + 0.05+ scale = neuqi_search_scale_sym(data, bits=4, qw=qw, coarse_n=64, fine_n=32)+ got = self._sym_loss(data, qw, scale)+ # dense positive-only reference over the same anchor+ s0 = data.abs().amax(dim=-1, keepdim=True) / 8+ fracs = torch.logspace(-0.9, 0.3, 2000)+ ref = torch.full_like(got, float("inf"))+ for f in fracs:+ l = self._sym_loss(data, qw, s0 * f)+ ref = torch.minimum(ref, l)+ # aggregate must win: the mirrored family is reachable, and the few+ # two-stage basin misses (coarse grid skips a narrow basin) stay small+ assert got.sum() <= ref.sum() * 1.001+ # and on genuinely skewed groups it must beat the positive-only optimum+ assert (got < ref - 1e-3).float().mean() > 0.25++ def test_covers_incumbent_uniform_grid(self):+ """Total weighted loss must not exceed search_scales (both searches see the+ same data, weights and signed-scale families)."""+ from auto_round.data_type.int import search_scales+ from auto_round.data_type.neuqi import neuqi_search_scale_sym++ data = self._skewed_data(512, 128, seed=11)+ qw = torch.rand(*data.shape) + 0.05+ s_neuqi = neuqi_search_scale_sym(data.clone(), bits=4, qw=qw.clone(), coarse_n=64, fine_n=32)+ s_old = search_scales(data.clone(), bits=4, qw=qw.clone())+ l_neuqi = self._sym_loss(data, qw, s_neuqi).sum().item()+ l_old = self._sym_loss(data, qw, s_old).sum().item()+ assert l_neuqi <= l_old * 1.001+++class TestNeuqiSymSharedCoarse:+ """Shared-multiplier coarse pass: gating, latch-down, and driver math."""++ _DEFAULTS = {"_sym_shared_triton": None, "_sym_shared_checked": False, "_sym_shared_broken": False}++ @pytest.fixture(autouse=True)+ def _reset_shared_state(self):+ _reset_neuqi_state(self._DEFAULTS)+ yield+ _reset_neuqi_state(self._DEFAULTS)++ def test_attempt_latches_on_failure(self, monkeypatch):+ import auto_round.data_type.neuqi as N++ _assert_launch_latches(+ monkeypatch,+ N,+ attempt="_sym_shared_triton_attempt",+ impl_attr="_sym_shared_triton",+ checked_attr="_sym_shared_checked",+ broken_attr="_sym_shared_broken",+ gates={"_sym_shared_wants_triton": True},+ tail_args=(None, None, None, 7),+ )++ def test_cpu_tensor_never_latches(self, monkeypatch):+ import auto_round.data_type.neuqi as N++ monkeypatch.delenv("AR_NEUQI_BACKEND", raising=False)+ data = torch.randn(8, 16)+ out = N._sym_shared_triton_attempt(data, None, None, None, 7)+ assert out is None+ assert not N._sym_shared_broken # gated before any launch++ def test_coarse_pass_parity_and_fallback(self, monkeypatch):+ """Driver math (dn normalization, s0^2 unnormalization, frac mapping) via an+ eager stand-in mirrors the per-candidate path; launch None falls back."""+ import auto_round.data_type.neuqi as N+ from auto_round.data_type.neuqi import _sym_loss_chunk++ torch.manual_seed(11)+ n, g, bits = 700, 128, 4+ nmax = int(2 ** (bits - 1))+ data = torch.randn(n, g)+ data[:, :3] *= 60.0+ qw = torch.rand(n, g) + 0.1+ s0 = (torch.abs(data).amax(dim=-1, keepdim=True) / nmax).clamp_(min=1e-5)+ import math++ coarse = torch.logspace(math.log10(0.25), math.log10(2.0), 96)++ def eager_shared(dn, q, fracs, invf, nm):+ # eager mirror of _sym_shared_kernel (normalized space)+ d = dn.unsqueeze(1)+ f = fracs.view(1, -1, 1)+ r = torch.round(d * invf.view(1, -1, 1))+ q_std = r.clamp(min=-nm, max=nm - 1)+ q_mir = r.clamp(min=-(nm - 1), max=nm)+ e_std = f * q_std - d+ e_mir = f * q_mir - d+ l_std = e_std * e_std+ l_mir = e_mir * e_mir+ if q is not None:+ l_std = l_std * q.unsqueeze(1)+ l_mir = l_mir * q.unsqueeze(1)+ l_std = l_std.sum(-1)+ l_mir = l_mir.sum(-1)+ mirror = l_mir < l_std+ loss = torch.where(mirror, l_mir, l_std)+ best_loss, best_idx = loss.min(dim=-1)+ return best_loss, best_idx, mirror.gather(1, best_idx.unsqueeze(1)).squeeze(1)++ # force small chunks so the loop runs multiple iterations+ monkeypatch.setattr(N, "_MAX_TMP_ELEMS", 2 * 96 * 128 * 4)+ out = N._sym_coarse_pass_shared(data, qw, s0, coarse, nmax, launch=eager_shared)+ assert out is not None+ loss_s, frac_s, mirror_s = out++ scales = s0 * coarse.view(1, -1)+ loss_r, idx_r, mirror_r = _sym_loss_chunk(data, qw, scales, nmax)+ torch.testing.assert_close(frac_s, coarse.index_select(0, idx_r), rtol=0, atol=0)+ torch.testing.assert_close(mirror_s, mirror_r, rtol=0, atol=0)+ torch.testing.assert_close(loss_s, loss_r, rtol=1e-4, atol=1e-5)++ # weighted arm agrees too+ out_w = N._sym_coarse_pass_shared(data, qw, s0, coarse, nmax, launch=eager_shared)+ assert out_w is not None++ # fallback: a launch that declines must produce None end to end+ assert N._sym_coarse_pass_shared(data, qw, s0, coarse, nmax, launch=lambda *a: None) is None++ def test_two_stage_core_uses_shared_launch(self, monkeypatch):+ """The two-stage core routes the coarse pass through the shared helper."""+ import auto_round.data_type.neuqi as N++ torch.manual_seed(12)+ data = torch.randn(64, 128)+ qw = torch.rand(64, 128) + 0.1++ def spy_launch(dn, q, fracs, invf, nm):+ spy_launch.called = True+ return None # decline -> per-candidate fallback++ spy_launch.called = False+ monkeypatch.setattr(N, "_sym_coarse_pass_shared", lambda *a, **k: (None if not spy_launch.called else None))+ # direct wiring check: the core calls _sym_coarse_pass_shared at all+ real = N._sym_coarse_pass_shared++ def wrapper(data_, qw_, s0_, coarse_, nmax_):+ wrapper.called = True+ return real(data_, qw_, s0_, coarse_, nmax_)++ wrapper.called = False+ monkeypatch.setattr(N, "_sym_coarse_pass_shared", wrapper)+ N.neuqi_search_scale_sym(data, 4, qw=qw, coarse_n=32, fine_n=8)+ assert wrapper.called+++class TestNeuqiGridEnv:+ """Grid resolution reaches both entries from the shared env pins. The+ unpinned default is backend-aware (wide only on the accelerated lanes --+ the eager sweep costs ~coarse x fine evaluations per group); env pins+ override it everywhere and invalid pins raise at the read sites."""++ @pytest.mark.parametrize("entry", ["asym", "sym"])+ def test_env_pins_and_defaults(self, monkeypatch, entry):+ import auto_round.data_type.neuqi as N++ for k in ("AR_NEUQI_COARSE", "AR_NEUQI_FINE"):+ monkeypatch.delenv(k, raising=False)+ seen = {}+ if entry == "asym":+ monkeypatch.setattr(N, "_log_search_engaged", lambda c, f: seen.update(coarse=c, fine=f))+ run = lambda: N.neuqi_search_scale_zero(torch.randn(4, 8), 4) # noqa: E731+ else:++ def spy(data, qw, bits, coarse, fine_n, q_scale_thresh):+ seen.update(coarse=coarse.numel(), fine=fine_n)+ return torch.ones(data.shape[0], 1), torch.zeros(data.shape[0])++ monkeypatch.setattr(N, "_two_stage_sym_core", spy)+ run = lambda: N.neuqi_search_scale_sym(torch.randn(4, 8), 4) # noqa: E731+ run()+ assert seen == {"coarse": 64, "fine": 32} # eager-lane default (CPU tensors)+ monkeypatch.setenv("AR_NEUQI_COARSE", "96")+ monkeypatch.setenv("AR_NEUQI_FINE", "24")+ run()+ assert seen == {"coarse": 96, "fine": 24} # pins override++ def test_default_grid_is_backend_aware(self, monkeypatch):+ import auto_round.data_type.neuqi as N++ monkeypatch.setenv("AR_NEUQI_BACKEND", "auto")+ assert N._default_grid("cuda") == (256, 64) # accelerated lanes keep the wide grid+ assert N._default_grid("cpu") == (64, 32) # eager lane narrows+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")+ assert N._default_grid("cuda") == (64, 32) # forced eager narrows even on CUDA+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")+ assert N._default_grid("cpu") == (256, 64) # forced compile keeps the wide grid++ @pytest.mark.parametrize("entry", ["asym", "sym"])+ def test_invalid_pins_raise(self, monkeypatch, entry):+ import auto_round.data_type.neuqi as N++ monkeypatch.setenv("AR_NEUQI_COARSE", "1")+ with pytest.raises(ValueError, match="too small"):+ if entry == "asym":+ N.neuqi_search_scale_zero(torch.randn(4, 8), 4)+ else:+ N.neuqi_search_scale_sym(torch.randn(4, 8), 4)+ monkeypatch.setenv("AR_NEUQI_COARSE", "8")+ monkeypatch.setenv("AR_NEUQI_FINE", "1")+ with pytest.raises(ValueError, match="too small"):+ if entry == "asym":+ N.neuqi_search_scale_zero(torch.randn(4, 8), 4)+ else:+ N.neuqi_search_scale_sym(torch.randn(4, 8), 4)+++class TestNeuqiAsymSharedCoarse:+ """Shared-multiplier coarse pass (joint search): gating, latch, driver math."""++ _DEFAULTS = {"_neuqi_shared_sweep": None, "_neuqi_shared_checked": False, "_neuqi_shared_broken": False}++ @pytest.fixture(autouse=True)+ def _reset_shared_state(self):+ _reset_neuqi_state(self._DEFAULTS)+ yield+ _reset_neuqi_state(self._DEFAULTS)++ def test_attempt_respects_backend_and_batch_gates(self, monkeypatch):+ import auto_round.data_type.neuqi as N++ fake = SimpleNamespace(is_cuda=True, device=torch.device("cuda"))+ for wants, batched, should_call in ((False, True, False), (True, False, False), (True, True, True)):+ calls = {"n": 0}++ def _fn(*a, **k):+ calls["n"] += 1+ return "sentinel"++ monkeypatch.setattr(N, "_zp_wants_triton", lambda dt: wants)+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: batched)+ monkeypatch.setattr(N, "_neuqi_shared_sweep", _fn)+ monkeypatch.setattr(N, "_neuqi_shared_checked", True)+ out = N._neuqi_shared_attempt(fake, None, None, None, 15)+ if should_call:+ assert out == "sentinel" and calls["n"] == 1+ else:+ assert out is None and calls["n"] == 0++ def test_attempt_latches_on_failure(self, monkeypatch):+ import auto_round.data_type.neuqi as N++ _assert_launch_latches(+ monkeypatch,+ N,+ attempt="_neuqi_shared_attempt",+ impl_attr="_neuqi_shared_sweep",+ checked_attr="_neuqi_shared_checked",+ broken_attr="_neuqi_shared_broken",+ gates={"_zp_wants_triton": True, "_zp_batch_wanted": True},+ tail_args=(None, None, None, 15),+ )++ def test_cpu_tensor_never_latches(self):+ import auto_round.data_type.neuqi as N++ data = torch.randn(8, 16)+ out = N._zp_coarse_pass_shared(data, None, torch.full((8, 1), 0.1), torch.tensor([0.5, 1.0]), 15)+ assert out is None+ assert not N._neuqi_shared_broken # gated before any launch++ def test_coarse_pass_parity_and_fallback(self, monkeypatch):+ """Driver math (dn normalization, s0^2 unnormalization, frac/zp mapping) via+ an eager stand-in mirrors the per-candidate reference; launch None falls back."""+ import auto_round.data_type.neuqi as N++ torch.manual_seed(13)+ n, g, bits = 600, 128, 4+ maxq = 2**bits - 1+ data = torch.randn(n, g)+ data[:, :3] *= 70.0+ qw = torch.rand(n, g) + 0.1+ wmin = torch.clamp(data.min(dim=-1).values, max=0).unsqueeze(-1)+ wmax = torch.clamp(data.max(dim=-1).values, min=0).unsqueeze(-1)+ s0 = ((wmax - wmin) / maxq).clamp_(min=1e-5)+ import math++ coarse = torch.logspace(math.log10(0.25), 0.0, 80)++ def eager_shared(dn, q, fracs, invf, mq):+ # eager mirror of _neuqi_shared_kernel (normalized space)+ d = dn.unsqueeze(-2).unsqueeze(-2) # [C, 1, 1, g]+ f = fracs.view(1, -1, 1, 1) # [1, K, 1, 1]+ zp = torch.arange(0, mq + 1, device=dn.device, dtype=dn.dtype).view(1, 1, -1, 1)+ r = torch.round(d * invf.view(1, -1, 1, 1)).clamp_(min=-mq - 1, max=2 * mq + 1)+ qd = (r + zp).clamp_(0, mq)+ err = f * (qd - zp) - d+ loss = err * err+ if q is not None:+ loss = loss * q.unsqueeze(-2).unsqueeze(-2)+ loss = loss.sum(dim=-1) # [C, K, Z]+ best_z, zp_arg = loss.min(dim=-1)+ best, k_arg = best_z.min(dim=1) # first-minimum over k+ zp_best = zp_arg.gather(1, k_arg.unsqueeze(1)).squeeze(1)+ return best, k_arg, zp_best++ # force small chunks so the loop runs multiple iterations+ monkeypatch.setattr(N, "_MAX_TMP_ELEMS", 262144 * 3)+ loss_s, frac_s, zp_s = N._zp_coarse_pass_shared(data, qw, s0, coarse, maxq, launch=eager_shared)++ # per-candidate reference over the identical grid+ scales = s0 * coarse.view(1, -1) # [C, K]+ d = data.unsqueeze(-2).unsqueeze(-2)+ zp = torch.arange(0, maxq + 1).view(1, 1, -1, 1)+ r = torch.round(d / scales.unsqueeze(-1).unsqueeze(-1)).clamp_(min=-maxq - 1, max=2 * maxq + 1)+ qd = (r + zp).clamp_(0, maxq)+ err = scales.unsqueeze(-1).unsqueeze(-1) * (qd - zp) - d+ loss = err * err * qw.unsqueeze(-2).unsqueeze(-2)+ loss = loss.sum(dim=-1) # [C, K, Z]+ best_z, zp_arg = loss.min(dim=-1)+ loss_ref, k_ref = best_z.min(dim=1)+ zp_ref = zp_arg.gather(1, k_ref.unsqueeze(1)).squeeze(1)++ torch.testing.assert_close(frac_s, coarse.index_select(0, k_ref), rtol=0, atol=0)+ torch.testing.assert_close(zp_s, zp_ref.to(torch.float32), rtol=0, atol=0)+ torch.testing.assert_close(loss_s, loss_ref, rtol=1e-4, atol=1e-4)++ # fallback: a launch that declines must produce None end to end+ assert N._zp_coarse_pass_shared(data, qw, s0, coarse, maxq, launch=lambda *a: None) is None++ def test_search_wires_shared_coarse(self, monkeypatch):+ """neuqi_search_scale_zero routes its coarse pass through the shared helper."""+ import auto_round.data_type.neuqi as N++ torch.manual_seed(14)+ data = torch.randn(64, 128)+ qw = torch.rand(64, 128) + 0.1++ def wrapper(data_, qw_, s0_, coarse_, maxq_):+ wrapper.called = True+ return None # decline -> batched fallback still correct++ wrapper.called = False+ monkeypatch.setattr(N, "_zp_coarse_pass_shared", wrapper)+ N.neuqi_search_scale_zero(data, 4, qw=qw, coarse_n=32, fine_n=8)+ assert wrapper.called+++class TestNeuqiSymTritonSweep:+ """Triton sym-search dispatch: gating, latch-down, and parity where CUDA exists."""++ @pytest.fixture(autouse=True)+ def _reset_backend_state(self):+ import auto_round.data_type.neuqi as N++ for attr in ("_sym_triton", "_sym_triton_checked", "_sym_triton_broken"):+ if hasattr(N, attr):+ setattr(N, attr, None if attr != "_sym_triton_broken" else False)+ N._sym_triton_checked = False+ N._sym_search_warmed.clear()+ N._sym_chunk_compiled = None+ N._sym_compile_failed = False+ yield+ import auto_round.data_type.neuqi as N2++ N2._sym_triton_broken = False+ N2._sym_triton_checked = False+ N2._sym_triton = None++ def test_eval_on_cpu_matches_reference(self):+ from auto_round.data_type.neuqi import _sym_loss_chunk, _sym_search_eval++ torch.manual_seed(3)+ data = torch.randn(96, 32)+ scales = torch.rand(96, 7) + 0.5+ qw = torch.rand(96, 32) + 0.1+ for q in (qw, None):+ ref = _sym_loss_chunk(data, q, scales, 7)+ out = _sym_search_eval(data, q, scales, 7)+ torch.testing.assert_close(out[0], ref[0], rtol=1e-5, atol=1e-6)+ mism = (out[1] != ref[1]).float().mean().item()+ assert mism < 5e-3, f"candidate index mismatch fraction {mism}"+ mism_m = (out[2] != ref[2]).float().mean().item()+ assert mism_m < 5e-2, f"convention mismatch fraction {mism_m}"++ def test_attempt_skips_and_latches_on_failure(self, monkeypatch):+ import auto_round.data_type.neuqi as N++ # a CUDA-shaped stand-in: the gate only consults is_cuda/device before+ # the launch attempt (real-CUDA parity is covered by the skipif test)+ _assert_launch_latches(+ monkeypatch,+ N,+ attempt="_sym_triton_attempt",+ impl_attr="_sym_triton",+ checked_attr="_sym_triton_checked",+ broken_attr="_sym_triton_broken",+ gates={"_sym_wants_triton": True},+ tail_args=(None, None, 7),+ )++ def test_cpu_never_reaches_triton(self, monkeypatch):+ import auto_round.data_type.neuqi as N++ def _boom(*a, **k):+ raise AssertionError("CPU tensors must never reach the Triton kernel")++ monkeypatch.setattr(N, "_sym_triton", _boom)+ monkeypatch.setattr(N, "_sym_triton_checked", True)+ out = N._sym_search_eval(torch.randn(8, 32), None, torch.rand(8, 2) + 0.5, 7)+ assert out[0].shape == (8,)++ def test_wrapper_rejects_cpu_tensors(self):+ pytest.importorskip("auto_round_extension.triton.neuqi_sweep")+ from auto_round_extension.triton.neuqi_sweep import sym_search_triton++ with pytest.raises(AssertionError):+ sym_search_triton(torch.randn(4, 32), None, torch.rand(4, 2) + 0.5, 7)++ @pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA for the Triton kernel")+ def test_triton_matches_reference_on_cuda(self):+ from auto_round.data_type.neuqi import _sym_loss_chunk+ from auto_round_extension.triton.neuqi_sweep import sym_search_triton++ torch.manual_seed(4)+ dev = torch.device("cuda")+ data = torch.randn(4096, 128, device=dev)+ data[:, :5] *= 120.0 # outlier columns like real weights+ scales = torch.rand(4096, 33, device=dev) + 0.25+ qw = torch.rand(4096, 128, device=dev) + 0.1+ for q in (qw, None):+ ref = _sym_loss_chunk(data, q, scales, 7)+ out = sym_search_triton(data.contiguous(), q.contiguous() if q is not None else None, scales, 7)+ torch.testing.assert_close(out[0], ref[0], rtol=1e-4, atol=1e-5)+ mism = (out[1] != ref[1]).float().mean().item()+ assert mism < 1e-2, f"candidate index mismatch fraction {mism}"+ mism_m = (out[2] != ref[2]).float().mean().item()+ assert mism_m < 5e-2, f"convention mismatch fraction {mism_m}"+++class TestNeuqiSymCompiledCore:+ """The compiled sym chunk core: parity with eager, backend gating, latch-down."""++ @pytest.fixture(autouse=True)+ def _reset_core_state(self):+ import auto_round.data_type.neuqi as N++ N._sym_chunk_compiled = None+ N._sym_compile_failed = False+ N._sym_compile_logged.clear()+ yield+ N._sym_chunk_compiled = None+ N._sym_compile_failed = False++ def test_compiled_matches_eager(self, monkeypatch):+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")+ torch.manual_seed(0)+ from auto_round.data_type.neuqi import _sym_loss_chunk, _sym_loss_chunk_eval++ data = torch.randn(64, 32)+ scales = torch.rand(64, 5) + 0.5+ for qw in (torch.rand(64, 32) + 0.1, None):+ out_e = _sym_loss_chunk(data, qw, scales, 7)+ out_c = _sym_loss_chunk_eval(data, qw, scales, 7)+ torch.testing.assert_close(out_c[0], out_e[0], rtol=1e-5, atol=1e-6)+ assert torch.equal(out_c[1], out_e[1])+ assert torch.equal(out_c[2], out_e[2])+ # where the toolchain allows inductor codegen the compiled core must+ # have ENGAGED (not silently latched down to eager); environments+ # without a C++ compiler (e.g. Windows without MSVC) latch by design+ import auto_round.data_type.neuqi as N++ if not N._sym_compile_failed:+ assert N._sym_chunk_compiled is not None++ def test_eager_backend_never_compiles(self, monkeypatch):+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")+ from auto_round.data_type import neuqi++ def _boom(*a, **k):+ raise AssertionError("torch.compile must not be called under backend=eager")++ monkeypatch.setattr(neuqi.torch, "compile", _boom)+ out = neuqi._sym_loss_chunk_eval(torch.randn(8, 32), None, torch.rand(8, 2) + 0.5, 7)+ assert out[0].shape == (8,)++ def test_failure_latches_down_to_eager(self, monkeypatch):+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")+ from auto_round.data_type import neuqi++ calls = {"n": 0}++ def _flaky_compile(fn):+ def _raise(*a, **k):+ calls["n"] += 1+ raise RuntimeError("inductor unavailable")++ return _raise++ monkeypatch.setattr(neuqi.torch, "compile", _flaky_compile)+ data, scales = torch.randn(8, 32), torch.rand(8, 2) + 0.5+ for _ in range(2): # second call must go straight to eager+ out = neuqi._sym_loss_chunk_eval(data, None, scales, 7)+ assert torch.isfinite(out[0]).all()+ assert calls["n"] == 1+ assert neuqi._sym_compile_failed++ def test_search_end_to_end_matches_eager_backend(self, monkeypatch):+ """The compiled core routes every chunk through; results must match an+ eager-backend run."""+ from auto_round.data_type.neuqi import neuqi_search_scale_sym++ torch.manual_seed(1)+ data = torch.randn(128, 32)++ def _run():+ return neuqi_search_scale_sym(data.clone(), 4, coarse_n=8, fine_n=4)++ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")+ eager_plain = _run()+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")+ comp_plain = _run()+ torch.testing.assert_close(comp_plain, eager_plain, rtol=0, atol=0)+++class TestNeuqiSymDynamoGuard:+ def test_sym_search_never_traced_by_dynamo(self):+ """The sym search is loop-heavy and eager-shaped; under+ --enable_torch_compile it must be skipped by dynamo (tracing it unrolls+ the chunk loops into giant graphs - minutes of CPU compile, GPU idle)."""+ from auto_round.data_type.neuqi import _two_stage_sym_core, neuqi_search_scale_sym++ for fn in (neuqi_search_scale_sym, _two_stage_sym_core):+ assert getattr(fn, "_torchdynamo_disable", False), fn.__name__+++class TestNeuqiSymIntegration:+ def test_sym_neuqi_dispatch(self):+ fn, name = get_quant_func("int", 4, True, disable_opt_rtn=False, group_size=32, iters=0, enable_neuqi=True)+ assert name == "opt_rtn_int_sym_neuqi"+ assert "opt_rtn_int_sym_neuqi" in QUANT_FUNC_WITH_DTYPE++ def test_wrapper_qdq_matches_returned_signed_scale(self):+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_sym_neuqi++ torch.manual_seed(13)+ w = torch.randn(32, 256)+ w[:, :20] *= -40.0 # negative-heavy columns exercise the mirrored family+ qdq, scale, zp = quant_tensor_opt_rtn_sym_neuqi(w, bits=4, group_size=32)+ assert zp == 8+ assert scale.shape == (32 * 256 // 32, 1)+ s2d = torch.repeat_interleave(scale.reshape(32, 8), 32, dim=1)+ manual = (w / s2d).round().clamp(-8, 7) * s2d+ assert torch.allclose(qdq, manual, rtol=1e-3, atol=1e-4)+ assert (scale < 0).any() # mirrored family won somewhere++ def test_sym_wrapper_supports_imatrix_off_1d_2d(self):+ """qw=None (imatrix off), per-column 1-D and per-row 2-D weights all flow;+ weighting must steer the search (lower weighted loss than the unweighted+ solution under the same weights)."""+ from auto_round.data_type.neuqi import neuqi_search_scale_sym, quant_tensor_opt_rtn_sym_neuqi++ torch.manual_seed(17)+ w = torch.randn(32, 256)+ w[:, :24] *= -40.0+ for im in (None, torch.rand(256) + 0.05, torch.rand(32, 256) + 0.05):+ qdq, scale, _ = quant_tensor_opt_rtn_sym_neuqi(w.clone(), bits=4, group_size=32, imatrix=im)+ assert torch.isfinite(qdq).all()++ d = w.reshape(-1, 32)+ qw = (torch.rand(32, 256) + 0.05).reshape(-1, 32)+ s_q = neuqi_search_scale_sym(d.clone(), bits=4, qw=qw.clone())+ s_p = neuqi_search_scale_sym(d.clone(), bits=4, qw=None)++ def wloss(s):+ iq = (d / s).round().clamp(-8, 7)+ return ((s * iq - d) ** 2 * qw).sum().item()++ assert wloss(s_q) < wloss(s_p)+++class _FakeQuantLayer(torch.nn.Module):+ """Minimal stand-in for a quantisable layer (see also hadamard test_patch)."""++ def __init__(self, out_features=8, in_features=32, bits=4, sym=False, group_size=32):+ super().__init__()+ self.bits = bits+ self.sym = sym+ self.group_size = group_size+ self.data_type = "int"+ self.act_bits = 16+ self.act_data_type = "int"+ self.act_sym = True+ self.act_dynamic = True+ self.iters = 200+ self.tuning_device = "cpu"+ torch.manual_seed(7)+ self.weight = torch.nn.Parameter(torch.randn(out_features, in_features))+ self.bias = None+ self.imatrix = None+++class TestNeuqiFrozenInit:+ """enable_neuqi on the tuning path: the search anchors the STE grid+ once at wrapper init and the range margins stay pinned at 1.0."""++ def _wrapper(self, enable_neuqi, sym=False, enable_round_tuning=True):+ from auto_round.wrapper import WrapperLinear++ layer = _FakeQuantLayer(sym=sym)+ wrapper = WrapperLinear(+ layer,+ device="cpu",+ enable_minmax_tuning=True,+ enable_norm_bias_tuning=False,+ enable_round_tuning=enable_round_tuning,+ enable_torch_compile=False,+ disable_opt_rtn=True,+ enable_neuqi=enable_neuqi,+ iters=5,+ )+ return wrapper, layer++ def test_neuqi_anchor_replaces_minmax_asym(self):+ from auto_round.data_type.neuqi import neuqi_search_scale_zero++ wrapper, layer = self._wrapper(True, sym=False)+ w = layer.weight.data.to(torch.float32).reshape(-1, layer.group_size)+ s, zp = neuqi_search_scale_zero(w, layer.bits)+ maxq = int(2**layer.bits) - 1+ assert torch.allclose(wrapper.weight_min, -(zp * s).squeeze(-1), atol=1e-6)+ assert torch.allclose(wrapper.weight_max, ((maxq - zp) * s).squeeze(-1), atol=1e-6)+ # frozen margins: the searched grid IS the init; SignRound tunes values only+ assert wrapper._neuqi_frozen_margins is True+ # non-tunable params become plain tensors, not entries in params+ assert "min_scale" not in wrapper.params and "max_scale" not in wrapper.params++ def test_sym_anchor_uses_two_stage_search(self):+ from auto_round.data_type.int import quant_tensor_sym+ from auto_round.data_type.neuqi import neuqi_search_scale_sym, quant_tensor_opt_rtn_sym_neuqi++ wrapper, layer = self._wrapper(True, sym=True)+ w = layer.weight.data.to(torch.float32).reshape(-1, layer.group_size)+ s_signed = neuqi_search_scale_sym(w, layer.bits)+ nmax = int(2 ** (layer.bits - 1))+ mag = (s_signed.abs() * nmax).squeeze(-1)+ mirrored = (s_signed < 0).squeeze(-1)+ # the STE tie-break maps wmax_abs < wmin_abs to the POSITIVE scale:+ # the losing side is stored at HALF magnitude so the comparison+ # resolves to the winner's convention in any storage dtype+ assert torch.allclose(wrapper.weight_min, torch.where(mirrored, -mag / 2, -mag))+ assert torch.allclose(wrapper.weight_max, torch.where(mirrored, mag, mag / 2))+ assert wrapper._neuqi_frozen_margins is True+ # R1-1/R2-1 regression guard: the anchored grid must reproduce the+ # search's winning qdq exactly, per group, under BOTH clamp conventions+ ref_qdq, _, _ = quant_tensor_opt_rtn_sym_neuqi(w, bits=layer.bits, group_size=layer.group_size)+ ste_qdq, ste_scale, _ = quant_tensor_sym(+ w,+ bits=layer.bits,+ group_size=layer.group_size,+ min_scale=torch.tensor(1.0),+ max_scale=torch.tensor(1.0),+ tensor_min=wrapper.weight_min,+ tensor_max=wrapper.weight_max,+ )+ assert torch.equal(ste_qdq.reshape(ref_qdq.shape), ref_qdq) or torch.allclose(+ ste_qdq.reshape(ref_qdq.shape), ref_qdq, atol=1e-7+ ), "anchored STE grid diverges from the search winner"++ def test_sym_anchor_survives_low_precision_storage(self, monkeypatch):+ # R2-1: the anchor is cast to the weight dtype; a sub-precision bias+ # (2^-20) would be erased by bf16/fp16 rounding and flip every+ # standard-convention group to the mirrored grid. The half-magnitude+ # bias must reproduce the search qdq for bf16 and fp16 weights too.+ import auto_round.wrapper as wrapper_mod+ from auto_round.data_type.int import quant_tensor_sym+ from auto_round.data_type.neuqi import neuqi_search_scale_sym, quant_tensor_opt_rtn_sym_neuqi++ for dtype in (torch.bfloat16, torch.float16):+ layer = _FakeQuantLayer(sym=True)+ layer.weight.data = layer.weight.data.to(torch.float32).to(dtype)+ wrapper = wrapper_mod.WrapperLinear(+ layer,+ device="cpu",+ enable_minmax_tuning=True,+ enable_round_tuning=True,+ enable_torch_compile=False,+ disable_opt_rtn=True,+ enable_neuqi=True,+ iters=5,+ )+ assert wrapper.weight_min.dtype == dtype+ w32 = layer.weight.data.to(torch.float32).reshape(-1, layer.group_size)+ s_signed = neuqi_search_scale_sym(w32, layer.bits)+ ref_qdq, _, _ = quant_tensor_opt_rtn_sym_neuqi(w32, bits=layer.bits, group_size=layer.group_size)+ ste_qdq, ste_scale, _ = quant_tensor_sym(+ layer.weight.data,+ bits=layer.bits,+ group_size=layer.group_size,+ min_scale=torch.tensor(1.0),+ max_scale=torch.tensor(1.0),+ tensor_min=wrapper.weight_min,+ tensor_max=wrapper.weight_max,+ )+ ste_qdq = ste_qdq.to(torch.float32).reshape(ref_qdq.shape)+ # (1) the R2-1 invariant: no group's clamp convention flips (the+ # pre-fix failure was a systematic all-group sign flip)+ ste_sign = torch.sign(ste_scale.reshape(-1))+ want_sign = torch.sign(s_signed.reshape(-1))+ assert torch.equal(+ ste_sign, want_sign+ ), f"{dtype}: convention flipped on {(ste_sign != want_sign).sum()} groups"+ # (2) residual diffs are bf16/fp16 boundary-rounding jitter of the+ # STE division on low-precision weights, bounded by one quantum+ diff = (ste_qdq - ref_qdq.to(torch.float32)).abs()+ quantum = s_signed.abs().reshape(-1, 1)+ assert (diff <= quantum + 1e-6).all(), f"{dtype}: diff {(diff.max()):.3e} exceeds one quantum"++ def test_zeroshot_asym_uses_joint_search(self):+ """Zero-shot optimized path, asymmetric: the wrapper resolves the+ joint (scale, zp) search quant func, and its qdq equals the direct+ search result exactly (the search IS the quant function here)."""+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_asym+ from auto_round.wrapper import WrapperLinear++ layer = _FakeQuantLayer(sym=False)+ layer.iters = 0+ wrapper = WrapperLinear(+ layer,+ device="cpu",+ enable_minmax_tuning=False,+ enable_norm_bias_tuning=False,+ enable_round_tuning=False,+ enable_torch_compile=False,+ disable_opt_rtn=False,+ enable_neuqi=True,+ iters=0,+ )+ assert wrapper.data_type == "opt_rtn_int_asym"+ w = layer.weight.data.to(torch.float32)+ ref_qdq, _, _ = quant_tensor_opt_rtn_asym(w, bits=layer.bits, group_size=layer.group_size)+ got_qdq, _, _ = wrapper.weight_quant_func(layer.weight.data)+ assert torch.allclose(+ got_qdq.to(torch.float32).reshape(ref_qdq.shape), ref_qdq.to(torch.float32), atol=1e-7+ ), "zero-shot asym qdq diverges from the joint search result"++ # fp16 weights: the division runs in the weight dtype (mirroring the+ # search function), so parity must hold there too+ layer_fp16 = _FakeQuantLayer(sym=False)+ layer_fp16.iters = 0+ layer_fp16.weight.data = layer_fp16.weight.data.to(torch.float16)+ wrapper_fp16 = WrapperLinear(+ layer_fp16,+ device="cpu",+ enable_minmax_tuning=False,+ enable_norm_bias_tuning=False,+ enable_round_tuning=False,+ enable_torch_compile=False,+ disable_opt_rtn=False,+ enable_neuqi=True,+ iters=0,+ )+ ref16, _, _ = quant_tensor_opt_rtn_asym(+ layer_fp16.weight.data, bits=layer_fp16.bits, group_size=layer_fp16.group_size+ )+ got16, _, _ = wrapper_fp16.weight_quant_func(layer_fp16.weight.data)+ assert torch.equal(got16, ref16), "fp16 zero-shot asym qdq diverges from the joint search result"++ def test_zeroshot_sym_uses_neuqi_search(self):+ """Zero-shot optimized path, symmetric: the wrapper resolves the+ two-stage NeUQI scale search (not the uniform search_scales grid),+ and its qdq equals quant_tensor_opt_rtn_sym_neuqi's result."""+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_sym_neuqi+ from auto_round.wrapper import WrapperLinear++ layer = _FakeQuantLayer(sym=True)+ layer.iters = 0+ wrapper = WrapperLinear(+ layer,+ device="cpu",+ enable_minmax_tuning=False,+ enable_norm_bias_tuning=False,+ enable_round_tuning=False,+ enable_torch_compile=False,+ disable_opt_rtn=False,+ enable_neuqi=True,+ iters=0,+ )+ assert wrapper.data_type == "opt_rtn_int_sym_neuqi"+ w = layer.weight.data.to(torch.float32)+ ref_qdq, ref_scale, _ = quant_tensor_opt_rtn_sym_neuqi(w, bits=layer.bits, group_size=layer.group_size)+ got_qdq, _, _ = wrapper.weight_quant_func(layer.weight.data)+ assert torch.allclose(+ got_qdq.to(torch.float32).reshape(ref_qdq.shape), ref_qdq.to(torch.float32), atol=1e-7+ ), "zero-shot sym qdq diverges from the two-stage NeUQI search"+ assert ref_scale is not None+ assert "min_scale" not in wrapper.params and "max_scale" not in wrapper.params++ def test_anchor_stays_off_when_inappropriate(self):+ """Flag off: plain min/max with tunable margins. Zero-shot path: the+ search runs inside the quant-func dispatch, so the wrapper-side anchor+ must not fire there either."""+ wrapper, layer = self._wrapper(False, sym=False)+ w = layer.weight.data.reshape(-1, layer.group_size)+ assert torch.allclose(wrapper.weight_min, w.min(1)[0].clamp(max=0), atol=1e-7)+ assert torch.allclose(wrapper.weight_max, w.max(1)[0].clamp(min=0), atol=1e-7)+ assert wrapper._neuqi_frozen_margins is False+ assert wrapper.params["min_scale"].requires_grad is True+ assert wrapper.params["max_scale"].requires_grad is True++ wrapper, _ = self._wrapper(True, enable_round_tuning=False)+ assert wrapper.weight_min is None and wrapper.weight_max is None+ assert wrapper._neuqi_frozen_margins is False+++class TestSignRoundEnableNeuqi:+ """SignRoundConfig carries enable_neuqi; True = the frozen-init anchor."""++ def test_config_flag(self):+ from auto_round.algorithms.quantization.sign_round.config import SignRoundConfig++ assert SignRoundConfig(bits=4, group_size=32).enable_neuqi is False+ assert SignRoundConfig(bits=4, group_size=32, sym=False, enable_neuqi=True).enable_neuqi is True+++class TestExpertBatching:+ """Same-shape expert projections quantize in one stacked search call+ (row-independent: results identical to per-module calls)."""++ @pytest.fixture(autouse=True)+ def _tie_free_grid(self, monkeypatch):+ # exact stacked-vs-per-module equality holds while the grid produces no+ # fp-summation near-ties for the seeded data; the narrow eager default+ # (64/32) hits ties that first-min resolves differently between the+ # batched expression and the per-candidate expression (the sweep parity+ # tests cover the tie semantics) -- pin the wide grid this exact-equality+ # contract was verified against+ monkeypatch.setenv("AR_NEUQI_COARSE", "256")+ monkeypatch.setenv("AR_NEUQI_FINE", "64")++ def teardown_method(self):+ import auto_round.algorithms.quantization.rtn.quantizer as _qmod++ if "model_context" in _qmod.OptimizedRTNQuantizer.__dict__:+ del _qmod.OptimizedRTNQuantizer.model_context++ def _expert(self, name, out=16, inn=32, bits=4, group_size=32, sym=False, seed=0):+ m = torch.nn.Linear(inn, out, bias=False)+ torch.manual_seed(seed)+ with torch.no_grad():+ m.weight.copy_(torch.randn(out, inn))+ m.global_name = name+ m.bits = bits+ m.group_size = group_size+ m.sym = sym+ m.data_type = "int"+ return m++ def _quantizer(self, enable_neuqi=True, disable_opt_rtn=False, is_moe=True):+ from auto_round.algorithms.quantization.rtn.config import OptimizedRTNConfig+ from auto_round.algorithms.quantization.rtn.quantizer import OptimizedRTNQuantizer++ cfg = OptimizedRTNConfig(bits=4, group_size=32, sym=False, enable_neuqi=enable_neuqi)+ cfg.disable_opt_rtn = disable_opt_rtn+ cfg.orig_disable_opt_rtn = False if not disable_opt_rtn else True+ q = OptimizedRTNQuantizer(cfg)+ if is_moe:+ # model_context is a read-only property backed by the run ctx;+ # patch the class property for the MoE-heuristic branch+ import auto_round.algorithms.quantization.rtn.quantizer as _qmod++ _qmod.OptimizedRTNQuantizer.model_context = property(lambda self: type("MC", (), {"is_moe_model": True})())+ return q++ def test_sym_and_asym_batched_and_equivalent(self):+ # THE sym gap fix: sym modules join the batches under neuqi, and the+ # stacked call matches per-module search outputs exactly+ q = self._quantizer()+ mods = [self._expert(f"blk.experts.{i}.gate_proj", sym=False, seed=i) for i in range(4)]+ sym_mods = [self._expert(f"blk.experts.{i}.up_proj", sym=True, seed=100 + i) for i in range(4)]+ batches, singles = q._split_expert_batches(mods + sym_mods)+ assert len(batches) == 2 and not singles+ assert {m.sym for m in batches[0]} == {False} and {m.sym for m in batches[1]} == {True}++ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_asym, quant_tensor_opt_rtn_sym_neuqi++ assert q._quantize_expert_batch(batches[0], "cpu") == []+ for m in mods:+ ref_qdq, ref_scale, ref_zp = quant_tensor_opt_rtn_asym(+ m.weight.data, bits=m.bits, group_size=m.group_size, v=0.0, q_scale_thresh=1e-5, imatrix=None+ )+ assert torch.equal(m.weight.data, ref_qdq)+ assert torch.equal(m.scale.reshape(-1), ref_scale.reshape(-1))+ assert torch.equal(m.zp.reshape(-1), ref_zp.reshape(-1))+ assert m.q_scale_thresh == 1e-5++ assert q._quantize_expert_batch(batches[1], "cpu") == []+ for m in sym_mods:+ ref_qdq, ref_scale, ref_zp = quant_tensor_opt_rtn_sym_neuqi(+ m.weight.data, bits=m.bits, group_size=m.group_size, v=0.0, q_scale_thresh=1e-5, imatrix=None+ )+ assert torch.equal(m.weight.data, ref_qdq)+ assert torch.equal(m.scale.reshape(-1), ref_scale.reshape(-1))+ assert m.zp == ref_zp+ # R2-5: the batch writeback must stamp the resolved data_type both classes+ assert all(m.data_type == "opt_rtn_int_asym" for m in mods)+ assert all(m.data_type == "opt_rtn_int_sym_neuqi" for m in sym_mods)++ def test_imatrix_weighting_stacked_equivalence(self):+ q = self._quantizer()+ mods = [self._expert(f"blk.experts.{i}.down_proj", sym=False, seed=7 + i) for i in range(3)]+ for i, m in enumerate(mods):+ torch.manual_seed(50 + i)+ m.imatrix = torch.rand(m.weight.shape[1])+ batches, singles = q._split_expert_batches(mods)+ assert len(batches) == 1 and not singles+ assert q._quantize_expert_batch(batches[0], "cpu") == []+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_asym++ for m in mods:+ ref_qdq, ref_scale, ref_zp = quant_tensor_opt_rtn_asym(+ m.weight.data,+ bits=m.bits,+ group_size=m.group_size,+ v=0.0,+ q_scale_thresh=1e-5,+ imatrix=m.imatrix,+ )+ assert torch.equal(m.weight.data, ref_qdq)+ assert torch.equal(m.scale.reshape(-1), ref_scale.reshape(-1))++ def test_batch_splitting_rules(self):+ """One setup for every _split_expert_batches rule: gating, expert-name+ filtering, shape/act-quant/symmetry segregation."""+ q = self._quantizer()++ # gating: off -> inactive; disable_opt_rtn -> inactive+ assert self._quantizer(enable_neuqi=False)._expert_search_active() is False+ assert self._quantizer(enable_neuqi=True, disable_opt_rtn=True)._expert_search_active() is False++ # non-expert and odd-shaped modules stay single+ dense = self._expert("blk.mlp.gate_proj")+ odd = self._expert("blk.experts.0.gate_proj", inn=48, group_size=32) # in % g != 0 is fine for grouping+ mixed_shape = self._expert("blk.experts.1.gate_proj", inn=64)+ batches, singles = q._split_expert_batches([dense, odd, mixed_shape])+ assert not batches and {dense, odd, mixed_shape} == set(singles)++ # act quantization needs the per-module wrapper path+ m = self._expert("blk.experts.0.gate_proj")+ m.act_bits = 8+ batches, singles = q._split_expert_batches([m])+ assert not batches and singles == [m]++ # same (parent, proj, shape, bits, gs) but different symmetry: not+ # mergeable into one stacked call -> both stay single (group of 1)+ a = self._expert("blk.experts.0.gate_proj", sym=False)+ sym_m = self._expert("blk.experts.1.gate_proj", sym=True)+ batches, singles = q._split_expert_batches([a, sym_m])+ assert not batches and len(singles) == 2++ def test_oom_returns_only_unwritten(self, monkeypatch):+ # R2-2: a mid-batch OOM must leave already-written modules untouched+ # and return ONLY the unwritten remainder (per-module fallback must+ # never re-quantize a qdq'd weight)+ import auto_round.algorithms.quantization.rtn.quantizer as Q+ import auto_round.data_type.neuqi as N++ q = self._quantizer()+ mods = [self._expert(f"blk.experts.{i}.gate_proj", seed=i) for i in range(3)]+ batches, singles = q._split_expert_batches(mods)+ assert len(batches) == 1++ calls = []+ orig = N.quant_tensor_opt_rtn_asym++ def oom_after_first(weights, *a, **kw):+ calls.append(1)+ if len(calls) > 1:+ raise torch.OutOfMemoryError("simulated")+ return orig(weights, *a, **kw)++ monkeypatch.setattr(N, "quant_tensor_opt_rtn_asym", oom_after_first)+ # one module per stacked call -> the failure hits mid-batch+ monkeypatch.setattr(Q, "_EXPERT_BATCH_MAX_ELEMS", mods[0].weight.numel())+ before = mods[0].weight.data.clone()+ remaining = q._quantize_expert_batch(batches[0], "cpu")+ assert len(remaining) == 2, "modules after the failed chunk must be returned unwritten"+ assert remaining == mods[1:]+ # the first chunk was written and must NOT be re-quantized later+ assert not torch.equal(mods[0].weight.data, before)+ assert hasattr(mods[0], "scale") and mods[0].data_type == "opt_rtn_int_asym"+ # unwritten modules are untouched+ assert not hasattr(mods[1], "scale") and not hasattr(mods[2], "scale")++ def test_act_quant_modules_stay_single(self):+ q = self._quantizer()+ m = self._expert("blk.experts.0.gate_proj")+ m.act_bits = 8 # act quantization needs the per-module wrapper path+ batches, singles = q._split_expert_batches([m])+ assert not batches and singles == [m]++ def test_sym_key_separates_batches(self):+ q = self._quantizer()+ a = self._expert("blk.experts.0.gate_proj", sym=False)+ s = self._expert("blk.experts.1.gate_proj", sym=True)+ batches, singles = q._split_expert_batches([a, s])+ # same (parent, proj, shape, bits, gs) but different symmetry: not+ # mergeable into one stacked call -> both stay single (group of 1)+ assert not batches and len(singles) == 2+++class TestImatrixAutoRule:+ """asym + neuqi must auto-collect the imatrix (the search consumes qw);+ plain asym keeps it off (min/max init ignores it)."""++ def test_rule(self):+ from auto_round.algorithms.quantization.rtn.config import OptimizedRTNConfig, RTNConfig+ from auto_round.autoround import _select_rtn_compressor_base_cls++ cfg = RTNConfig(bits=4, group_size=128, sym=False, enable_neuqi=True)+ _select_rtn_compressor_base_cls(cfg, scheme=None, format="auto_round", base_kwargs={})+ assert cfg.enable_imatrix is True and isinstance(cfg, OptimizedRTNConfig)++ cfg = RTNConfig(bits=4, group_size=128, sym=False, enable_neuqi=False)+ _select_rtn_compressor_base_cls(cfg, scheme=None, format="auto_round", base_kwargs={})+ assert cfg.enable_imatrix is False+++class TestEnableNeuqiCliFlag:+ """The CLI gate is --enable_neuqi (per the governing PR design); the+ boolean enable_neuqi kwarg is the API surface."""++ def test_cli_flag_registration(self):+ from auto_round.algorithms.quantization.rtn.config import RTNConfig+ from auto_round.algorithms.quantization.sign_round.config import SignRoundConfig++ reg = RTNConfig.get_registered_args()+ p = next(p for p in reg.parameters if "--enable_neuqi" in p.option_strings)+ assert p.field == "enable_neuqi"+ assert p.argparse_kwargs["action"] == "store_true"+ assert any("--enable_neuqi" in q.option_strings for q in SignRoundConfig.get_registered_args().parameters)diff --git a/test/unit/test_cuda/quantization/test_neuqi_e2e.py b/test/unit/test_cuda/quantization/test_neuqi_e2e.pynew file mode 100644index 000000000..ca001b794--- /dev/null+++ b/test/unit/test_cuda/quantization/test_neuqi_e2e.py@@ -0,0 +1,75 @@+# Copyright (c) 2026 Intel Corporation+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+"""End-to-end NeUQI runs through the public AutoRound API on CUDA.++The CPU tier covers the search math and dispatch tables; these runs verify+the full wiring (config routing, imatrix collection, frozen-init anchor,+layer_config output) with the REAL backend ladder engaged -- on CUDA the+Triton/compile sweep paths serve the search instead of the eager fallback,+so default grids (256/64) run in seconds. Hosts without CUDA skip."""++import pytest+import torch++pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")++_DS = ["local NeUQI calibration sample with enough tokens for quantization"] * 2+++def _autoround(model_path, sym, iters):+ from auto_round import AutoRound++ return AutoRound(+ model_path,+ bits=4,+ group_size=32,+ sym=sym,+ enable_neuqi=True,+ iters=iters,+ nsamples=2,+ seqlen=8,+ batch_size=2,+ dataset=list(_DS),+ device_map="cuda:0",+ enable_torch_compile=False,+ )+++def test_zero_shot_neuqi_e2e(tiny_opt_model_path, tmp_path):+ ar = _autoround(tiny_opt_model_path, sym=False, iters=0)+ model, layer_config = ar.quantize()+ assert model is not None and len(layer_config) > 0+ for name, cfg in layer_config.items():+ assert cfg["bits"] == 4, f"Layer {name} expected bits=4, got {cfg['bits']}"+ assert cfg["data_type"] == "int", f"Layer {name} expected int, got {cfg.get('data_type')}"++[email protected]("sym", [False, True])+def test_frozen_init_e2e(tiny_opt_model_path, monkeypatch, sym):+ """Both symmetry classes share the tuning wiring; only the anchor's+ internals (joint vs two-stage search) differ."""+ import auto_round.data_type.neuqi as neuqi_mod++ calls = []+ orig_anchor = neuqi_mod.neuqi_anchor_margins++ def spy(weight, *a, **kw):+ calls.append(tuple(weight.shape))+ return orig_anchor(weight, *a, **kw)++ monkeypatch.setattr(neuqi_mod, "neuqi_anchor_margins", spy)+ ar = _autoround(tiny_opt_model_path, sym=sym, iters=1)+ model, layer_config = ar.quantize()+ assert model is not None and len(layer_config) > 0+ assert calls, f"the NeUQI frozen-init anchor never fired on the iters>0 path (sym={sym})"diff --git a/test/unit/test_cuda/quantization/test_neuqi_triton.py b/test/unit/test_cuda/quantization/test_neuqi_triton.pynew file mode 100644index 000000000..4450813a5--- /dev/null+++ b/test/unit/test_cuda/quantization/test_neuqi_triton.py@@ -0,0 +1,76 @@+# Copyright (c) 2026 Intel Corporation+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+"""CUDA parity test for the NeUQI Triton sweep kernel against the eager reference.++The kernel rounds with round-half-to-even and keeps the first-minimum zero-point+rule, so selections must match the eager per-candidate sweep up to the usual+fp32 summation ties (fp64-oracle checked). Skipped on hosts without CUDA+triton.+"""++import pytest+import torch++from auto_round.data_type.neuqi import neuqi_search_scale_zero++triton_sweep = pytest.importorskip("auto_round_extension.triton.neuqi_sweep").neuqi_sweep_triton++pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")+++def _oracle_loss_fp64(data, qw, scale, zp, maxq):+ r = torch.round(data.double() / scale.double()).clamp_(min=-maxq - 1, max=2 * maxq + 1)+ q = (r + zp.double()).clamp_(0, maxq)+ deq = scale.double() * (q - zp.double())+ loss = (deq - data.double()) ** 2+ if qw is not None:+ loss = loss * qw.double()+ return loss.sum(-1)++[email protected]("bits", [2, 3, 4, 8])[email protected]("weighted", [False, True])[email protected]("g", [32, 64, 100]) # 100: non-power-of-two group size (masked)+def test_triton_sweep_matches_eager(bits, weighted, g, monkeypatch):+ gen = torch.Generator().manual_seed(bits * 13 + int(weighted) * 7 + g)+ n = 512+ data = torch.randn(n, g, generator=gen)+ data[:, :5] *= 90.0+ qw = torch.rand(n, g, generator=gen) + 0.1 if weighted else None+ data_c, qw_c = data.cuda(), (qw.cuda() if qw is not None else None)++ scale_ref, zp_ref = neuqi_search_scale_zero(data.clone(), bits, qw=qw, coarse_n=8, fine_n=3)++ import auto_round.data_type.neuqi as N+ from auto_round import envs++ monkeypatch.setenv("AR_NEUQI_BACKEND", "triton") # envs reads os.getenv per access; setenv auto-restores+ saved_batch = N._zp_batch_wanted+ N._zp_batch_wanted = lambda dev: True # batching is device-automatic now; force it for the parity run+ try:+ N._triton_checked, N._triton_sweep, N._triton_broken = True, triton_sweep, False+ N._fused_zp_fns, N._fused_zp_broken, N._batched_broken = {}, False, False+ scale_t, zp_t = neuqi_search_scale_zero(data_c, bits, qw=qw_c, coarse_n=8, fine_n=3)+ finally:+ N._zp_batch_wanted = saved_batch++ assert not N._triton_broken, "kernel raised; fallback engaged instead"+ scale_ref, zp_ref = scale_ref.cpu(), zp_ref.cpu()+ scale_t, zp_t = scale_t.cpu(), zp_t.cpu()++ mismatch = (zp_ref != zp_t).logical_or(scale_ref != scale_t).nonzero().flatten()+ maxq = 2**bits - 1+ for i in mismatch.tolist():+ loss_ref = _oracle_loss_fp64(data[i], qw[i] if qw is not None else None, scale_ref[i], zp_ref[i], maxq)+ loss_t = _oracle_loss_fp64(data[i], qw[i] if qw is not None else None, scale_t[i], zp_t[i], maxq)+ assert (loss_t - loss_ref).abs().item() <= 1e-4 * loss_ref.abs().item()