Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

NeUQI grid search for optimized RTN (--enable_neuqi: joint asym scale/zp search, two-stage sym search, frozen-init anchor) (#2341)

avtc44b48d60875e
diff --git a/README.md b/README.md
index 5dd3c4535..ed040ce09 100644
--- a/README.md
+++ b/README.md
@@ -41,6 +41,8 @@ See our papers [SignRoundV1](https://arxiv.org/pdf/2309.05516) and [SignRoundV2]
* [2026/09] We now support 5/6/7-bit WOQ models in vLLM and Transformers on CUDA devices, thanks to Humming Kernel.
+* [2026/09] The **NeUQI** grid search ([arXiv 2505.17595](https://arxiv.org/abs/2505.17595)) is available for calibration-free quantization: `--enable_neuqi` with `--iters 0` runs a joint (scale, zero-point) search for asymmetric layers and a two-stage scale search for symmetric layers, and anchors the tuning grid when `iters > 0`. [*Results*](./docs/neuqi_acc.md)
+
* [2026/08] We experimentally support **algorithm composition** (e.g., `--algs awq,signround` or `--algs hadamard,awq,signround`) to improve accuracy [*Overview*](./docs/algorithm_combinations.md). We welcome any practical, deployable algorithms that are ready for real-world use. Feel free to submit a PR or leave a comment in Issues.
* [2026/06] AutoScheme has been refined to improve accuracy for gguf format. See [AutoScheme Accuracy](./docs/auto_scheme_acc.md) for details. This enhancement incurs additional tuning cost.
@@ -78,7 +80,7 @@ Support **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** for maximum compatibility. De
Automatically configure in minutes, with about 1.1X-1.5X the model’s BF16 RAM size as overhead. Accuracy [results](./docs/auto_scheme_acc.md) and [user guide](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#autoscheme).
✅ **Optimized Round-to-Nearest Mode**
-Use `--iters 0` for fast quantization with some accuracy drop for 4 bits. Details are shown in [opt_rtn mode](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#opt-rtn-mode)
+Use `--iters 0` for fast quantization with some accuracy drop for 4 bits; an optional NeUQI grid search (`--enable_neuqi`) improves the zero-shot scales. Details are shown in [opt_rtn mode](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#opt-rtn-mode)
✅ **Affordable Quantization Cost**
Quantize 7B models in about 10 minutes on a single GPU. Details are shown in [quantization costs](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#quantization-costs)
diff --git a/README_CN.md b/README_CN.md
index cf1863231..e1ca93044 100644
--- a/README_CN.md
+++ b/README_CN.md
@@ -39,6 +39,8 @@ AutoRound 是专为大语言模型(LLMs)和视觉-语言模型(VLMs)设
* [2026/09] 现在支持在 CUDA 设备上通过 vLLM 和 Transformers 使用 5/6/7-bit WOQ 模型,感谢 Humming Kernel 的支持。
+* [2026/09] 面向免标定量化的 **NeUQI** 网格搜索([arXiv 2505.17595](https://arxiv.org/abs/2505.17595))现已可用:`--enable_neuqi` 配合 `--iters 0` 时,非对称层执行联合 (scale, zero-point) 搜索、对称层执行两阶段 scale 搜索;`iters > 0` 时搜索结果将作为调优网格的锚点:[*结果*](./docs/neuqi_acc.md)。
+
* [2026/08] 我们实验性地支持**算法组合**(例如 `--algs awq,signround` 或 `--algs hadamard,awq,signround`),以提升精度:[*总览*](./docs/algorithm_combinations_CN.md). 我们欢迎能真正落地的任何算法组合,欢迎提交 PR 或在 Issues 中留言。
* [2026/08] 感谢 Humming Kernel,AutoScheme WOQ 在 vLLM 上的部署已实验性恢复:[*vLLM PR*](https://github.com/vllm-project/vllm/pull/52890),[*示例模型*](https://huggingface.co/Intel/Qwen3.8-27B-bpw2.8-AutoRound)。注意:共享层需按 vLLM 的融合模式进行配置。
diff --git a/auto_round/algorithms/quantization/base.py b/auto_round/algorithms/quantization/base.py
index 70b3299e4..f9cfaebd4 100644
--- a/auto_round/algorithms/quantization/base.py
+++ b/auto_round/algorithms/quantization/base.py
@@ -231,6 +231,7 @@ class BaseQuantizer(BaseAlgorithm):
enable_round_tuning=False,
enable_torch_compile=self.compress_context.enable_torch_compile,
disable_opt_rtn=disable_opt_rtn,
+ enable_neuqi=getattr(self.config, "enable_neuqi", False),
iters=0,
)
layer = layer.unwrapper({})
@@ -247,6 +248,8 @@ class BaseQuantizer(BaseAlgorithm):
enable_norm_bias_tuning=False,
enable_round_tuning=False,
enable_torch_compile=self.compress_context.enable_torch_compile,
+ disable_opt_rtn=disable_opt_rtn,
+ enable_neuqi=getattr(self.config, "enable_neuqi", False),
iters=0,
)
layer = layer.unwrapper({})
diff --git a/auto_round/algorithms/quantization/rtn/config.py b/auto_round/algorithms/quantization/rtn/config.py
index b7b90ef39..ac4fc376c 100644
--- a/auto_round/algorithms/quantization/rtn/config.py
+++ b/auto_round/algorithms/quantization/rtn/config.py
@@ -41,12 +41,24 @@ class RTNConfig(QuantizationConfig):
const=False,
help="Force optimized RTN path.",
)
+ registry.add_argument(
+ "--enable_neuqi",
+ field="enable_neuqi",
+ action="store_true",
+ help=(
+ "Enable the NeUQI grid search (arXiv 2505.17595): the asymmetric zero-shot "
+ "optimized-RTN path runs a joint (scale, integer zero-point) search instead of plain "
+ "min/max, symmetric layers run the two-stage scale search, and SignRound tuning "
+ "anchors its grid to the search result (frozen init)."
+ ),
+ )
def __init__(
self,
*,
disable_opt_rtn: bool = None,
enable_opt_rtn: bool = None,
+ enable_neuqi: bool = False,
**kwargs,
) -> None:
"""Initialize an RTN configuration.
@@ -56,6 +68,14 @@ class RTNConfig(QuantizationConfig):
``None`` keeps the default heuristic, True forces plain
RTN, and False forces the optimized implementation.
enable_opt_rtn: Convenience alias for ``disable_opt_rtn=False``.
+ enable_neuqi: Opt into the NeUQI grid search (arXiv 2505.17595):
+ the asymmetric zero-shot optimized-RTN path runs a joint
+ (scale, integer zero-point) search instead of plain min/max,
+ symmetric layers run the two-stage scale search, and SignRound
+ tuning (``iters > 0``) anchors its grid to the search result
+ (frozen init). Requires the optimized path (``disable_opt_rtn``
+ not forced True). The search grid sizes are tunable via
+ ``AR_NEUQI_COARSE`` and ``AR_NEUQI_FINE``.
**kwargs: Common quantization arguments forwarded to
QuantizationConfig, such as bits, group_size, sym,
data_type, and activation quantization fields.
@@ -77,6 +97,16 @@ class RTNConfig(QuantizationConfig):
disable_opt_rtn = False
self.disable_opt_rtn = disable_opt_rtn
+ if enable_neuqi and disable_opt_rtn:
+ # validated AFTER the W8A16/W8A8 heuristic above so the auto-disable
+ # cannot silently neutralize the search opt-in
+ raise ValueError(
+ "--enable_neuqi requires the optimized-RTN path: it replaces the plain min/max "
+ "initialization with a search. Drop --disable_opt_rtn. Note "
+ "that W8A16/W8A8 schemes auto-disable the optimized path for efficiency."
+ )
+ self.enable_neuqi = enable_neuqi
+
class OptimizedRTNConfig(RTNConfig):
need_calib = True
diff --git a/auto_round/algorithms/quantization/rtn/quantizer.py b/auto_round/algorithms/quantization/rtn/quantizer.py
index e4c6466e2..38584dee6 100644
--- a/auto_round/algorithms/quantization/rtn/quantizer.py
+++ b/auto_round/algorithms/quantization/rtn/quantizer.py
@@ -12,7 +12,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+import re
+
import torch
+import torch.nn as nn
from auto_round.algorithms.quantization.base import BaseQuantizer
from auto_round.algorithms.quantization.rtn.config import OptimizedRTNConfig, RTNConfig
@@ -22,6 +25,11 @@ from auto_round.utils import (
SUPPORTED_LAYER_TYPES,
check_to_quantized,
)
+from auto_round.utils.device_manager import device_manager
+
+_EXPERT_BATCH_MAX_ELEMS = 2**28 # ~1 GiB fp32 stacked weights per batched call
+
+_EXPERT_RE = re.compile(r"^(?P<parent>.*)\.experts\.\d+\.(?P<proj>[^.]+)$")
@register_pipeline_member(RTNConfig)
@@ -132,9 +140,200 @@ class OptimizedRTNQuantizer(RTNQuantizer):
input_ids: Raw token IDs from the tokenizer (unused in RTN).
**kwargs: Reserved for forward-compatibility with future parameters.
"""
- # Normalize imatrix and quantize layers
+ # Normalize imatrix (cheap elementwise divides), then quantize the
+ # block's target modules - same-shape expert projections batched into
+ # single stacked search calls when the search is active
+ targets = []
for name, m in block.named_modules():
if hasattr(m, "imatrix"):
m.imatrix /= m.imatrix_cnt
if hasattr(m, "global_name") and check_to_quantized(m):
- self.quantize_layer_outside_block(m)
+ targets.append(m)
+ self._quantize_targets(targets)
+
+ def _split_expert_batches(self, targets: list):
+ """Partition same-shape expert projections into batchable groups.
+
+ Experts of one layer share weight shapes (e.g. 192 x ``[1536, 4096]``
+ gate/up/down projections). The per-group search is row-independent, so
+ a whole group can be quantized in one call by stacking weights along
+ the output dim - same results as per-module calls, one search's worth
+ of per-module overhead instead of 192.
+ """
+ grouped = {}
+ singles = []
+ for m in targets:
+ match = _EXPERT_RE.match(getattr(m, "global_name", "") or "")
+ if (
+ match is None
+ or type(m) is not nn.Linear
+ or not isinstance(getattr(m, "group_size", None), int)
+ or getattr(m, "group_size", None) <= 0
+ or getattr(m, "super_bits", None) is not None
+ or getattr(m, "data_type", "int") != "int"
+ or m.weight.shape[1] % m.group_size != 0 # not row-divisible: keep per-module
+ or getattr(m, "act_bits", 16) <= 8 # act-quant layers need the per-module wrapper path
+ ):
+ singles.append(m)
+ continue
+ key = (
+ match["parent"],
+ match["proj"],
+ tuple(m.weight.shape),
+ m.bits,
+ m.group_size,
+ bool(getattr(m, "sym", False)),
+ )
+ grouped.setdefault(key, []).append(m)
+ batches = []
+ for g in grouped.values():
+ if len(g) >= 2:
+ batches.append(g)
+ else:
+ singles.extend(g) # singleton groups must stay per-module, never dropped
+ return batches, singles
+
+ def _expert_search_active(self) -> bool:
+ """Whether the optimized-RTN NeUQI search runs for expert modules.
+
+ Mirrors the MoE heuristic in ``_quantize_layer_via_rtn``: experts skip
+ the search unless explicitly forced on (``enable_opt_rtn``), so batching
+ must not change that decision - only batch when the search would run.
+ Both symmetry classes search under ``enable_neuqi`` (asym: joint
+ (scale, zp); sym: two-stage scale), so both batch.
+ """
+ cfg = self.config
+ if bool(getattr(cfg, "disable_opt_rtn", False)):
+ return False
+ if getattr(cfg, "orig_disable_opt_rtn", None) is None and getattr(self.model_context, "is_moe_model", False):
+ return False
+ return bool(getattr(cfg, "enable_neuqi", False))
+
+ def _quantize_expert_batch(self, mods: list, device) -> list:
+ """Quantize same-shape expert projections in one stacked search call.
+
+ Returns the modules NOT quantized here (empty list = success); the
+ caller falls back to per-module calls for them. Reproduces the
+ per-module wrapper outputs: quantize-dequantized weights written in
+ place, plus ``scale``/``zp``/``q_scale_thresh``/``data_type``
+ attributes matching the unwrapper's conventions. On a mid-batch
+ failure (e.g. OOM) already-written modules keep their results and
+ only the remainder is returned -- per-module fallback must never
+ re-quantize an already quantized-dequantized weight.
+ """
+ m0 = mods[0]
+ bits, g = m0.bits, m0.group_size
+ sym = bool(getattr(m0, "sym", False))
+ # mirror the wrapper's threshold rule: fp32-scale modules use a tighter
+ # q_scale_thresh than the fp16 default
+ scale_dtype = getattr(m0, "scale_dtype", torch.float16)
+ q_scale_thresh = 1e-8 if scale_dtype == torch.float32 else 1e-5
+ resolved_dtype = "opt_rtn_int_sym_neuqi" if sym else "opt_rtn_int_asym"
+ try:
+ if sym:
+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_sym_neuqi
+ else:
+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_asym
+ except ImportError:
+ return list(mods)
+
+ max_elems = _EXPERT_BATCH_MAX_ELEMS
+ per_call = max(1, max_elems // m0.weight.numel())
+ # divisibility is guaranteed by _split_expert_batches; re-checked here
+ # so a direct call can never half-write a chunk before bailing out
+ if any(m.weight.shape[1] % g != 0 for m in mods):
+ return list(mods)
+ written = 0 # modules fully quantized so far (chunk writes are atomic)
+ try:
+ for s in range(0, len(mods), per_call):
+ chunk = mods[s : s + per_call]
+ dev = torch.device(device)
+ weights = torch.cat([m.weight.data.reshape(m.weight.shape[0], -1) for m in chunk], dim=0).to(dev)
+ imat_rows = []
+ for m in chunk:
+ if hasattr(m, "imatrix"):
+ imat_rows.append(m.imatrix.to(dev).unsqueeze(0).expand(m.weight.shape[0], -1))
+ if imat_rows and len(imat_rows) < len(chunk): # mixed: fill gaps uniformly
+ it = iter(imat_rows)
+ imat_rows = [
+ next(it) if hasattr(m, "imatrix") else torch.ones(m.weight.shape, device=dev) for m in chunk
+ ]
+ # uniform weighting needs no materialized tensor: qw=None is
+ # bit-identical to a full ones imatrix and skips ~module-size
+ # allocations per expert
+ imat = torch.cat(imat_rows, dim=0) if imat_rows else None
+ if dev.type == "cuda":
+ torch.cuda.synchronize(dev)
+
+ if sym:
+ # returns follow the quant_tensor_opt_rtn_sym conventions: the
+ # zero point is the scalar nmax
+ qdq, scale, zp = quant_tensor_opt_rtn_sym_neuqi(
+ weights,
+ bits=bits,
+ group_size=g,
+ v=0.0,
+ q_scale_thresh=q_scale_thresh,
+ imatrix=imat,
+ scale_dtype=scale_dtype,
+ )
+ else:
+ qdq, scale, zp = quant_tensor_opt_rtn_asym(
+ weights,
+ bits=bits,
+ group_size=g,
+ v=0.0,
+ q_scale_thresh=q_scale_thresh,
+ imatrix=imat,
+ scale_dtype=scale_dtype,
+ )
+
+ if dev.type == "cuda":
+ torch.cuda.synchronize(dev)
+ row_w = row_g = 0
+ with torch.no_grad():
+ for m in chunk:
+ out = m.weight.shape[0]
+ n_groups_m = out * (m.weight.shape[1] // g)
+ m.weight.data.copy_(qdq[row_w : row_w + out].reshape(m.weight.shape).to(m.weight.device))
+ m.scale = scale[row_g : row_g + n_groups_m].reshape(out, -1).to("cpu")
+ if sym:
+ m.zp = zp
+ else:
+ m.zp = zp[row_g : row_g + n_groups_m].reshape(out, -1).to("cpu")
+ m.q_scale_thresh = q_scale_thresh
+ m.data_type = resolved_dtype
+ m.weight.grad = None
+ row_w += out
+ row_g += n_groups_m
+ written = s + len(chunk)
+ if dev.type == "cuda":
+ torch.cuda.synchronize(dev)
+ except torch.OutOfMemoryError:
+ # fall back to per-module calls for the unwritten remainder only:
+ # smaller transients, identical results, no double quantization
+ pass
+ return mods[written:]
+
+ def _quantize_targets(self, targets: list) -> None:
+ """Quantize a block's target modules; same-shape expert groups in one
+ stacked search call each (row-independent search: results identical to
+ per-module calls)."""
+ if not targets:
+ return
+ batches = []
+ if self._expert_search_active():
+ batches, targets = self._split_expert_batches(targets)
+ if batches:
+ n_batched = sum(len(b) for b in batches)
+ logger.info(
+ "[OptRTN] expert batching: %d expert modules in %d batched groups.",
+ n_batched,
+ len(batches),
+ )
+ device = device_manager.device
+ for b in batches:
+ for m in self._quantize_expert_batch(b, device):
+ self.quantize_layer_outside_block(m) # unwritten remainder: per-module fallback
+ for m in targets:
+ self.quantize_layer_outside_block(m)
diff --git a/auto_round/algorithms/quantization/sign_round/config.py b/auto_round/algorithms/quantization/sign_round/config.py
index 03de8728e..a362a34e1 100644
--- a/auto_round/algorithms/quantization/sign_round/config.py
+++ b/auto_round/algorithms/quantization/sign_round/config.py
@@ -120,6 +120,16 @@ class SignRoundConfig(QuantizationConfig):
action="boolean_optional",
help="Enable last-block LM cross-entropy (LFQ) loss for the final transformer block (experimental).",
)
+ registry.add_argument(
+ "--enable_neuqi",
+ field="enable_neuqi",
+ action="store_true",
+ help=(
+ "Enable the NeUQI grid search (arXiv 2505.17595): the tuning grid is anchored to "
+ "the searched (scale, zero-point) result once at layer init and frozen during "
+ "tuning (joint search for asym layers, two-stage scale search for sym layers)."
+ ),
+ )
def __init__(
self,
@@ -140,6 +150,7 @@ class SignRoundConfig(QuantizationConfig):
optimizer: str | None = None, # TODO later wenhuach delete this
enable_adam: bool = False, # TODO later wenhuach delete this
enable_lfq: bool = False,
+ enable_neuqi: bool = False,
**kwargs,
) -> None:
"""Initialize a SignRound configuration.
@@ -170,11 +181,18 @@ class SignRoundConfig(QuantizationConfig):
quantized output of previous blocks during calibration.
optimizer: Optional optimizer name override.
enable_adam: Whether to use the Adam-based SignRound variant.
+ enable_neuqi: Opt into the NeUQI grid search (arXiv 2505.17595):
+ the joint (scale, integer zero-point) search runs once at
+ layer init and anchors the tuning grid to the winner, frozen
+ during tuning (the two-stage symmetric scale search for sym
+ layers). The search grid sizes are tunable via
+ ``AR_NEUQI_COARSE`` and ``AR_NEUQI_FINE``.
**kwargs: Common quantization arguments forwarded to
QuantizationConfig, such as bits, group_size, sym,
data_type, and activation quantization fields.
"""
super().__init__(**kwargs)
+ self.enable_neuqi = enable_neuqi
self.gradient_accumulate_steps = gradient_accumulate_steps
self.iters = iters
if self.iters < 0:
diff --git a/auto_round/algorithms/quantization/sign_round/quantizer.py b/auto_round/algorithms/quantization/sign_round/quantizer.py
index 116e5b2e9..e5a7e0257 100644
--- a/auto_round/algorithms/quantization/sign_round/quantizer.py
+++ b/auto_round/algorithms/quantization/sign_round/quantizer.py
@@ -374,6 +374,7 @@ class SignRoundQuantizer(BaseQuantizer):
self.enable_norm_bias_tuning,
enable_torch_compile=self.compress_context.enable_torch_compile,
device=device,
+ enable_neuqi=getattr(self.config, "enable_neuqi", False),
)
round_params = []
@@ -666,6 +667,7 @@ class SignRoundQuantizer(BaseQuantizer):
enable_minmax_tuning=self.enable_minmax_tuning,
enable_torch_compile=self.compress_context.enable_torch_compile,
device=device,
+ enable_neuqi=getattr(self.config, "enable_neuqi", False),
).to(device)
round_params = []
minmax_params = []
diff --git a/auto_round/autoround.py b/auto_round/autoround.py
index 22baaf226..4826d1b0c 100644
--- a/auto_round/autoround.py
+++ b/auto_round/autoround.py
@@ -187,6 +187,12 @@ def _select_rtn_compressor_base_cls(quant_config: "RTNConfig", scheme, format, b
bits = resolved_attrs.get("bits")
if sym is not None and sym is False:
enable_imatrix = False
+ # enable_neuqi: the joint (scale, zero-point) search
+ # consumes the activation imatrix as a per-element weighting,
+ # so it must be collected even on the asymmetric path where
+ # the plain min/max initialization ignores it
+ if getattr(quant_config, "enable_neuqi", False):
+ enable_imatrix = True
elif data_type == "int" and (bits is None or bits < 8):
enable_imatrix = True
elif is_weight_scheme(scheme):
diff --git a/auto_round/data_type/__init__.py b/auto_round/data_type/__init__.py
index 58c746470..f51cb80ed 100644
--- a/auto_round/data_type/__init__.py
+++ b/auto_round/data_type/__init__.py
@@ -24,3 +24,4 @@ from auto_round.data_type.utils import (
)
import auto_round.data_type.nvfp
import auto_round.data_type.gguf
+import auto_round.data_type.neuqi
diff --git a/auto_round/data_type/neuqi.py b/auto_round/data_type/neuqi.py
new file mode 100644
index 000000000..d83afeb6d
--- /dev/null
+++ b/auto_round/data_type/neuqi.py
@@ -0,0 +1,1439 @@
+# Copyright (c) 2026 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Near-optimal uniform (scale, zero-point) search for asymmetric integer quantization.
+
+This is a clean-room reimplementation of the integer-zero-point variant of the NeUQI
+objective (arXiv 2505.17595, "NeUQI: Near-Optimal Uniform Quantization Parameter
+Initialization for Low-Bit LLMs"). The reference implementation carries no license,
+so no code is copied from it; this module implements the published optimization from
+scratch. Only the integer-zero-point variant is implemented because the paper's
+floating-point zero point is not consumable by stock int4 kernels (GPTQ/AWQ/Marlin
+assume an integral zero point).
+
+For each quantization group the following problem is solved:
+
+ min_{s > 0, z in {0, .., 2**bits - 1}}
+ sum_i qw_i * (s * (clamp(round(w_i / s) + z, 0, 2**bits - 1) - z) - w_i)**2
+
+where ``qw`` is an optional per-element weighting (e.g. the activation ``imatrix``).
+For a fixed scale the loss decomposes over the integer zero points, so the optimal
+``z`` is found exactly by enumerating all ``2**bits`` candidates (16 for 4-bit).
+The scale is searched on a coarse log grid bounded by the min-max scale (scales
+beyond the min-max range are never optimal) followed by a fine additive refinement
+around the best coarse entry.
+
+The search replaces the plain min/max initialization of the asymmetric optimized-RTN
+path and is a strict generalization of the symmetric ``search_scales`` grid (which
+fixes ``z = 0``).
+
+The symmetric counterpart (:func:`neuqi_search_scale_sym`) fixes the zero point at
+0 and searches only the scale on the same two-stage grid; it replaces the uniform
+``search_scales`` grid of ``opt_rtn_int_sym`` when engaged via
+``enable_neuqi`` with ``sym=True``.
+
+Performance: the per-candidate zero-point sweep is a pure elementwise + reduction chain,
+so on CUDA it is fused into a single generated kernel via ``torch.compile``
+(``AR_NEUQI_BACKEND=auto``, the default), and all coarse/fine candidates of a pass
+are additionally folded into one such kernel per group chunk (the batched sweep)
+so the search costs a handful of launches instead of candidates x chunks. On CUDA
+the batched chunk is preferentially served by a hand-written Triton kernel
+(``auto_round_extension.triton.neuqi_sweep``) that loads each weight element into
+registers once and computes every (candidate, zero-point) loss from registers,
+removing the L2 re-read amplification of the compiled expression. Measured on an
+RTX 3090 (2M groups, g=64, bits=4, 64+32 grid): 12.3 s eager -> 0.59 s fused
+per-candidate -> 0.49 s compiled-batched -> 0.22 s Triton (~57x). The fused sweeps
+evaluate exactly the same losses on exactly the same candidate grid; only the fp32
+summation order inside the reduction may differ, so selections can flip solely on
+near-ties: on random, heavy-tailed and imatrix-weighted inputs the flips stay below
+~0.1% of groups with worst-case relative loss differences of ~5e-5 (measured on an
+RTX 3090; ~1e-6 on CPU), with the Triton kernel matching that tie profile exactly.
+The eager chunked sweep remains the reference path (CPU default) and the ultimate
+fallback; every stage latches down permanently on failure (Triton -> compiled
+batched -> compiled per-candidate -> eager).
+"""
+
+import math
+import threading
+from functools import lru_cache
+
+import torch
+
+from auto_round import envs
+from auto_round.data_type.register import register_dtype
+from auto_round.data_type.utils import reshape_pad_tensor_by_group_size, revert_tensor_by_pad, round_ste
+from auto_round.logger import logger
+
+# Upper bound on the number of elements of the temporary [chunk, group, zero_point]
+# loss tensor, used to cap peak memory regardless of the input size. The bound is
+# honored by both the eager and the fused sweep, so a dynamo recompile-limit fallback
+# to eager execution degrades speed only, never correctness or memory.
+_MAX_TMP_ELEMS = 2**25
+
+
+def _align_qw(qw, data):
+ """Move per-element loss weights onto the data device when they disagree.
+
+ Parallel fan-out workers relocate a module to their GPU for the search;
+ a plain-tensor ``imatrix`` attribute does not travel with ``module.to()``,
+ leaving qw on the block's home device. Cross-device inputs would trip the
+ kernel same-device asserts mid-run (and, under a compiled caller, dynamo's
+ fake-tensor device propagation), so align once here.
+ """
+ if qw is not None and qw.device != data.device:
+ qw = qw.to(data.device)
+ return qw
+
+
+@lru_cache(maxsize=1)
+def _log_search_engaged(coarse_n: int, fine_n: int) -> None:
+ logger.info("[NeUQI] joint (scale, zero-point) search active (coarse=%d, fine=%d)", coarse_n, fine_n)
+
+
+def _coarse_grid(lo: float, n: int, device: torch.device, dtype: torch.dtype, hi: float = 1.0) -> torch.Tensor:
+ """Log-spaced coarse scale fractions, always built eagerly.
+
+ torch.logspace lowers to an fp64 ``libdevice.pow`` inside compiled
+ regions, and inductor has been observed to fuse that into the batched
+ sweep's index-select math, where Triton rejects fp64 in the integer
+ index guards ("unexpected type fp64" -- BPT worker crash, first MoE
+ block under ``--enable_torch_compile``). Building the tiny [n] grid
+ eagerly keeps it a plain tensor input to any traced region.
+
+ Deliberately NOT cached: the expert search fans out from a thread pool
+ (one thread per device), and a cached tensor would be created on the
+ first caller's stream and read from the others' streams with no event
+ ordering -- a cross-stream read-before-write window that surfaced as a
+ device-side assert on the streaming parent (hidden under
+ CUDA_LAUNCH_BLOCKING=1). Recreating 64 floats per call is free and
+ keeps creation on the calling thread's own stream.
+ """
+ return torch.logspace(math.log10(lo), math.log10(hi), n, device=device, dtype=dtype)
+
+
+def _zp_expr_last(data, qw, scale, zp_grid, maxq):
+ """Zero-point sweep with the group dimension last: ``[C, Z, g]``, sum over ``dim=-1``.
+
+ Reduction over the contiguous last dimension is the canonical fused shape for
+ the Triton/CUDA codegen: the scheduler can keep the whole pointwise chain
+ (round -> clamp -> dequant -> squared error -> group sum -> zero-point min) in
+ registers and stream the group axis with coalesced loads. The equivalent
+ middle-dim reduction (``[C, g, Z]``) measured no faster than eager on CUDA
+ (RTX 3090) because the expanded buffer must be materialized or transposed to
+ reduce a non-contiguous axis; it is kept as ``_zp_expr_mid`` where it is the
+ faster layout (CPU, including eager execution).
+
+ Same operands and operand order per logical (group, zero-point) element as the
+ reference sweep, so losses are identical up to the fp32 summation order over
+ the group (last-ulp ties only).
+
+ Args:
+ data: [chunk, g] float32 group data.
+ qw: [chunk, g] float32 per-element weights or ``None``.
+ scale: [chunk, 1] float32 scale candidate.
+ zp_grid: [n_zp] float32 integral zero-point candidates ``0 .. maxq``.
+ maxq: maximum integer value (``2**bits - 1``).
+
+ Returns:
+ (min_loss [chunk], argmin_zp [chunk]) with the first-minimum tie rule of
+ ``Tensor.min`` (same winner as the eager sweep on equal losses).
+ """
+ # Stabilize rounding: values far outside the grid saturate identically after
+ # clamping, so clamping r first never changes clamp(r + z, 0, maxq) for z >= 0.
+ # Broadcasts give [C, Z, g]: data/qw over the middle (stride-0) zero-point
+ # axis, scale shared across the group, reduction over the last (contiguous) dim.
+ d = data.unsqueeze(-2) # [C, 1, g]
+ sc = scale.unsqueeze(-1) # [C, 1, 1]
+ zp = zp_grid.view(1, -1, 1) # [1, Z, 1]
+ r = torch.round(d / sc).clamp_(min=-maxq - 1, max=2 * maxq + 1)
+ q = (r + zp).clamp_(0, maxq)
+ deq = sc * (q - zp)
+ diff = deq - d
+ loss = diff * diff
+ if qw is not None:
+ loss = loss * qw.unsqueeze(-2)
+ loss = loss.sum(dim=-1) # [C, Z]
+ return loss.min(dim=-1)
+
+
+def _zp_expr_mid(data, qw, scale, zp_grid, maxq):
+ """Zero-point sweep with the zero-point dimension last: ``[C, g, Z]``, sum over ``dim=1``.
+
+ Byte-identical operand order to the historical chunked sweep. This is the
+ faster layout for eager execution and for the compiled CPU backend (the
+ zero-point broadcast rides the contiguous last axis, which TensorIterator
+ vectorizes well); it loses to ``_zp_expr_last`` under the Triton/CUDA
+ scheduler for the reasons given above.
+
+ Same args and returns as ``_zp_expr_last``.
+ """
+ # Stabilize rounding: values far outside the grid saturate identically after
+ # clamping, so clamping r first never changes clamp(r + z, 0, maxq) for z >= 0.
+ r = torch.round(data / scale).clamp_(min=-maxq - 1, max=2 * maxq + 1)
+ q = (r.unsqueeze(-1) + zp_grid).clamp_(0, maxq)
+ deq = scale.unsqueeze(-1) * (q - zp_grid)
+ diff = deq - data.unsqueeze(-1)
+ loss = diff * diff
+ if qw is not None:
+ loss = loss * qw.unsqueeze(-1)
+ loss = loss.sum(dim=1) # [chunk, n_zp]
+ return loss.min(dim=-1)
+
+
+def _zp_expr_batched(data, qw, scales, zp_grid, maxq):
+ """All-candidates zero-point sweep in one call: ``[C, K, Z, g]``, group last.
+
+ Evaluates every per-group scale candidate (``scales``, [C, K]) against every
+ integer zero point in a single fused kernel: the group-axis reduction and the
+ min over zero points are fused in-kernel, so the kernel outputs only the
+ [C, K] best losses and [C, K] winning zero points -- one launch covers what
+ the per-candidate sweep needs K launches and K bookkeeping rounds for. This
+ removes the Python-dispatch bound of the per-candidate loop (the residual
+ cost after single-candidate fusion) at the price of a larger symbolic
+ intermediate, which is why eager execution of this expression is never used
+ (see ``_eval_batched_chunk``).
+
+ Same per-(group, candidate, zero-point) operand order as ``_zp_expr_last``;
+ the caller applies the first-minimum-over-candidates rule with ``Tensor.min``,
+ matching the sequential sweep's strict-improvement tie semantics.
+
+ Args:
+ data: [chunk, g] float32 group data.
+ qw: [chunk, g] float32 per-element weights or ``None``.
+ scales: [chunk, K] float32 per-group scale candidates.
+ zp_grid: [n_zp] float32 integral zero-point candidates ``0 .. maxq``.
+ maxq: maximum integer value (``2**bits - 1``).
+
+ Returns:
+ (best_loss [chunk, K], best_zp [chunk, K]) -- per-candidate minimum over
+ zero points, first-minimum tie rule.
+ """
+ d = data.unsqueeze(-2).unsqueeze(-2) # [C, 1, 1, g]
+ sc = scales.unsqueeze(-1).unsqueeze(-1) # [C, K, 1, 1]
+ zp = zp_grid.view(1, 1, -1, 1) # [1, 1, Z, 1]
+ r = torch.round(d / sc).clamp_(min=-maxq - 1, max=2 * maxq + 1)
+ q = (r + zp).clamp_(0, maxq)
+ deq = sc * (q - zp)
+ diff = deq - d
+ loss = diff * diff
+ if qw is not None:
+ loss = loss * qw.unsqueeze(-2).unsqueeze(-2)
+ loss = loss.sum(dim=-1) # [C, K, Z]
+ return loss.min(dim=-1) # [C, K], [C, K]
+
+
+def _zp_expr_for(device_type: str):
+ """Pick the sweep layout for a device.
+
+ The last-dim-group reduction on CUDA (canonical Triton fusion shape),
+ the zero-point-last layout elsewhere (faster under TensorIterator and the
+ compiled CPU backend).
+ """
+ return _zp_expr_last if device_type == "cuda" else _zp_expr_mid
+
+
+# torch.compile'd zero-point sweeps, keyed by the expression (layout) -- lazily
+# created on first CUDA use / forced backend.
+_fused_zp_fns = {}
+# Set when a fused invocation raised (e.g. no host C++ compiler for Inductor);
+# the process then permanently uses the eager sweep.
+_fused_zp_broken = False
+# Set when the all-candidates batched sweep raised; the process then permanently
+# uses the per-candidate fused sweep.
+_batched_broken = False
+
+
+def _zp_wants_compile(device_type: str) -> bool:
+ """Whether a fused torch.compile sweep should be used, per ``AR_NEUQI_BACKEND``."""
+ if envs.AR_NEUQI_BACKEND == "eager":
+ return False
+ backend = envs.AR_NEUQI_BACKEND
+ if backend in ("compile", "triton"):
+ return True
+ if backend == "auto":
+ return device_type == "cuda"
+ return False # "eager" (and any unrecognized value): reference sweep
+
+
+# Extension Triton sweep (auto_round_extension.triton.neuqi_sweep), lazily
+# resolved once; ``None`` when the extension/triton is unavailable on the host.
+_triton_sweep = None
+_triton_checked = False
+_triton_broken = False
+_neuqi_shared_sweep = None
+_neuqi_shared_checked = False
+_neuqi_shared_broken = False
+
+
+def _triton_sweep_fn():
+ """Resolve the extension Triton sweep once; ``None`` when unavailable."""
+ global _triton_sweep, _triton_checked
+ if not _triton_checked:
+ _triton_checked = True
+ try:
+ from auto_round_extension.triton.neuqi_sweep import neuqi_sweep_triton
+
+ _triton_sweep = neuqi_sweep_triton
+ logger.debug("[NeUQI] Triton zero-point sweep engaged (auto_round_extension.triton.neuqi_sweep)")
+ except Exception as e:
+ logger.info("[NeUQI] Triton sweep unavailable (%s); using the torch.compile sweeps", e)
+ _triton_sweep = None
+ return _triton_sweep
+
+
+def _neuqi_shared_sweep_fn():
+ """Resolve the extension shared-multiplier coarse kernel once."""
+ global _neuqi_shared_sweep, _neuqi_shared_checked
+ if not _neuqi_shared_checked:
+ _neuqi_shared_checked = True
+ try:
+ from auto_round_extension.triton.neuqi_sweep import neuqi_shared_sweep_triton
+
+ _neuqi_shared_sweep = neuqi_shared_sweep_triton
+ logger.debug(
+ "[NeUQI] Triton shared-multiplier coarse sweep engaged (auto_round_extension.triton.neuqi_sweep)"
+ )
+ except Exception as e:
+ _neuqi_shared_sweep = None
+ logger.info("[NeUQI] shared coarse sweep unavailable (%s); using the batched path", e)
+ return _neuqi_shared_sweep
+
+
+def _neuqi_shared_attempt(dn, qw, fracs, invf, maxq):
+ """Try the shared-multiplier coarse kernel; ``None`` (and a latch) on failure."""
+ global _neuqi_shared_broken
+ if _neuqi_shared_broken or not dn.is_cuda:
+ return None
+ if not _zp_wants_triton(dn.device.type) or not _zp_batch_wanted(dn.device.type):
+ return None
+ fn = _neuqi_shared_sweep_fn()
+ if fn is None:
+ return None
+ try:
+ return fn(dn, qw, fracs, invf, maxq)
+ except Exception as e:
+ _neuqi_shared_broken = True
+ torch.cuda.empty_cache() # a failed launch may hold pooled memory
+ logger.warning("[NeUQI] shared coarse sweep failed (%s); using the batched path", e)
+ return None
+
+
+def _zp_coarse_pass_shared(data, qw, s0, coarse, maxq, launch=None):
+ """Coarse pass of the joint search in normalized space (shared-multiplier grid).
+
+ Computes dn = w / s0 once, evaluates every group against the SHARED frac
+ grid through ``launch`` (default: the gated Triton attempt), and
+ unnormalizes each winner's loss by s0^2 so the fine stage can compare
+ across stages. Returns (best_loss, best_frac, best_zp) or ``None`` when
+ the kernel is unavailable -- the caller then serves the identical
+ candidate grid through the batched / per-candidate paths.
+
+ ``launch`` is injectable so the driver math (normalization, s0^2
+ unnormalization, frac/zp mapping) stays unit-testable on CPU.
+ """
+ n_groups = data.shape[0]
+ if launch is None:
+ # pre-gate before any tensor math: CPU / non-triton backends must not
+ # pay for the normalized copy
+ if (
+ _neuqi_shared_broken
+ or not data.is_cuda
+ or not _zp_wants_triton(data.device.type)
+ or not _zp_batch_wanted(data.device.type)
+ ):
+ return None
+ launch = _neuqi_shared_attempt
+ dn = data / s0
+ invf = 1.0 / coarse
+ best_loss = torch.full((n_groups,), float("inf"), device=data.device, dtype=torch.float32)
+ best_frac = torch.ones(n_groups, device=data.device, dtype=torch.float32)
+ best_zp = torch.zeros(n_groups, device=data.device, dtype=torch.float32)
+ chunk = 262144
+ for start in range(0, n_groups, chunk):
+ stop = min(start + chunk, n_groups)
+ out = launch(
+ dn[start:stop].contiguous(),
+ qw[start:stop].contiguous() if qw is not None else None,
+ coarse,
+ invf,
+ maxq,
+ )
+ if out is None:
+ return None
+ loss_n, k_idx, zp = out
+ loss = loss_n * s0[start:stop].squeeze(-1) ** 2
+ frac = coarse.index_select(0, k_idx)
+ best_loss[start:stop] = loss
+ best_frac[start:stop] = frac
+ best_zp[start:stop] = zp.to(torch.float32)
+ return best_loss, best_frac, best_zp
+
+
+_sweep_warm_lock = threading.Lock()
+_sweep_warmed_devices: set = set() # device indices whose Triton sweep was warmed once
+
+
+def ensure_sweep_warmup(device) -> None:
+ """Serialized, synchronized first launch of the extension Triton sweep.
+
+ The first Triton launch JIT-compiles the kernel, initializes the launcher
+ and writes the on-disk cache; doing that while unrelated GPU work is in
+ flight (e.g. inductor compile workers active during SignRound wrapper
+ init, or several fan-out threads first-hitting the sweep at once) has been
+ observed to leave the sweep producing corrupt selections that surface as
+ out-of-bounds gather indices downstream (device-side asserts, vanishing
+ under CUDA_LAUNCH_BLOCKING=1). Warming once per device -- under a lock,
+ followed by a full device synchronize -- makes every subsequent launch a
+ cached, race-free launch. Failures latch the existing Triton-broken
+ fallback instead of raising.
+ """
+ if device.type != "cuda":
+ return
+ global _triton_broken, _neuqi_shared_broken
+ with _sweep_warm_lock:
+ if device.index in _sweep_warmed_devices:
+ return
+ _sweep_warmed_devices.add(device.index)
+ engaged = []
+ if _zp_wants_triton(device.type):
+ fn = _triton_sweep_fn()
+ if fn is not None:
+ try:
+ data = torch.randn(4, 128, device=device, dtype=torch.float32)
+ scales = torch.rand(4, 2, device=device, dtype=torch.float32) * 0.01 + 0.001
+ with torch.cuda.device(device): # Triton launches on the CURRENT device
+ loss, _zp = fn(data, torch.ones_like(data), scales, 15)
+ torch.cuda.synchronize(device)
+ if not torch.isfinite(loss).all():
+ raise RuntimeError("warmup produced non-finite losses")
+ logger.debug("[NeUQI] Triton sweep warmed on %s", device)
+ engaged.append("zero-point sweep")
+ except Exception as e: # noqa: BLE001 warmup failure must never kill a run
+ _triton_broken = True
+ logger.warning(
+ "[NeUQI] Triton sweep warmup failed on %s (%s); using torch.compile sweeps", device, e
+ )
+ if _zp_wants_triton(device.type) and _zp_batch_wanted(device.type):
+ fn = _neuqi_shared_sweep_fn()
+ if fn is not None:
+ try:
+ dn = torch.randn(4, 128, device=device, dtype=torch.float32)
+ fracs = torch.rand(8, device=device, dtype=torch.float32) * 0.5 + 0.5
+ with torch.cuda.device(device):
+ loss, _k, _z = fn(dn, torch.ones_like(dn), fracs, 1.0 / fracs, 15)
+ torch.cuda.synchronize(device)
+ if not torch.isfinite(loss).all():
+ raise RuntimeError("warmup produced non-finite losses")
+ logger.debug("[NeUQI] shared coarse sweep warmed on %s", device)
+ engaged.append("shared coarse sweep")
+ except Exception as e: # noqa: BLE001 warmup failure must never kill a run
+ _neuqi_shared_broken = True
+ logger.warning(
+ "[NeUQI] shared coarse sweep warmup failed on %s (%s); using the batched path", device, e
+ )
+
+ if engaged:
+ logger.info(
+ "[NeUQI] Triton backend engaged for the joint search on %s (kernels: %s)", device, ", ".join(engaged)
+ )
+
+
+def _zp_wants_triton(device_type: str) -> bool:
+ """Whether the extension Triton sweep should be tried, per ``AR_NEUQI_BACKEND``."""
+ if envs.AR_NEUQI_BACKEND == "triton":
+ return True # explicit user intent outranks the wrapper-init pin
+ if envs.AR_NEUQI_BACKEND in ("compile", "eager"):
+ return False
+ backend = envs.AR_NEUQI_BACKEND
+ if backend == "triton":
+ return True
+ return backend == "auto" and device_type == "cuda"
+
+
+def _get_fused_zp_fn(expr):
+ """Lazily compile one sweep layout; callers fall back to eager on any failure."""
+ fn = _fused_zp_fns.get(expr)
+ if fn is None:
+ from auto_round.utils.device import _bump_dynamo_cache_limit
+
+ # Shape variants (group size x bits x qw-present x chunk tail) must never
+ # trip dynamo's recompile limit: exceeding it silently runs the frame in
+ # eager, which is memory-safe here (chunking bounds the expansion) but slow.
+ _bump_dynamo_cache_limit(64)
+ fn = torch.compile(expr, dynamic=False)
+ _fused_zp_fns[expr] = fn
+ logger.info("[NeUQI] zero-point sweep fused via torch.compile (layout=%s)", expr.__name__)
+ return fn
+
+
+def _default_grid(device_type: str):
+ """Default (coarse, fine) grid for the search entries.
+
+ The wide 256/64 grid only pays off on the accelerated sweep lanes
+ (extension Triton kernels / torch.compile); the eager sweep evaluates
+ ~coarse x fine candidates per group and is ~8x slower at 256/64 than at
+ 64/32, while the measured quality ladder is flat between the two
+ (Qwen3.8-27B W4A16 g128: mean KL 0.02570 @ 64/32 vs 0.02579 @ 256/64).
+ Narrow the default whenever the backend policy does not select an
+ accelerated path; explicit ``AR_NEUQI_COARSE``/``AR_NEUQI_FINE`` pins
+ override this everywhere."""
+ fast = _zp_wants_compile(device_type) or _zp_wants_triton(device_type)
+ return (256, 64) if fast else (64, 32)
+
+
+def _eval_zp_chunk(data, qw, scale, zp_grid, maxq):
+ """Dispatch one chunk of the zero-point sweep to the fused or eager path."""
+ global _fused_zp_broken
+ expr = _zp_expr_for(data.device.type)
+ if not _fused_zp_broken and _zp_wants_compile(data.device.type):
+ try:
+ return _get_fused_zp_fn(expr)(data, qw, scale, zp_grid, maxq)
+ except Exception as e: # pragma: no cover - depends on host toolchain
+ _fused_zp_broken = True
+ logger.warning("[NeUQI] fused zero-point sweep failed (%s); using the eager sweep", e)
+ return expr(data, qw, scale, zp_grid, maxq)
+
+
+def _zp_batch_wanted(device_type: str) -> bool:
+ """Whether the all-candidates batched sweep should run.
+
+ Batches on CUDA when the fused backend engages there. The batched
+ expression is only ever executed compiled: eager execution of
+ [C, K, Z, g] would materialize the full expansion.
+ """
+ return device_type == "cuda" and _zp_wants_compile(device_type)
+
+
+def _eval_batched_chunk(data, qw, scales, zp_grid, maxq):
+ """One chunk of the all-candidates batched sweep, or ``None`` when unavailable.
+
+ Preference order: the extension Triton sweep (registers-resident, one data
+ read per element -- the fastest evaluator on CUDA), then the compiled
+ batched expression, then ``None`` so the caller falls back to the
+ per-candidate sweep. Strict-improvement bookkeeping makes a partial pass
+ followed by a fallback converge to the same selections (ties within the
+ usual fp32 summation tolerance).
+ """
+ global _triton_broken, _batched_broken
+ if not _triton_broken and _zp_wants_triton(data.device.type) and _zp_batch_wanted(data.device.type):
+ fn = _triton_sweep_fn()
+ if fn is not None:
+ try:
+ return fn(data, qw, scales, maxq)
+ except Exception as e:
+ _triton_broken = True
+ if data.device.type == "cuda":
+ torch.cuda.empty_cache()
+ logger.warning("[NeUQI] Triton sweep failed (%s); using the torch.compile sweeps", e)
+ if _batched_broken or not _zp_wants_compile(data.device.type):
+ return None
+ try:
+ return _get_fused_zp_fn(_zp_expr_batched)(data, qw, scales, zp_grid, maxq)
+ except Exception as e:
+ _batched_broken = True
+ if data.device.type == "cuda":
+ torch.cuda.empty_cache() # a failed launch may hold pooled memory
+ logger.warning("[NeUQI] batched zero-point sweep failed (%s); using the per-candidate sweep", e)
+ return None
+
+
+def _best_zp_for_scale(data, qw, scale, zp_grid, maxq, chunk):
+ """Evaluate all integer zero points for one per-group scale candidate.
+
+ Args:
+ data: [N, g] float32 group data.
+ qw: [N, g] float32 per-element weights or ``None``.
+ scale: [N, 1] float32 scale candidates.
+ zp_grid: [n_zp] float32 integral zero-point candidates.
+ maxq: maximum integer value (``2**bits - 1``).
+ chunk: number of groups per evaluation slice (bounds the temporary).
+
+ Returns:
+ best_loss: [N] float32, loss of the best zero point per group.
+ best_zp: [N] float32, integral-valued optimal zero point per group.
+ """
+ n_groups = data.shape[0]
+ best_loss = torch.full((n_groups,), float("inf"), device=data.device, dtype=data.dtype)
+ best_zp = torch.zeros(n_groups, device=data.device, dtype=data.dtype)
+
+ for start in range(0, n_groups, chunk):
+ stop = min(start + chunk, n_groups)
+ min_loss, argmin_zp = _eval_zp_chunk(
+ data[start:stop],
+ qw[start:stop] if qw is not None else None,
+ scale[start:stop],
+ zp_grid,
+ maxq,
+ )
+ best_loss[start:stop] = min_loss
+ best_zp[start:stop] = argmin_zp.to(data.dtype)
+
+ return best_loss, best_zp
+
+
+def _lower_scale_ratio(bits: int) -> float:
+ """Fraction of the min-max scale below which clipping is never beneficial."""
+ if bits >= 4:
+ return 0.25
+ if bits == 3:
+ return 0.15
+ return 0.05
+
+
[email protected] # loop-heavy eager search: never trace into dynamo graphs
+def neuqi_search_scale_zero(data, bits, qw=None, q_scale_thresh=1e-5, coarse_n=None, fine_n=None):
+ """Joint near-optimal (scale, integer zero-point) search per quantization group.
+
+ Args:
+ data: [N, g] float32 tensor of already-grouped weights.
+ bits: quantization bit width.
+ qw: optional [N, g] float32 per-element loss weights (e.g. imatrix).
+ q_scale_thresh: minimum scale magnitude.
+ coarse_n: number of coarse log-spaced scale candidates (env ``AR_NEUQI_COARSE``).
+ fine_n: number of fine additive candidates around the best coarse entry
+ (env ``AR_NEUQI_FINE``).
+
+ Returns:
+ scale: [N, 1] float32 per-group scales.
+ zp: [N, 1] float32 per-group integral-valued zero points.
+ """
+ if coarse_n is None or fine_n is None:
+ d_coarse, d_fine = _default_grid(data.device.type)
+ if coarse_n is None:
+ coarse_n = envs.AR_NEUQI_COARSE or d_coarse
+ if coarse_n < 2:
+ raise ValueError("AR_NEUQI_COARSE=%d is too small; need at least 2 candidates." % coarse_n)
+ if fine_n is None:
+ fine_n = envs.AR_NEUQI_FINE or d_fine
+ if fine_n < 2:
+ raise ValueError("AR_NEUQI_FINE=%d is too small; need at least 2 candidates." % fine_n)
+ _log_search_engaged(coarse_n, fine_n)
+ if torch.is_tensor(data):
+ ensure_sweep_warmup(data.device)
+
+ maxq = int(2**bits) - 1
+ data = data.to(torch.float32)
+ if qw is not None:
+ qw = _align_qw(qw, data).to(torch.float32)
+
+ zp_grid = torch.arange(0, maxq + 1, device=data.device, dtype=data.dtype)
+ chunk = max(1, _MAX_TMP_ELEMS // (data.shape[1] * (maxq + 1)))
+
+ wmin = torch.clamp(data.min(dim=-1).values, max=0).unsqueeze(-1)
+ wmax = torch.clamp(data.max(dim=-1).values, min=0).unsqueeze(-1)
+ s0 = ((wmax - wmin) / maxq).clamp_(min=q_scale_thresh) # [N, 1] min-max scale
+
+ lo = _lower_scale_ratio(bits)
+ coarse = _coarse_grid(lo, coarse_n, data.device, torch.float32)
+
+ best_loss = torch.full((data.shape[0],), float("inf"), device=data.device, dtype=torch.float32)
+ best_frac = torch.ones(data.shape[0], device=data.device, dtype=torch.float32)
+ best_zp = torch.zeros(data.shape[0], device=data.device, dtype=torch.float32)
+ n_groups = data.shape[0]
+ batch_wanted = _zp_batch_wanted(data.device.type)
+
+ # Coarse pass: shared log-spaced fractions of the per-group min-max scale.
+ # Fastest path: the shared-multiplier Triton kernel evaluates the whole
+ # grid in normalized space (dn = w / s0; one shared 1/frac multiply per
+ # candidate, argmin over (k, z) in-kernel). Batched fast path next: every
+ # candidate in one fused call per group chunk (the batched sweep); the
+ # per-candidate loop is the reference and the fallback.
+ shared_out = _zp_coarse_pass_shared(data, qw, s0, coarse, maxq)
+ coarse_done = False
+ if shared_out is not None:
+ best_loss, best_frac, best_zp = shared_out
+ coarse_done = True
+ elif batch_wanted:
+ chunk_b = max(1, _MAX_TMP_ELEMS // (coarse_n * (maxq + 1)))
+ for start in range(0, n_groups, chunk_b):
+ stop = min(start + chunk_b, n_groups)
+ scales = s0[start:stop] * coarse.view(1, -1) # [chunk, K] shared fracs
+ out = _eval_batched_chunk(
+ data[start:stop], qw[start:stop] if qw is not None else None, scales, zp_grid, maxq
+ )
+ if out is None:
+ break # latched failure / fused backend off: redo per candidate
+ loss_k, zp_k = out # [chunk, K] each
+ loss, k_idx = loss_k.min(dim=1) # first-min over candidates
+ zp = zp_k.gather(1, k_idx.unsqueeze(1)).squeeze(1)
+ frac = coarse.index_select(0, k_idx)
+ improved = loss < best_loss[start:stop]
+ best_loss[start:stop] = torch.where(improved, loss, best_loss[start:stop])
+ best_zp[start:stop] = torch.where(improved, zp, best_zp[start:stop])
+ best_frac[start:stop] = torch.where(improved, frac, best_frac[start:stop])
+ else:
+ coarse_done = True
+ if not coarse_done:
+ for idx in range(coarse_n):
+ scale = s0 * coarse[idx]
+ loss, zp = _best_zp_for_scale(data, qw, scale, zp_grid, maxq, chunk)
+ improved = loss < best_loss
+ best_loss = torch.where(improved, loss, best_loss)
+ best_zp = torch.where(improved, zp, best_zp)
+ best_frac = torch.where(improved, torch.full_like(best_frac, coarse[idx]), best_frac)
+
+ # Fine pass: additive grid between the coarse neighbors bracketing each winner.
+ best_idx = torch.argmin(torch.abs(coarse.unsqueeze(0) - best_frac.unsqueeze(1)), dim=1)
+ frac_lo = coarse[(best_idx - 1).clamp_(min=0)]
+ frac_hi = coarse[(best_idx + 1).clamp_(max=coarse_n - 1)]
+
+ steps = torch.arange(1, fine_n + 1, device=data.device, dtype=torch.float32) / (fine_n + 1)
+ fine_done = fine_n == 0
+ if batch_wanted and not fine_done:
+ # per-group fracs: frac_lo/hi differ per group, steps shared
+ one_m_steps = 1.0 - steps
+ chunk_b = max(1, _MAX_TMP_ELEMS // (fine_n * (maxq + 1)))
+ for start in range(0, n_groups, chunk_b):
+ stop = min(start + chunk_b, n_groups)
+ fracs = frac_lo[start:stop].unsqueeze(1) * one_m_steps + frac_hi[start:stop].unsqueeze(1) * steps
+ scales = s0[start:stop] * fracs # [chunk, K]
+ out = _eval_batched_chunk(
+ data[start:stop], qw[start:stop] if qw is not None else None, scales, zp_grid, maxq
+ )
+ if out is None:
+ break
+ loss_k, zp_k = out
+ loss, k_idx = loss_k.min(dim=1)
+ zp = zp_k.gather(1, k_idx.unsqueeze(1)).squeeze(1)
+ frac = fracs.gather(1, k_idx.unsqueeze(1)).squeeze(1)
+ improved = loss < best_loss[start:stop]
+ best_loss[start:stop] = torch.where(improved, loss, best_loss[start:stop])
+ best_zp[start:stop] = torch.where(improved, zp, best_zp[start:stop])
+ best_frac[start:stop] = torch.where(improved, frac, best_frac[start:stop])
+ else:
+ fine_done = True
+ if not fine_done:
+ for step in steps:
+ frac = frac_lo * (1.0 - step) + frac_hi * step
+ scale = s0 * frac.unsqueeze(-1)
+ loss, zp = _best_zp_for_scale(data, qw, scale, zp_grid, maxq, chunk)
+ improved = loss < best_loss
+ best_loss = torch.where(improved, loss, best_loss)
+ best_zp = torch.where(improved, zp, best_zp)
+ best_frac = torch.where(improved, frac, best_frac)
+
+ scale = (best_frac.unsqueeze(-1) * s0).clamp_(min=q_scale_thresh)
+ return scale, best_zp.unsqueeze(-1)
+
+
+_sym_engaged_logged: set = set()
+
+
[email protected] # logging side effect: graph-break here instead of tracing
+def _log_sym_search_engaged(coarse_n: int, fine_n: int) -> None:
+ """Once per unique grid: grid sweeps cycle configs in one process.
+
+ Set-based memo instead of ``lru_cache`` so dynamo (the search runs inside
+ compiled regions under ``--enable_torch_compile``) never has to reason
+ about a cache wrapper; the decorator makes the logging call graph-break.
+ """
+ if (coarse_n, fine_n) in _sym_engaged_logged:
+ return
+ _sym_engaged_logged.add((coarse_n, fine_n))
+ logger.info("[NeUQI] two-stage symmetric scale search active (coarse=%d, fine=%d)", coarse_n, fine_n)
+
+
+def _sym_loss_chunk(data, qw, scales, nmax):
+ """Batched symmetric-candidate weighted qdq losses for one chunk of groups.
+
+ Evaluates every per-group scale candidate in one vectorized call under BOTH
+ symmetric clamp conventions and keeps the better one per (group, candidate):
+
+ * standard: ``q = clamp(round(w / s), -nmax, nmax - 1)``, dequant ``s * q``
+ * mirrored: ``q = clamp(round(w / s), -(nmax - 1), nmax)``, dequant ``-s * q``
+
+ The mirrored family is the exact arithmetic of a NEGATIVE scale through the
+ standard formula (``round(w / -s)`` flips the rounding sign, ``clamp`` bounds
+ swap roles, ``-s * q`` restores magnitude). The incumbent
+ ``search_scales`` reaches it implicitly by anchoring on the signed max-abs
+ value, so groups whose largest-magnitude element is negative get a negative
+ scale -- for skewed groups the one-level clamp asymmetry makes the mirrored
+ grid notably cheaper (measured: the worst synthetic groups lose ~2x total
+ weighted loss when the family is unreachable). Searching ``|s|`` plus a sign
+ per group covers both families explicitly.
+
+ Plain eager math (the reference path): with a single zero-point-free axis
+ there is no (candidate, zero-point) expansion, so the chunk is a handful of
+ elementwise kernels. Wide grids make the chunking fine-grained enough to
+ become launch-bound; ``_sym_loss_chunk_eval`` routes these calls through a
+ torch.compile-fused core while this function stays the eager reference.
+
+ Args:
+ data: [chunk, g] float32 group data.
+ qw: [chunk, g] float32 per-element weights or ``None``.
+ scales: [chunk, K] float32 POSITIVE per-group scale candidates.
+ nmax: symmetric integer bound (``2 ** (bits - 1)``).
+
+ Returns:
+ (best_loss [chunk], best_idx [chunk], best_mirror [chunk] bool) with the
+ first-minimum tie rule of ``Tensor.min`` over candidates (matching the
+ asymmetric sweep's tie semantics); ``best_mirror`` marks whether the
+ winning candidate used the mirrored convention (negative scale).
+ """
+ d = data.unsqueeze(1) # [C, 1, g]
+ sc = scales.unsqueeze(-1) # [C, K, 1]
+ r = torch.round(d / sc)
+ q_std = r.clamp(min=-nmax, max=nmax - 1)
+ e_std = sc * q_std - d
+ l_std = e_std * e_std
+ q_mir = r.clamp(min=-(nmax - 1), max=nmax)
+ e_mir = sc * q_mir - d # dequant of the mirrored family is sc * q_mir (neg scale flipped inside q_mir)
+ l_mir = e_mir * e_mir
+ if qw is not None:
+ w = qw.unsqueeze(1)
+ l_std = l_std * w
+ l_mir = l_mir * w
+ l_std = l_std.sum(dim=-1) # [C, K]
+ l_mir = l_mir.sum(dim=-1) # [C, K]
+ mirror = l_mir < l_std
+ loss = torch.where(mirror, l_mir, l_std)
+ best_loss, best_idx = loss.min(dim=-1)
+ best_mirror = mirror.gather(1, best_idx.unsqueeze(1)).squeeze(1)
+ return best_loss, best_idx, best_mirror
+
+
+# Symmetric-search Triton backend (auto_round_extension.triton.neuqi_sweep), lazily
+# resolved once; ``None`` when the extension/triton is unavailable on the host.
+_sym_triton = None
+_sym_triton_checked = False
+_sym_triton_broken = False
+_sym_shared_triton = None
+_sym_shared_checked = False
+_sym_shared_broken = False
+_sym_search_warmed: set = set()
+
+
+def _sym_shared_triton_fn():
+ """Resolve the extension shared-multiplier coarse kernel once."""
+ global _sym_shared_triton, _sym_shared_checked
+ if not _sym_shared_checked:
+ _sym_shared_checked = True
+ try:
+ from auto_round_extension.triton.neuqi_sweep import sym_search_shared_triton
+
+ _sym_shared_triton = sym_search_shared_triton
+ logger.debug(
+ "[NeUQI] Triton shared-multiplier sym coarse search engaged "
+ "(auto_round_extension.triton.neuqi_sweep)"
+ )
+ except Exception as e:
+ _sym_shared_triton = None
+ logger.info("[NeUQI] shared sym search unavailable (%s); using the per-candidate path", e)
+ return _sym_shared_triton
+
+
+def _sym_shared_wants_triton(device_type: str) -> bool:
+ """Whether the shared-multiplier coarse kernel should run, per backend rules.
+
+ AUTO resolves the COARSE stage to this dedicated kernel: every group
+ shares the same candidate fracs, so normalizing once (dn = w / s0) turns
+ the inner loop into one shared 1/frac multiply -- no per-(group, candidate)
+ scales materialization and no division in the loop. Measured RTX 3090,
+ 2M groups, g=128, 256+64 candidates: 0.24 s vs 0.57-0.79 s for the
+ compiled core (eager 8.5 s); parity stays in the tie-flip class (~0.5%
+ flipped groups, max fp64 rel diff 1.7e-6 vs eager). The FINE stage keeps
+ its per-group candidate grid and stays on the compiled core under AUTO.
+ ``compile``/``eager`` overrides keep the reference numerics end to end.
+ """
+ backend = str(envs.AR_NEUQI_BACKEND or "auto")
+ if backend in ("compile", "eager"):
+ return False
+ return backend in ("auto", "triton") and device_type == "cuda"
+
+
+def _sym_shared_triton_attempt(dn, qw, fracs, invf, nmax):
+ """Try the shared-multiplier coarse kernel; ``None`` (and a latch) on failure."""
+ global _sym_shared_broken
+ if _sym_shared_broken or not dn.is_cuda or not _sym_shared_wants_triton(dn.device.type):
+ return None
+ fn = _sym_shared_triton_fn()
+ if fn is None:
+ return None
+ try:
+ return fn(dn, qw, fracs, invf, nmax)
+ except Exception as e:
+ _sym_shared_broken = True
+ torch.cuda.empty_cache() # a failed launch may hold pooled memory
+ logger.warning("[NeUQI] shared sym coarse search failed (%s); using the per-candidate path", e)
+ return None
+
+
+def _sym_triton_fn():
+ """Resolve the extension symmetric-search Triton kernel once."""
+ global _sym_triton, _sym_triton_checked
+ if not _sym_triton_checked:
+ _sym_triton_checked = True
+ try:
+ from auto_round_extension.triton.neuqi_sweep import sym_search_triton
+
+ _sym_triton = sym_search_triton
+ logger.debug("[NeUQI] Triton symmetric scale search engaged (auto_round_extension.triton.neuqi_sweep)")
+ except Exception as e:
+ _sym_triton = None
+ logger.info("[NeUQI] Triton sym search unavailable (%s); using the compiled core", e)
+ return _sym_triton
+
+
+def _sym_wants_triton(device_type: str) -> bool:
+ """Whether the extension Triton sym search should be tried, per backend rules.
+
+ AUTO resolves to the compiled core, not the Triton kernel: the sym chunk
+ keeps the whole [C, g] data tile L2-resident (chunking bounds the [C, K, g]
+ expansion), so the compiled core's wide-vector candidate re-reads from L2
+ beat the registers-resident kernel's serialized candidate loop -- measured
+ RTX 3090, 2M groups, g=128, 256+64 candidates: compiled 0.72 s vs Triton
+ 1.35 s (eager 8.53 s). Parity is identical either way (same ~0.5% tie
+ flips, max fp64 rel diff 1.7e-6 vs eager). The kernel stays available
+ under AR_NEUQI_BACKEND=triton and is the tuning base if a shape regime
+ where it wins shows up.
+ """
+ backend = str(envs.AR_NEUQI_BACKEND or "auto")
+ return backend == "triton"
+
+
+def _sym_triton_attempt(data, qw, scales, nmax):
+ """Try the Triton sym search once; ``None`` (and a permanent latch) on failure."""
+ global _sym_triton_broken
+ if _sym_triton_broken or not data.is_cuda or not _sym_wants_triton(data.device.type):
+ return None
+ fn = _sym_triton_fn()
+ if fn is None:
+ return None
+ try:
+ return fn(data, qw, scales, nmax)
+ except Exception as e:
+ _sym_triton_broken = True
+ torch.cuda.empty_cache() # a failed launch may hold pooled memory
+ logger.warning("[NeUQI] Triton sym search failed (%s); using the compiled core", e)
+ return None
+
+
+def ensure_sym_search_warmup(device) -> None:
+ """Serialized, synchronized first launch of the sym search kernel.
+
+ Same rationale as :func:`ensure_sweep_warmup`: the first launch JIT-compiles
+ the kernel and writes the on-disk cache; doing that concurrently with other
+ GPU work has been observed to corrupt selections. Failures latch the
+ Triton fallback instead of raising.
+ """
+ if device.type != "cuda":
+ return
+ global _sym_triton_broken, _sym_shared_broken
+ with _sweep_warm_lock:
+ if device.index in _sym_search_warmed:
+ return
+ _sym_search_warmed.add(device.index)
+ engaged = []
+ if _sym_wants_triton(device.type):
+ fn = _sym_triton_fn()
+ if fn is not None:
+ try:
+ data = torch.randn(4, 128, device=device, dtype=torch.float32)
+ scales = torch.rand(4, 2, device=device, dtype=torch.float32) * 0.01 + 0.001
+ loss, _idx, _mir = fn(data, torch.ones_like(data), scales, 7)
+ torch.cuda.synchronize(device)
+ if not torch.isfinite(loss).all():
+ raise RuntimeError("warmup produced non-finite losses")
+ logger.debug("[NeUQI] Triton sym search warmed on %s", device)
+ engaged.append("symmetric scale sweep")
+ except Exception as e: # noqa: BLE001 warmup failure must never kill a run
+ _sym_triton_broken = True
+ logger.warning(
+ "[NeUQI] Triton sym search warmup failed on %s (%s); using the compiled core", device, e
+ )
+ if _sym_shared_wants_triton(device.type):
+ fn = _sym_shared_triton_fn()
+ if fn is not None:
+ try:
+ dn = torch.randn(4, 128, device=device, dtype=torch.float32)
+ fracs = torch.rand(8, device=device, dtype=torch.float32) * 0.5 + 0.5
+ loss, _idx, _mir = fn(dn, torch.ones_like(dn), fracs, 1.0 / fracs, 7)
+ torch.cuda.synchronize(device)
+ if not torch.isfinite(loss).all():
+ raise RuntimeError("warmup produced non-finite losses")
+ logger.debug("[NeUQI] shared sym coarse search warmed on %s", device)
+ engaged.append("shared-multiplier coarse sweep")
+ except Exception as e: # noqa: BLE001 warmup failure must never kill a run
+ _sym_shared_broken = True
+ logger.warning(
+ "[NeUQI] shared sym coarse search warmup failed on %s (%s); using the per-candidate path",
+ device,
+ e,
+ )
+
+ if engaged:
+ logger.info(
+ "[NeUQI] Triton backend engaged for the symmetric search on %s (kernels: %s)",
+ device,
+ ", ".join(engaged),
+ )
+
+
+def _sym_search_eval(data, qw, scales, nmax):
+ """Best evaluator for one chunk: Triton (registers-resident) -> compiled core -> eager."""
+ out = _sym_triton_attempt(data, qw, scales, nmax)
+ if out is not None:
+ return out
+ return _sym_loss_chunk_eval(data, qw, scales, nmax)
+
+
+# Compiled core for the symmetric search chunk: each eager chunk is ~20 small
+# kernel launches and the chunk size shrinks with the candidate count
+# (_MAX_TMP_ELEMS // (2 * K * g)), so wide grids (the union variant at
+# coarse=256 reaches K ~ 440) make big tensors pure launch overhead. One fused
+# inductor kernel per chunk is the symmetric analogue of the asym sweep's
+# compiled stage. The dedicated Triton kernels cover the stages where they
+# measured faster: the shared-multiplier kernel serves the plain coarse pass
+# under AUTO, the per-candidate kernel is AR_NEUQI_BACKEND=triton opt-in; this
+# compiled core stays the AUTO choice for the fine stage (per-group grids).
+_sym_chunk_compiled = None
+_sym_compile_failed = False
+_sym_compile_logged: set = set()
+
+
[email protected] # logging side effect: graph-break instead of tracing
+def _log_sym_compile_engaged(backend: str) -> None:
+ if backend in _sym_compile_logged:
+ return
+ _sym_compile_logged.add(backend)
+ logger.info("[NeUQI] sym search chunk core compiled (backend=%s)", backend)
+
+
+def _sym_loss_chunk_eval(data, qw, scales, nmax):
+ """``_sym_loss_chunk`` through a lazily compiled core, latching down to eager.
+
+ Called only from inside ``torch.compiler.disable``-decorated search
+ functions, so the explicit compile unit is the sole dynamo boundary and
+ implicit tracing never reaches it. Any failure latches down permanently:
+ retrying per chunk would repeat the failure cost every chunk.
+ """
+ global _sym_chunk_compiled, _sym_compile_failed
+ backend = str(envs.AR_NEUQI_BACKEND or "auto")
+ if _sym_compile_failed or backend not in ("auto", "compile", "triton"):
+ return _sym_loss_chunk(data, qw, scales, nmax)
+ if _sym_chunk_compiled is None:
+ try:
+ _sym_chunk_compiled = torch.compile(_sym_loss_chunk)
+ except Exception: # pragma: no cover - compilation unavailable in this environment
+ _sym_compile_failed = True
+ return _sym_loss_chunk(data, qw, scales, nmax)
+ _log_sym_compile_engaged(backend)
+ try:
+ return _sym_chunk_compiled(data, qw, scales, nmax)
+ except Exception:
+ _sym_chunk_compiled = None
+ _sym_compile_failed = True
+ logger.warning("[NeUQI] sym compiled chunk core failed; using eager for the rest of this process.")
+ return _sym_loss_chunk(data, qw, scales, nmax)
+
+
+# Fraction of the no-clip max-abs scale above which coarser grids are never
+# competitive: beyond the no-clip scale the step only coarsens with no clipping
+# benefit, and the slack region far above it is dominated.
+_SYM_UPPER_SCALE_RATIO = 2.0
+
+
+def _sym_coarse_pass_shared(data, qw, s0, coarse, nmax, launch=None):
+ """Normalized-space coarse pass through the shared-multiplier kernel.
+
+ Computes dn = w / s0 once, evaluates every group against the SHARED frac
+ grid through ``launch`` (default: the gated Triton attempt), and
+ unnormalizes each winner's loss by s0^2 so the fine stage can compare
+ across stages. Returns (best_loss, best_frac, best_mirror) or ``None``
+ when the kernel is unavailable anywhere -- the caller then serves the
+ identical candidate grid through the per-candidate evaluator.
+
+ ``launch`` is injectable so the driver math (normalization, s0^2
+ unnormalization, frac mapping) stays unit-testable on CPU.
+ """
+ n_groups = data.shape[0]
+ if launch is None:
+ # pre-gate before any tensor math: CPU / non-shared backends must not
+ # pay for the normalized copy
+ if _sym_shared_broken or not data.is_cuda or not _sym_shared_wants_triton(data.device.type):
+ return None
+ launch = _sym_shared_triton_attempt
+ dn = data / s0
+ invf = 1.0 / coarse
+ best_loss = torch.full((n_groups,), float("inf"), device=data.device, dtype=torch.float32)
+ best_frac = torch.ones(n_groups, device=data.device, dtype=torch.float32)
+ best_mirror = torch.zeros(n_groups, device=data.device, dtype=torch.bool)
+ chunk = 262144
+ for start_idx in range(0, n_groups, chunk):
+ stop = min(start_idx + chunk, n_groups)
+ out = launch(
+ dn[start_idx:stop].contiguous(),
+ qw[start_idx:stop].contiguous() if qw is not None else None,
+ coarse,
+ invf,
+ nmax,
+ )
+ if out is None:
+ return None
+ loss, idx, mirror = out
+ best_loss[start_idx:stop] = loss * s0[start_idx:stop].squeeze(-1) ** 2
+ best_frac[start_idx:stop] = coarse.index_select(0, idx)
+ best_mirror[start_idx:stop] = mirror
+ return best_loss, best_frac, best_mirror
+
+
[email protected] # loop-heavy eager search: never trace into dynamo graphs
+def _two_stage_sym_core(data, qw, bits, coarse, fine_n, q_scale_thresh):
+ """Coarse+fine scale search core over a given (sorted, positive) coarse grid.
+
+ Runs the batched coarse evaluation, then the additive fine refinement
+ bracketed by the value-neighbors of each group's winning coarse entry,
+ under both clamp conventions (see ``_sym_loss_chunk``). Strict-improvement
+ updates only, so the result never worsens versus any evaluated candidate.
+
+ Returns:
+ (signed per-group scale [N, 1], per-group best weighted loss [N]).
+ """
+ nmax = int(2 ** (bits - 1))
+ g = data.shape[1]
+ n_groups = data.shape[0]
+ s0 = (torch.abs(data).amax(dim=-1, keepdim=True) / nmax).clamp_(min=q_scale_thresh) # [N, 1]
+ coarse_n = coarse.numel()
+
+ best_loss = torch.full((n_groups,), float("inf"), device=data.device, dtype=torch.float32)
+ best_frac = torch.ones(n_groups, device=data.device, dtype=torch.float32)
+ best_mirror = torch.zeros(n_groups, device=data.device, dtype=torch.bool)
+
+ # Coarse pass: shared fractions of the per-group no-clip scale. Under AUTO
+ # on CUDA this runs in normalized space through the dedicated
+ # shared-multiplier Triton kernel (dn = w / s0 once; every group multiplies
+ # by the same 1/frac scalars -- no [C, K] scales, no division in the loop);
+ # the winner loss is unnormalized by s0^2 for the cross-stage compare.
+ # Anywhere else the per-candidate evaluator serves the identical grid.
+ shared_out = _sym_coarse_pass_shared(data, qw, s0, coarse, nmax)
+ if shared_out is not None:
+ best_loss, best_frac, best_mirror = shared_out
+ else:
+ chunk = max(1, _MAX_TMP_ELEMS // (2 * coarse_n * g))
+ for start_idx in range(0, n_groups, chunk):
+ stop = min(start_idx + chunk, n_groups)
+ scales = s0[start_idx:stop] * coarse.view(1, -1) # [C, K]
+ loss, idx, mirror = _sym_search_eval(
+ data[start_idx:stop], qw[start_idx:stop] if qw is not None else None, scales, nmax
+ )
+ best_loss[start_idx:stop] = loss
+ best_frac[start_idx:stop] = coarse.index_select(0, idx)
+ best_mirror[start_idx:stop] = mirror
+
+ # Fine pass: additive grid between the coarse neighbors bracketing each winner.
+ best_idx = torch.argmin(torch.abs(coarse.unsqueeze(0) - best_frac.unsqueeze(1)), dim=1)
+ frac_lo = coarse[(best_idx - 1).clamp_(min=0)]
+ frac_hi = coarse[(best_idx + 1).clamp_(max=coarse_n - 1)]
+ if fine_n:
+ steps = torch.arange(1, fine_n + 1, device=data.device, dtype=torch.float32) / (fine_n + 1)
+ one_m_steps = 1.0 - steps
+ chunk = max(1, _MAX_TMP_ELEMS // (2 * fine_n * g))
+ for start_idx in range(0, n_groups, chunk):
+ stop = min(start_idx + chunk, n_groups)
+ fracs = frac_lo[start_idx:stop].unsqueeze(1) * one_m_steps + frac_hi[start_idx:stop].unsqueeze(1) * steps
+ scales = s0[start_idx:stop] * fracs # [C, F]
+ loss, k, mirror = _sym_search_eval(
+ data[start_idx:stop], qw[start_idx:stop] if qw is not None else None, scales, nmax
+ )
+ frac = fracs.gather(1, k.unsqueeze(1)).squeeze(1)
+ improved = loss < best_loss[start_idx:stop]
+ best_loss[start_idx:stop] = torch.where(improved, loss, best_loss[start_idx:stop])
+ best_frac[start_idx:stop] = torch.where(improved, frac, best_frac[start_idx:stop])
+ best_mirror[start_idx:stop] = torch.where(improved, mirror, best_mirror[start_idx:stop])
+
+ scale = best_frac.unsqueeze(-1) * s0
+ scale = torch.where(best_mirror.unsqueeze(-1), -scale, scale)
+ # magnitude floor on both signs (the incumbent keeps the sign and clamps
+ # the magnitude), so a near-zero winner never produces a degenerate scale
+ scale = torch.where(scale < 0, torch.clamp(scale, max=-q_scale_thresh), torch.clamp(scale, min=q_scale_thresh))
+ return scale, best_loss
+
+
[email protected] # loop-heavy eager search: never trace into dynamo graphs
+def neuqi_search_scale_sym(data, bits, qw=None, q_scale_thresh=1e-5, coarse_n=None, fine_n=None):
+ """Two-stage (coarse log-spaced + fine additive) scale search, symmetric.
+
+ Symmetric counterpart of :func:`neuqi_search_scale_zero`: the zero point is
+ fixed at 0 and the integer range is clamped to
+ ``[-2**(bits-1), 2**(bits-1) - 1]``. The grid is anchored at the per-group
+ no-clip scale ``s0 = max|w| / nmax`` and searches log-spaced fractions in
+ ``[_lower_scale_ratio(bits), _SYM_UPPER_SCALE_RATIO]`` followed by an
+ additive refinement between the coarse neighbors bracketing each winner --
+ the same two-stage scheme as the asymmetric search, applied to the
+ zero-point-free slice. Both clamp conventions of the one-level-asymmetric
+ symmetric grid are searched per group (negative scales, the family the
+ incumbent reaches implicitly via its signed anchor; see
+ ``_sym_loss_chunk``), and the returned scale carries the winning sign.
+
+ Args:
+ data: [N, g] float32 tensor of already-grouped weights.
+ bits: quantization bit width.
+ qw: optional [N, g] float32 per-element loss weights (e.g. imatrix).
+ q_scale_thresh: minimum scale magnitude.
+ coarse_n: number of coarse log-spaced candidates (env
+ ``AR_NEUQI_COARSE``; default 256).
+ fine_n: number of fine additive candidates (env
+ ``AR_NEUQI_FINE``; default 64).
+
+ Returns:
+ scale: [N, 1] float32 per-group signed scales (negative = mirrored clamp
+ convention won for that group).
+ """
+ if coarse_n is None or fine_n is None:
+ d_coarse, d_fine = _default_grid(data.device.type)
+ if coarse_n is None:
+ # shared AR_NEUQI_COARSE/FINE pins apply (see envs.py); the unpinned
+ # default is backend-aware (wide only on the accelerated lanes)
+ coarse_n = envs.AR_NEUQI_COARSE or d_coarse
+ if coarse_n < 2:
+ raise ValueError("AR_NEUQI_COARSE=%d is too small; need at least 2 candidates." % coarse_n)
+ if fine_n is None:
+ fine_n = envs.AR_NEUQI_FINE or d_fine
+ if fine_n < 2:
+ raise ValueError("AR_NEUQI_FINE=%d is too small; need at least 2 candidates." % fine_n)
+ _log_sym_search_engaged(coarse_n, fine_n)
+ ensure_sym_search_warmup(data.device)
+
+ data = data.to(torch.float32)
+ if qw is not None:
+ qw = _align_qw(qw, data).to(torch.float32)
+
+ log_grid = _coarse_grid(_lower_scale_ratio(bits), coarse_n, data.device, torch.float32, hi=_SYM_UPPER_SCALE_RATIO)
+ return _two_stage_sym_core(data, qw, bits, log_grid, fine_n, q_scale_thresh)[0]
+
+
+@register_dtype("opt_rtn_int_sym_neuqi")
+def quant_tensor_opt_rtn_sym_neuqi(
+ tensor, bits=4, group_size=-1, v=0, q_scale_thresh=1e-5, imatrix=None, scale_dtype=torch.float16, **kwargs
+):
+ """Quantize/dequantize with the two-stage symmetric scale search (NeUQI grid).
+
+ Symmetric sibling of ``opt_rtn_int_asym``: the zero point is fixed at 0 and
+ the joint search collapses to the scale axis, searched on the NeUQI
+ coarse+fine grid instead of ``opt_rtn_int_sym``'s uniform grid. Returns
+ follow the ``quant_tensor_opt_rtn_sym`` conventions (zero point = ``nmax``).
+ ``v`` is accepted and ignored, matching the incumbent symmetric path.
+
+ Args:
+ tensor: Tensor to quantize.
+ bits: Number of bits for the quantization (e.g., 2, 3, 4, 8).
+ group_size: Number of elements sharing a scale.
+ v: Rounding value perturbation (ignored, matching ``opt_rtn_int_sym``).
+ q_scale_thresh: Minimum scale magnitude for numerical stability.
+ imatrix: Optional per-column (1-D) or per-row (2-D, stacked modules)
+ importance weights (activation imatrix).
+ scale_dtype: dtype of the returned scale (kernels support fp16/fp32).
+
+ Returns:
+ (qdq_result, scale, zero_point) matching ``quant_tensor_opt_rtn_sym`` conventions.
+ """
+ from auto_round.data_type.gguf import _imatrix_handle_zero
+
+ tensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)
+ nmax = int(2 ** (bits - 1))
+
+ qw = None
+ if imatrix is not None:
+ imatrix = _align_qw(imatrix, tensor)
+ if imatrix.dim() == 1:
+ qw = imatrix.reshape(1, -1)
+ qw = reshape_pad_tensor_by_group_size(qw, group_size, val=1e-5)[0].view(1, -1)
+ qw = qw.expand(tensor.numel() // qw.numel(), -1)
+ qw = qw.reshape(tensor.shape)
+ else:
+ qw = reshape_pad_tensor_by_group_size(imatrix, group_size, val=1e-5)[0]
+ if qw.shape != tensor.shape:
+ raise ValueError(
+ f"per-row imatrix shape {tuple(imatrix.shape)} incompatible with tensor {tuple(tensor.shape)}"
+ )
+ qw = _imatrix_handle_zero(qw, tensor, bits, group_size)
+
+ scale = neuqi_search_scale_sym(tensor.to(torch.float32), bits, qw=qw, q_scale_thresh=q_scale_thresh)
+ scale = scale.to(scale_dtype)
+ # sign-preserving magnitude floor, mirroring quant_tensor_opt_rtn_sym
+ scale = torch.where(scale < 0, torch.clamp(scale, max=-q_scale_thresh), torch.clamp(scale, min=q_scale_thresh))
+
+ int_w = tensor.div(scale).round_().clamp_(-nmax, nmax - 1)
+ qdq_result = (int_w.mul_(scale)).to(tensor.dtype)
+ qdq_result = revert_tensor_by_pad(qdq_result, orig_shape=orig_shape, pad_len=pad_len)
+ return qdq_result, scale, nmax
+
+
+@register_dtype("opt_rtn_int_asym")
+def quant_tensor_opt_rtn_asym(
+ tensor, bits=4, group_size=-1, v=0, q_scale_thresh=1e-5, imatrix=None, scale_dtype=torch.float16, **kwargs
+):
+ """Quantize/dequantize with a joint near-optimal (scale, integer zero-point) search.
+
+ Asymmetric counterpart of ``opt_rtn_int_sym``: fills the previously missing
+ optimized path for asymmetric int quantization (plain min/max before). Set
+ the per-group scale and integer zero point by a two-stage coarse+fine grid
+ search instead of the plain min/max initialization (arXiv 2505.17595).
+
+ Args:
+ tensor: Tensor to quantize.
+ bits: Number of bits for quantization (e.g., 2, 3, 4, 8).
+ group_size: Number of elements sharing a scale.
+ v: Rounding value perturbation.
+ q_scale_thresh: Minimum scale magnitude for numerical stability.
+ imatrix: Optional per-column importance weights (activation imatrix).
+ scale_dtype: dtype of the returned scale (kernels support fp16/fp32).
+
+ Returns:
+ (qdq_result, scale, zero_point) matching ``quant_tensor_asym`` conventions.
+ """
+ from auto_round.data_type.gguf import _imatrix_handle_zero
+
+ tensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)
+ maxq = int(2**bits) - 1
+
+ qw = None
+ if imatrix is not None:
+ imatrix = _align_qw(imatrix, tensor)
+ if imatrix.dim() == 1:
+ # per-column importance shared by every row of this tensor
+ qw = imatrix.reshape(1, -1)
+ qw = reshape_pad_tensor_by_group_size(qw, group_size, val=1e-5)[0].view(1, -1)
+ qw = qw.expand(tensor.numel() // qw.numel(), -1)
+ qw = qw.reshape(tensor.shape)
+ else:
+ # per-row importance (stacked same-shape modules: each module keeps
+ # its own column weights); must already match the tensor's shape
+ qw = reshape_pad_tensor_by_group_size(imatrix, group_size, val=1e-5)[0]
+ if qw.shape != tensor.shape:
+ raise ValueError(
+ f"per-row imatrix shape {tuple(imatrix.shape)} incompatible with tensor {tuple(tensor.shape)}"
+ )
+ qw = _imatrix_handle_zero(qw, tensor, bits, group_size)
+
+ scale, zp = neuqi_search_scale_zero(tensor.to(torch.float32), bits, qw=qw, q_scale_thresh=q_scale_thresh)
+ scale = torch.clamp(scale.to(scale_dtype), min=q_scale_thresh)
+ zp = zp.to(scale_dtype)
+
+ int_w = round_ste(tensor / scale + v)
+ q = torch.clamp(int_w + zp, 0, maxq)
+ qdq_result = (scale * (q - zp)).to(tensor.dtype)
+ qdq_result = revert_tensor_by_pad(qdq_result, orig_shape=orig_shape, pad_len=pad_len)
+ return qdq_result, scale, zp
+
+
+logger.debug("[NeUQI] opt_rtn_int_asym registered (clean-room, arXiv 2505.17595)")
+logger.debug("[NeUQI] opt_rtn_int_sym_neuqi registered (two-stage sym scale search)")
+
+
+def neuqi_sync_devices(device):
+ """Join in-flight GPU work before a search launches.
+
+ The data-driven pipeline's input-caching pass moves activations across
+ GPUs on side streams (accelerate). A not-yet-joined producer makes the
+ search read stale tensors -- an intermittent device-side assert that
+ disappears under CUDA_LAUNCH_BLOCKING=1. One sync per layer-init is noise
+ next to the search cost.
+
+ Syncs EVERY visible device, not just this layer's: under accelerate
+ dispatch the cache-inputs forward moves activations GPU->GPU, and an
+ un-joined peer copy recorded on ANOTHER device's stream can still be
+ reading memory the search is about to reuse (allocator reuse -> the
+ intermittent OOB-gather asserts that vanish under blocking mode).
+ """
+ if torch.device(device).type == "cuda":
+ for dev_idx in range(torch.cuda.device_count()):
+ torch.cuda.synchronize(dev_idx)
+
+
+def neuqi_anchor_margins(weight, bits, group_size, imatrix, device, sym=False):
+ """Searched (scale, zp) grid for the ``enable_neuqi`` init, expressed as the
+ per-group (tensor_min, tensor_max) pair the STE quant functions re-derive
+ the grid from.
+
+ Asym anchoring (quant_tensor_asym: scale=(wmax-wmin)/maxq,
+ zp=round(-wmin/scale)): min=-zp*s, max=(maxq-zp)*s reproduces (s, zp)
+ exactly. Sym anchoring (quant_tensor_sym: scale=max(|wmax|,|wmin|)/maxq,
+ signed): min=-s*maxq, max=+s*maxq.
+
+ Both symmetry classes anchor: asym layers through the joint
+ (scale, zero-point) search, sym layers through the two-stage symmetric
+ scale search. The sym search returns a SIGNED scale whose sign cancels in
+ qdq reconstruction (the mirrored convention is the arithmetic of a
+ negative scale through the standard formula), so anchoring the magnitude
+ reproduces the winning qdq grid exactly.
+
+ On the tuning path the grid is FROZEN: SignRound tunes the rounding values
+ against it, and the range margins stay pinned at 1.0 (the search's grid IS
+ the init). On the zero-shot asymmetric path the same margins make the
+ plain asym quant reproduce the searched (scale, zp) exactly.
+
+ Returns (wmin, wmax), each squeezed over the trailing group axis.
+ """
+ import time
+
+ t0 = time.perf_counter()
+ w = weight.to(device=device, dtype=torch.float32)
+ qw = None
+ if imatrix is not None:
+ qw = _prepare_neuqi_qw(imatrix, w, bits, group_size).to(device)
+ neuqi_sync_devices(device)
+
+ if sym:
+ # two-stage symmetric scale search; the returned scale is SIGNED
+ # (negative = mirrored clamp convention won). quant_tensor_sym derives
+ # scale = (2*(wmax_abs < wmin_abs) - 1) * max(wmax_abs, wmin_abs)/maxq,
+ # so an exact magnitude tie (and wmax_abs >= wmin_abs generally)
+ # resolves to the NEGATIVE scale. Bias the losing side down to HALF
+ # magnitude so the comparison resolves to the search winner's sign in
+ # ANY storage dtype (a 2^-20 bias would be erased by bf16/fp16
+ # rounding when the anchor is cast to the weight dtype), while max()
+ # still returns the unperturbed magnitude -- the derived scale is the
+ # searched |s| exactly (the halved side only ever participates in the
+ # comparison; margins are pinned at 1.0 so nothing else reads it).
+ s_signed = neuqi_search_scale_sym(w, bits, qw=qw)
+ nmax = int(2 ** (bits - 1))
+ mag = s_signed.abs() * nmax
+ mirrored = s_signed < 0
+ # standard winner (s > 0) needs wmax_abs strictly smaller: scale positive
+ wmin = torch.where(mirrored, -mag / 2, -mag)
+ wmax = torch.where(mirrored, mag, mag / 2)
+ else:
+ s, zp = neuqi_search_scale_zero(w, bits, qw=qw)
+ maxq = int(2**bits) - 1
+ wmin = -(zp * s)
+ wmax = (maxq - zp) * s
+ global _ANCHOR_LOGGED
+ if not _ANCHOR_LOGGED:
+ _ANCHOR_LOGGED = True
+ logger.info(
+ "[NeUQI] init-search anchored the quantization grid (%.2fs first layer; margins pinned at 1.0)",
+ time.perf_counter() - t0,
+ )
+ return wmin.squeeze(-1), wmax.squeeze(-1)
+
+
+def _prepare_neuqi_qw(imatrix, weight_reshape, bits, group_size):
+ """Normalize an imatrix to the padded ``[N, gs]`` grouped layout.
+
+ Mirrors the zero-shot search path: the calibration hooks attach a 1D
+ ``[in_features]`` per-column importance vector -- it must be padded by
+ group size, expanded to the weight's grouped rows, and zero-guarded
+ before any per-group search consumes it. ``None`` never reaches here
+ (uniform weights).
+ """
+ from auto_round.data_type.gguf import _imatrix_handle_zero
+
+ qw = imatrix.reshape(1, -1).to(torch.float32)
+ qw = reshape_pad_tensor_by_group_size(qw, group_size, val=1e-5)[0].view(1, -1)
+ qw = qw.expand(weight_reshape.numel() // qw.numel(), -1).reshape(weight_reshape.shape)
+ return _imatrix_handle_zero(qw, weight_reshape, bits, group_size)
+
+
+_ANCHOR_LOGGED = False
diff --git a/auto_round/data_type/utils.py b/auto_round/data_type/utils.py
index 7bca6fe97..676b2822e 100644
--- a/auto_round/data_type/utils.py
+++ b/auto_round/data_type/utils.py
@@ -103,7 +103,7 @@ def revert_tensor_by_pad(data: torch.Tensor, orig_shape: tuple, pad_len: Union[i
def get_quant_func(
- dtype: str, bits: int, sym: bool, disable_opt_rtn=False, group_size=None, iters=200
+ dtype: str, bits: int, sym: bool, disable_opt_rtn=False, group_size=None, iters=200, enable_neuqi=False
) -> tuple[callable, str]:
"""Retrieve the quantization function based on data type, bit width, and symmetry.
@@ -117,6 +117,12 @@ def get_quant_func(
sym (bool): A flag indicating whether the quantization is symmetric (True) or asymmetric (False).
disable_opt_rtn(bool): whether to disable optimized rtn.
group_size (tuple): The block size for weight quantization (e.g., (128, 128)).
+ enable_neuqi (bool): Opt into the NeUQI grid search (arXiv 2505.17595).
+ For ``sym=False`` the joint (scale, zero-point) search replaces the
+ plain min/max initialization on the optimized path; for ``sym=True``
+ the two-stage symmetric scale search
+ (``opt_rtn_int_sym_neuqi``) replaces the uniform ``search_scales``
+ grid. ``False`` (default) keeps the incumbent behavior byte-identical.
Returns:
function: The quantization function corresponding to the specified parameters.
@@ -136,6 +142,15 @@ def get_quant_func(
if not disable_opt_rtn and iters == 0:
rtn_data_type = "opt_rtn_" + dtype
data_types = [rtn_data_type, pad_bits(rtn_data_type), pad_sym(rtn_data_type), pad_sym(pad_bits(rtn_data_type))]
+ if not enable_neuqi and not sym:
+ # the joint NeUQI search is opt-in: without it the asymmetric
+ # optimized path has no search (the integer zero point breaks the
+ # scale-only grid) and resolves to the plain min/max function
+ data_types = [dt for dt in data_types if not dt.endswith("_asym") and not dt.endswith(f"_asym{bits}")]
+ if sym and enable_neuqi:
+ # two-stage symmetric scale search (NeUQI grid machinery, zero point
+ # fixed at 0) -- preferred over the uniform search_scales grid
+ data_types = [f"{rtn_data_type}_sym_neuqi"] + data_types
for data_type in data_types:
from auto_round.data_type import QUANT_FUNC_WITH_DTYPE
diff --git a/auto_round/envs.py b/auto_round/envs.py
index 0a64af5a8..e7a8b706c 100644
--- a/auto_round/envs.py
+++ b/auto_round/envs.py
@@ -210,6 +210,16 @@ environment_variables: dict[str, Callable[[], Any]] = {
# Exposed on the CLI as ``--num_hidden_layers``. The resulting model is a
# partial model and must not be used for a real/production quantization run.
"AR_DEBUG_LAYER_NUM": lambda: _get_optional_positive_int_env("AR_DEBUG_LAYER_NUM"),
+ # NeUQI joint (scale, zero-point) search knobs: coarse/fine candidate
+ # counts (shared by both symmetry classes), backend selection for the
+ # zero-point sweep (auto|eager|compile|triton; latches down permanently
+ # on failure), and the sweep/knobs are otherwise internal (layout and
+ # candidate batching are device-automatic).
+ # unset -> None so the search entries can apply their backend-aware
+ # defaults (wide 256/64 on Triton/compile lanes, narrow 64/32 on eager)
+ "AR_NEUQI_COARSE": lambda: int(v) if (v := os.getenv("AR_NEUQI_COARSE")) is not None else None,
+ "AR_NEUQI_FINE": lambda: int(v) if (v := os.getenv("AR_NEUQI_FINE")) is not None else None,
+ "AR_NEUQI_BACKEND": lambda: os.getenv("AR_NEUQI_BACKEND", "auto").lower(),
}
diff --git a/auto_round/wrapper.py b/auto_round/wrapper.py
index 36ffbb173..66897972e 100644
--- a/auto_round/wrapper.py
+++ b/auto_round/wrapper.py
@@ -84,6 +84,7 @@ class WrapperLinear(torch.nn.Module):
enable_round_tuning=True,
enable_torch_compile=True,
disable_opt_rtn=True,
+ enable_neuqi=False,
**kwargs,
):
"""Initializes the WrapperLinear module.
@@ -98,6 +99,7 @@ class WrapperLinear(torch.nn.Module):
self.orig_layer = orig_layer
self.orig_layer.iters = kwargs.pop("iters", 200)
self.disable_opt_rtn = disable_opt_rtn
+ self.enable_neuqi = enable_neuqi
self.output_device = device
self.device = self.orig_layer.tuning_device if hasattr(self.orig_layer, "tuning_device") else device
self.enable_minmax_tuning = enable_minmax_tuning
@@ -181,13 +183,50 @@ class WrapperLinear(torch.nn.Module):
if clip_max_flat.numel() == self.weight_max.numel() and clip_min_flat.numel() == self.weight_min.numel():
self.weight_max = torch.minimum(self.weight_max, clip_max_flat)
self.weight_min = torch.maximum(self.weight_min, clip_min_flat)
+ # NeUQI frozen init (enable_neuqi on the tuning path): anchor
+ # the grid to the searched (scale, zp) instead of the raw per-group
+ # min/max. Placement mirrors the AWQ clip-as-init above; unsupported
+ # layouts (tuple group sizes, >=16 bits, non-int data types) keep the
+ # status-quo min/max grid. The zero-shot path needs no anchor -- its
+ # search runs inside the quant function dispatch itself.
+ self._neuqi_frozen_margins = False
+ if (
+ self.enable_neuqi
+ and self.enable_round_tuning
+ and weight_reshape is not None
+ and self.weight_min is not None
+ and self.orig_layer.bits < 16
+ and not isinstance(orig_layer.group_size, tuple)
+ and self.orig_layer.data_type == "int"
+ ):
+ if getattr(orig_layer, "awq_clip_max", None) is not None:
+ logger.warning_once(
+ "enable_neuqi anchors the tuning grid to its own search result, which "
+ "overrides the AWQ clip-as-init range on this layer; the clip is ignored."
+ )
+ from auto_round.data_type.neuqi import neuqi_anchor_margins
+
+ _wmin, _wmax = neuqi_anchor_margins(
+ weight_reshape,
+ self.orig_layer.bits,
+ orig_layer.group_size,
+ getattr(orig_layer, "imatrix", None),
+ self.device,
+ sym=bool(getattr(orig_layer, "sym", True)),
+ )
+ self._neuqi_frozen_margins = True
+ self.weight_min = _wmin.to(self.weight_min.dtype)
+ self.weight_max = _wmax.to(self.weight_max.dtype)
self._init_params(
"value", p_dtype, weight_reshape.shape, 0, self.enable_round_tuning and self.orig_layer.bits < 16
)
# Min-max scale initialization
shape = get_scale_shape(orig_weight, orig_layer.group_size)
- self._init_params("min_scale", p_dtype, shape, 1.0, (self.enable_minmax_tuning and self.orig_layer.bits < 16))
- self._init_params("max_scale", p_dtype, shape, 1.0, (self.enable_minmax_tuning and self.orig_layer.bits < 16))
+ _margins_tunable = (self.enable_minmax_tuning and self.orig_layer.bits < 16) and (
+ not self._neuqi_frozen_margins
+ )
+ self._init_params("min_scale", p_dtype, shape, 1.0, _margins_tunable)
+ self._init_params("max_scale", p_dtype, shape, 1.0, _margins_tunable)
self.weight_quant_func, self.data_type = get_quant_func(
orig_layer.data_type,
@@ -196,6 +235,7 @@ class WrapperLinear(torch.nn.Module):
self.disable_opt_rtn,
orig_layer.group_size,
iters=orig_layer.iters,
+ enable_neuqi=self.enable_neuqi,
)
if self.enable_torch_compile:
self.weight_quant_func = compile_func(self.weight_quant_func, self.device)
diff --git a/auto_round_extension/triton/neuqi_sweep.py b/auto_round_extension/triton/neuqi_sweep.py
new file mode 100644
index 000000000..251d5557b
--- /dev/null
+++ b/auto_round_extension/triton/neuqi_sweep.py
@@ -0,0 +1,568 @@
+# Copyright (c) 2026 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Triton kernel for the NeUQI all-candidates zero-point sweep.
+
+One launch evaluates every per-group scale candidate (``scales``, [C, K])
+against every integer zero point for a chunk of groups. Unlike the
+``torch.compile``-fused expression -- whose kernel parallelizes over the
+(candidate, zero-point) lanes and therefore re-reads each weight element from
+L2 once per lane -- this kernel loads each element of a [BC, g] data tile into
+registers exactly once and computes all K x Z losses from registers
+(amplification 1x). Measured on an RTX 3090 (2M groups, g=64, bits=4, 64+32
+candidate grid, fp32): 0.216 s vs 0.488 s (compiled batched expression) and
+12.3 s (eager chunked sweep). Selections match the torch paths' tie profile
+(same ~0.07% flipped groups, worst fp64 relative loss difference ~5e-5): the
+kernel rounds with round-half-to-even (matching ``torch.round``; libdevice
+``round`` is half-away-from-zero and would change selections) and keeps the
+first-minimum zero-point rule via a strict ``<`` with ascending zero points.
+
+Contract (``neuqi_sweep_triton``):
+ data: [C, g] float32, contiguous, CUDA
+ qw: [C, g] float32, contiguous, CUDA, or ``None``
+ scales: [C, K] float32, contiguous, CUDA
+ maxq: maximum integer value (``2**bits - 1``)
+ -> (loss [C, K] float32, argmin zero point [C, K] int32)
+
+Contract (``sym_search_triton``): the symmetric scale search with the same
+registers-resident structure, one data load per element and all K candidates
+under BOTH clamp conventions computed from registers. Rounding and tie rules
+match ``_sym_loss_chunk`` (round-half-to-even; the standard convention wins a
+loss tie against the mirrored one; ascending-candidate first-minimum).
+
+ data: [C, g] float32, contiguous, CUDA
+ qw: [C, g] float32, contiguous, CUDA, or ``None``
+ scales: [C, K] float32 POSITIVE per-group candidates, contiguous, CUDA
+ nmax: symmetric integer bound (``2**(bits - 1)``)
+ -> (best_loss [C] float32, best candidate index [C] int64,
+ best_mirror [C] bool: the mirrored convention won that group)
+
+Contract (``sym_search_shared_triton``): the coarse stage of the symmetric
+search in NORMALIZED space. The driver precomputes dn = w / s0 once (s0 =
+per-group no-clip scale); every group then shares the same candidate fracs
+f_k, so the kernel multiplies dn by the SHARED 1/f_k scalar instead of a
+per-(group, candidate) scale -- no [C, K] scales tensor at all, no division
+inside the candidate loop. The loss omits the s0^2 factor (argmin within a
+group is invariant); the driver unnormalizes the winner for cross-stage
+comparisons. Measured RTX 3090, 2M groups, g=128, 256+64 candidates: 0.24 s
+vs 0.57-0.79 s for the compiled core (parity: same ~0.5% tie-flip profile,
+max fp64 rel diff 1.7e-6 vs eager).
+
+ dn: [C, g] float32, contiguous, CUDA (w / s0)
+ qw: [C, g] float32, contiguous, CUDA, or ``None``
+ fracs: [K] float32 POSITIVE shared multipliers, contiguous, CUDA
+ invf: [K] float32, 1 / fracs, contiguous, CUDA, same device
+ nmax: symmetric integer bound (``2**(bits - 1)``)
+ -> (best_loss [C] float32 NORMALIZED, best candidate index [C] int64,
+ best_mirror [C] bool: the mirrored convention won that group)
+"""
+
+import torch
+
+try:
+ import triton
+ import triton.language as tl
+
+ _TRITON_IMPORT_ERROR = None
+except Exception as e: # pragma: no cover - depends on host toolchain
+ triton = None
+ tl = None
+ _TRITON_IMPORT_ERROR = e
+
+
+if triton is not None:
+
+ @triton.jit
+ def _rint_f32(x):
+ """Native round-half-to-even: cvt.rni.f32.f32 (exact torch.round)."""
+ return tl.inline_asm_elementwise("cvt.rni.f32.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1)
+
+ @triton.jit
+ def _neuqi_sweep_kernel(
+ data_ptr,
+ qw_ptr,
+ scales_ptr,
+ loss_ptr,
+ zp_ptr,
+ C,
+ G,
+ K,
+ NZ,
+ MAXQ: tl.constexpr,
+ HAS_QW: tl.constexpr,
+ BC: tl.constexpr,
+ GP: tl.constexpr,
+ NOMASK: tl.constexpr,
+ ZU: tl.constexpr,
+ ):
+ """Registers-resident (candidate, zero-point) sweep.
+
+ Rounding is native cvt.rni.f32.f32 (round-half-to-even, exactly
+ torch.round; measured selection-identical to the previous arithmetic
+ emulation). NOMASK skips the tail-group mask selects when GP == G
+ (pow2 group size). ZU statically unrolls the zero-point loop when
+ MAXQ + 1 <= 32 (bits <= 5) for ILP -- wider bit widths keep the
+ runtime loop to bound compile time and register pressure.
+ """
+ pid = tl.program_id(0)
+ c_off = pid * BC + tl.arange(0, BC)
+ g_off = tl.arange(0, GP)
+ cm = c_off < C
+ gm = g_off < G
+ d = tl.load(data_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)
+ if HAS_QW:
+ qwt = tl.load(qw_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)
+ else:
+ qwt = tl.zeros((BC, GP), dtype=tl.float32) + 1.0
+ for k in range(0, K):
+ sc = tl.load(scales_ptr + c_off * K + k, mask=cm, other=1.0) # [BC]
+ x = d / sc[:, None]
+ r = _rint_f32(x)
+ r = tl.minimum(tl.maximum(r, -MAXQ - 1.0), 2.0 * MAXQ + 1.0)
+ best = tl.zeros((BC,), dtype=tl.float32) + float("inf")
+ bz = tl.zeros((BC,), dtype=tl.int32)
+ if ZU:
+ for z in tl.static_range(0, MAXQ + 1):
+ q = tl.minimum(tl.maximum(r + z, 0.0), MAXQ)
+ err = sc[:, None] * (q - z) - d
+ loss2 = err * err * qwt
+ if NOMASK:
+ acc = tl.sum(loss2, axis=1)
+ else:
+ acc = tl.sum(tl.where(gm[None, :], loss2, 0.0), axis=1)
+ better = acc < best # strict: ascending z keeps the first minimum
+ best = tl.where(better, acc, best)
+ bz = tl.where(better, z, bz)
+ else:
+ for z in range(0, NZ):
+ q = tl.minimum(tl.maximum(r + z, 0.0), MAXQ)
+ err = sc[:, None] * (q - z) - d
+ loss2 = err * err * qwt
+ if NOMASK:
+ acc = tl.sum(loss2, axis=1)
+ else:
+ acc = tl.sum(tl.where(gm[None, :], loss2, 0.0), axis=1)
+ better = acc < best
+ best = tl.where(better, acc, best)
+ bz = tl.where(better, z, bz)
+ tl.store(loss_ptr + c_off * K + k, best, mask=cm)
+ tl.store(zp_ptr + c_off * K + k, bz, mask=cm)
+
+
+if triton is not None:
+
+ @triton.jit
+ def _sym_search_kernel(
+ data_ptr,
+ qw_ptr,
+ scales_ptr,
+ loss_ptr,
+ idx_ptr,
+ mirror_ptr,
+ C,
+ G,
+ K,
+ NMAX: tl.constexpr,
+ HAS_QW: tl.constexpr,
+ BC: tl.constexpr,
+ GP: tl.constexpr,
+ ):
+ pid = tl.program_id(0)
+ c_off = pid * BC + tl.arange(0, BC)
+ g_off = tl.arange(0, GP)
+ cm = c_off < C
+ gm = g_off < G
+ d = tl.load(data_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)
+ if HAS_QW:
+ qwt = tl.load(qw_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)
+ else:
+ qwt = tl.zeros((BC, GP), dtype=tl.float32) + 1.0
+ best = tl.zeros((BC,), dtype=tl.float32) + float("inf")
+ bk = tl.zeros((BC,), dtype=tl.int32)
+ bmir = tl.zeros((BC,), dtype=tl.int32)
+ for k in range(0, K):
+ sc = tl.load(scales_ptr + c_off * K + k, mask=cm, other=1.0) # [BC]
+ x = d / sc[:, None]
+ fl = tl.floor(x)
+ fr = x - fl
+ # round-half-to-even (matches torch.round)
+ fl_odd = fl - 2.0 * tl.floor(fl * 0.5) == 1.0
+ take_up = (fr > 0.5) | ((fr == 0.5) & fl_odd)
+ r = fl + take_up.to(tl.float32)
+ # standard convention: clamp(-nmax, nmax - 1), dequant s * q
+ q_std = tl.minimum(tl.maximum(r, -NMAX), NMAX - 1.0)
+ e_std = sc[:, None] * q_std - d
+ l_std = tl.sum(tl.where(gm[None, :], e_std * e_std * qwt, 0.0), axis=1)
+ # mirrored convention (negative-scale family): clamp(-(nmax - 1), nmax)
+ q_mir = tl.minimum(tl.maximum(r, -(NMAX - 1.0)), NMAX)
+ e_mir = sc[:, None] * q_mir - d
+ l_mir = tl.sum(tl.where(gm[None, :], e_mir * e_mir * qwt, 0.0), axis=1)
+ mir = l_mir < l_std # strict: the standard convention wins a tie
+ loss = tl.where(mir, l_mir, l_std)
+ better = loss < best # strict: ascending k keeps the first minimum
+ best = tl.where(better, loss, best)
+ bk = tl.where(better, k, bk)
+ bmir = tl.where(better, mir.to(tl.int32), bmir)
+ tl.store(loss_ptr + c_off, best, mask=cm)
+ tl.store(idx_ptr + c_off, bk, mask=cm)
+ tl.store(mirror_ptr + c_off, bmir, mask=cm)
+
+
+if triton is not None:
+
+ @triton.jit
+ def _neuqi_shared_kernel(
+ dn_ptr,
+ qw_ptr,
+ fracs_ptr,
+ invf_ptr,
+ loss_ptr,
+ kp_ptr,
+ zp_ptr,
+ C,
+ G,
+ K,
+ MAXQ: tl.constexpr,
+ HAS_QW: tl.constexpr,
+ BC: tl.constexpr,
+ GP: tl.constexpr,
+ NOMASK: tl.constexpr,
+ ZU: tl.constexpr,
+ ):
+ """Coarse stage of the joint (scale, zero-point) search in NORMALIZED space.
+
+ The driver precomputes dn = w / s0 once (s0 = per-group min-max
+ scale); every group then shares the same candidate fracs f_k, so the
+ kernel multiplies dn by the SHARED 1/f_k scalar -- no [C, K] scales
+ tensor, no division inside the candidate loop. The loss omits the
+ s0^2 factor (the argmin over (k, z) within a group is invariant); the
+ driver unnormalizes the winner. In-kernel argmin over k keeps the
+ ascending-k first-minimum rule of torch's min(dim=1). Rounding is
+ native cvt.rni.f32.f32; tie rules match the reference sweep (ascending
+ z keeps the first minimum, strict <).
+ """
+ pid = tl.program_id(0)
+ c_off = pid * BC + tl.arange(0, BC)
+ g_off = tl.arange(0, GP)
+ cm = c_off < C
+ gm = g_off < G
+ d = tl.load(dn_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)
+ if HAS_QW:
+ qwt = tl.load(qw_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)
+ else:
+ qwt = tl.zeros((BC, GP), dtype=tl.float32) + 1.0
+ best = tl.zeros((BC,), dtype=tl.float32) + float("inf")
+ bk = tl.zeros((BC,), dtype=tl.int32)
+ bz = tl.zeros((BC,), dtype=tl.int32)
+ for k in range(0, K):
+ f = tl.load(fracs_ptr + k)
+ invf = tl.load(invf_ptr + k)
+ x = d * invf
+ r = _rint_f32(x)
+ r = tl.minimum(tl.maximum(r, -MAXQ - 1.0), 2.0 * MAXQ + 1.0)
+ l_best = tl.zeros((BC,), dtype=tl.float32) + float("inf")
+ l_bz = tl.zeros((BC,), dtype=tl.int32)
+ if ZU:
+ for z in tl.static_range(0, MAXQ + 1):
+ q = tl.minimum(tl.maximum(r + z, 0.0), MAXQ)
+ err = f * (q - z) - d
+ loss2 = err * err * qwt
+ if NOMASK:
+ acc = tl.sum(loss2, axis=1)
+ else:
+ acc = tl.sum(tl.where(gm[None, :], loss2, 0.0), axis=1)
+ better = acc < l_best
+ l_best = tl.where(better, acc, l_best)
+ l_bz = tl.where(better, z, l_bz)
+ else:
+ for z in range(0, MAXQ + 1):
+ q = tl.minimum(tl.maximum(r + z, 0.0), MAXQ)
+ err = f * (q - z) - d
+ loss2 = err * err * qwt
+ if NOMASK:
+ acc = tl.sum(loss2, axis=1)
+ else:
+ acc = tl.sum(tl.where(gm[None, :], loss2, 0.0), axis=1)
+ better = acc < l_best
+ l_best = tl.where(better, acc, l_best)
+ l_bz = tl.where(better, z, l_bz)
+ better_k = l_best < best
+ best = tl.where(better_k, l_best, best)
+ bk = tl.where(better_k, k, bk)
+ bz = tl.where(better_k, l_bz, bz)
+ tl.store(loss_ptr + c_off, best, mask=cm)
+ tl.store(kp_ptr + c_off, bk, mask=cm)
+ tl.store(zp_ptr + c_off, bz, mask=cm)
+
+
+def neuqi_shared_sweep_triton(dn, qw, fracs, invf, maxq):
+ """Coarse-stage joint (scale, zero-point) search in normalized space on CUDA.
+
+ Returns (best NORMALIZED loss [C] float32, best candidate index [C] int64,
+ best zero point [C] int32); the driver unnormalizes the loss by s0^2 and
+ maps the index through the shared frac grid.
+ """
+ assert triton is not None, f"triton is not importable: {_TRITON_IMPORT_ERROR}"
+ assert dn.is_cuda and dn.dtype == torch.float32 and dn.is_contiguous()
+ assert fracs.is_contiguous() and fracs.dtype == torch.float32 and fracs.ndim == 1
+ assert invf.is_contiguous() and invf.dtype == torch.float32 and invf.shape == fracs.shape
+ if qw is not None:
+ assert qw.dtype == torch.float32 and qw.is_contiguous() and qw.shape == dn.shape
+ _dev = dn.device
+ assert fracs.device == _dev, f"neuqi_shared: fracs on {fracs.device} but dn on {_dev}"
+ assert invf.device == _dev, f"neuqi_shared: invf on {invf.device} but dn on {_dev}"
+ if qw is not None:
+ assert qw.device == _dev, f"neuqi_shared: qw on {qw.device} but dn on {_dev}"
+
+ c, g = dn.shape
+ k = fracs.numel()
+ gp = _next_pow2(g)
+ bc = max(1, min(64, 4096 // gp))
+ loss = torch.empty((c,), device=_dev, dtype=torch.float32)
+ kp = torch.empty((c,), device=_dev, dtype=torch.int32)
+ zp = torch.empty((c,), device=_dev, dtype=torch.int32)
+ with torch.cuda.device(_dev): # Triton launches on the CURRENT device
+ _neuqi_shared_kernel[(triton.cdiv(c, bc),)](
+ dn,
+ qw if qw is not None else dn, # dummy pointer when unweighted (dead lanes)
+ fracs,
+ invf,
+ loss,
+ kp,
+ zp,
+ c,
+ g,
+ k,
+ MAXQ=maxq,
+ HAS_QW=qw is not None,
+ BC=bc,
+ GP=gp,
+ NOMASK=gp == g,
+ ZU=maxq + 1 <= 32,
+ num_warps=4,
+ )
+ return loss, kp.long(), zp
+
+
+def _next_pow2(n):
+ p = 1
+ while p < n:
+ p *= 2
+ return p
+
+
+def neuqi_sweep_triton(data, qw, scales, maxq):
+ """Run the all-candidates zero-point sweep on CUDA (see module docstring)."""
+ assert triton is not None, f"triton is not importable: {_TRITON_IMPORT_ERROR}"
+ assert data.is_cuda and data.dtype == torch.float32 and data.is_contiguous()
+ assert scales.is_contiguous() and scales.dtype == torch.float32
+ if qw is not None:
+ assert qw.dtype == torch.float32 and qw.is_contiguous() and qw.shape == data.shape
+ # Triton dereferences raw pointers with no device consistency check -- a
+ # cross-device input surfaces as a delayed "device-side assert" far from
+ # the launch. Fail here, naming the offending tensor, instead.
+ _dev = data.device
+ assert scales.device == _dev, f"neuqi_sweep: scales on {scales.device} but data on {_dev}"
+ if qw is not None:
+ assert qw.device == _dev, f"neuqi_sweep: qw on {qw.device} but data on {_dev}"
+
+ c, g = data.shape
+ k = scales.shape[1]
+ nz = maxq + 1
+ gp = _next_pow2(g)
+ bc = max(1, min(64, 4096 // gp))
+ loss = torch.empty((c, k), device=data.device, dtype=torch.float32)
+ zp = torch.empty((c, k), device=data.device, dtype=torch.int32)
+ # Triton launches on the CURRENT cuda device; with tensors on another GPU
+ # (block homes under streaming rotation) the launcher rejects the pointer
+ # ("cannot be accessed from Triton (cpu tensor?)") or misbehaves. Pin the
+ # context to the data's device for the launch.
+ with torch.cuda.device(_dev):
+ return _launch_sweep(data, qw, scales, loss, zp, c, g, k, nz, maxq, bc, gp)
+
+
+def sym_search_triton(data, qw, scales, nmax):
+ """Evaluate all per-group scale candidates under both symmetric clamp
+ conventions on CUDA (see module docstring)."""
+ assert triton is not None, f"triton is not importable: {_TRITON_IMPORT_ERROR}"
+ assert data.is_cuda and data.dtype == torch.float32 and data.is_contiguous()
+ assert scales.is_contiguous() and scales.dtype == torch.float32
+ if qw is not None:
+ assert qw.dtype == torch.float32 and qw.is_contiguous() and qw.shape == data.shape
+ _dev = data.device
+ assert scales.device == _dev, f"sym_search: scales on {scales.device} but data on {_dev}"
+ if qw is not None:
+ assert qw.device == _dev, f"sym_search: qw on {qw.device} but data on {_dev}"
+
+ c, g = data.shape
+ k = scales.shape[1]
+ gp = _next_pow2(g)
+ bc = max(1, min(64, 4096 // gp))
+ loss = torch.empty((c,), device=_dev, dtype=torch.float32)
+ idx = torch.empty((c,), device=_dev, dtype=torch.int32)
+ mirror = torch.empty((c,), device=_dev, dtype=torch.int32)
+ with torch.cuda.device(_dev): # Triton launches on the CURRENT device
+ _sym_search_kernel[(triton.cdiv(c, bc),)](
+ data,
+ qw if qw is not None else data, # dummy pointer when unweighted
+ scales,
+ loss,
+ idx,
+ mirror,
+ c,
+ g,
+ k,
+ NMAX=nmax,
+ HAS_QW=qw is not None,
+ BC=bc,
+ GP=gp,
+ num_warps=4,
+ )
+ return loss, idx.long(), mirror.bool()
+
+
+if triton is not None:
+
+ @triton.jit
+ def _sym_shared_kernel(
+ dn_ptr,
+ qw_ptr,
+ fracs_ptr,
+ invf_ptr,
+ loss_ptr,
+ idx_ptr,
+ mirror_ptr,
+ C,
+ G,
+ K,
+ NMAX: tl.constexpr,
+ HAS_QW: tl.constexpr,
+ BC: tl.constexpr,
+ GP: tl.constexpr,
+ NOMASK: tl.constexpr,
+ ):
+ """Coarse-stage symmetric search in normalized space (see module docstring).
+
+ Rounding is native cvt.rni.f32.f32 (round-half-to-even, matching
+ torch.round). Tie rules match _sym_loss_chunk: the standard convention
+ wins a loss tie (strict <), ascending-k first-minimum (strict <).
+ NOMASK skips the tail-group mask selects when GP == G (pow2 group size).
+ """
+ pid = tl.program_id(0)
+ c_off = pid * BC + tl.arange(0, BC)
+ g_off = tl.arange(0, GP)
+ cm = c_off < C
+ gm = g_off < G
+ d = tl.load(dn_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)
+ if HAS_QW:
+ qwt = tl.load(qw_ptr + c_off[:, None] * G + g_off[None, :], mask=cm[:, None] & gm[None, :], other=0.0)
+ best = tl.zeros((BC,), dtype=tl.float32) + float("inf")
+ bk = tl.zeros((BC,), dtype=tl.int32)
+ bmir = tl.zeros((BC,), dtype=tl.int32)
+ for k in range(0, K):
+ f = tl.load(fracs_ptr + k)
+ invf = tl.load(invf_ptr + k)
+ x = d * invf
+ r = tl.inline_asm_elementwise(
+ "cvt.rni.f32.f32 $0, $1;", "=r,r", [x], dtype=tl.float32, is_pure=True, pack=1
+ )
+ q_std = tl.minimum(tl.maximum(r, -NMAX), NMAX - 1.0)
+ e_std = f * q_std - d
+ q_mir = tl.minimum(tl.maximum(r, -(NMAX - 1.0)), NMAX)
+ e_mir = f * q_mir - d
+ if HAS_QW:
+ se = e_std * e_std * qwt
+ sm = e_mir * e_mir * qwt
+ else:
+ se = e_std * e_std
+ sm = e_mir * e_mir
+ if NOMASK:
+ l_std = tl.sum(se, axis=1)
+ l_mir = tl.sum(sm, axis=1)
+ else:
+ l_std = tl.sum(tl.where(gm[None, :], se, 0.0), axis=1)
+ l_mir = tl.sum(tl.where(gm[None, :], sm, 0.0), axis=1)
+ mir = l_mir < l_std
+ loss = tl.where(mir, l_mir, l_std)
+ better = loss < best
+ best = tl.where(better, loss, best)
+ bk = tl.where(better, k, bk)
+ bmir = tl.where(better, mir.to(tl.int32), bmir)
+ tl.store(loss_ptr + c_off, best, mask=cm)
+ tl.store(idx_ptr + c_off, bk, mask=cm)
+ tl.store(mirror_ptr + c_off, bmir, mask=cm)
+
+
+def sym_search_shared_triton(dn, qw, fracs, invf, nmax):
+ """Coarse-stage symmetric search in normalized space on CUDA (see module docstring)."""
+ assert triton is not None, f"triton is not importable: {_TRITON_IMPORT_ERROR}"
+ assert dn.is_cuda and dn.dtype == torch.float32 and dn.is_contiguous()
+ assert fracs.is_contiguous() and fracs.dtype == torch.float32 and fracs.ndim == 1
+ assert invf.is_contiguous() and invf.dtype == torch.float32 and invf.shape == fracs.shape
+ if qw is not None:
+ assert qw.dtype == torch.float32 and qw.is_contiguous() and qw.shape == dn.shape
+ _dev = dn.device
+ assert fracs.device == _dev, f"sym_shared: fracs on {fracs.device} but dn on {_dev}"
+ assert invf.device == _dev, f"sym_shared: invf on {invf.device} but dn on {_dev}"
+ if qw is not None:
+ assert qw.device == _dev, f"sym_shared: qw on {qw.device} but dn on {_dev}"
+
+ c, g = dn.shape
+ k = fracs.numel()
+ gp = _next_pow2(g)
+ bc = max(1, min(64, 4096 // gp))
+ loss = torch.empty((c,), device=_dev, dtype=torch.float32)
+ idx = torch.empty((c,), device=_dev, dtype=torch.int32)
+ mirror = torch.empty((c,), device=_dev, dtype=torch.int32)
+ with torch.cuda.device(_dev): # Triton launches on the CURRENT device
+ _sym_shared_kernel[(triton.cdiv(c, bc),)](
+ dn,
+ qw if qw is not None else dn, # dummy pointer when unweighted
+ fracs,
+ invf,
+ loss,
+ idx,
+ mirror,
+ c,
+ g,
+ k,
+ NMAX=nmax,
+ HAS_QW=qw is not None,
+ BC=bc,
+ GP=gp,
+ NOMASK=gp == g,
+ num_warps=4,
+ )
+ return loss, idx.long(), mirror.bool()
+
+
+def _launch_sweep(data, qw, scales, loss, zp, c, g, k, nz, maxq, bc, gp):
+ _neuqi_sweep_kernel[(triton.cdiv(c, bc),)](
+ data,
+ qw if qw is not None else data, # dummy pointer when unweighted (dead lanes)
+ scales,
+ loss,
+ zp,
+ c,
+ g,
+ k,
+ nz,
+ MAXQ=maxq,
+ HAS_QW=qw is not None,
+ BC=bc,
+ GP=gp,
+ NOMASK=gp == g,
+ ZU=maxq + 1 <= 32,
+ num_warps=4,
+ )
+ return loss, zp
diff --git a/docs/environments.md b/docs/environments.md
index c47fe1cb0..7a45324f1 100644
--- a/docs/environments.md
+++ b/docs/environments.md
@@ -130,6 +130,36 @@ export AR_ENABLE_ACT_MINMAX_TUNING=1
export AR_SEARCH_SCALE_RATIO=0.75
```
+### AR_NEUQI_COARSE
+- **Description**: Number of coarse-stage candidates in the NeUQI grid search (`--enable_neuqi`, arXiv 2505.17595). The coarse stage scans a wide log-spaced grid; the fine stage refines around the winner (`AR_NEUQI_FINE`). Applies to both the asymmetric joint (scale, zero-point) search and the symmetric two-stage scale search.
+- **Default**: `256` on accelerated backends (Triton/torch.compile), `64` on the eager sweep — the eager path evaluates ~coarse x fine candidates per group, so it narrows the unpinned default (measured quality is flat between the two grids)
+- **Valid Values**: positive integer
+- **Usage**: Lower for faster searches at the cost of grid coverage.
+
+```bash
+export AR_NEUQI_COARSE=128
+```
+
+### AR_NEUQI_FINE
+- **Description**: Number of fine-stage candidates in the NeUQI grid search, refined around the coarse winner. See `AR_NEUQI_COARSE`.
+- **Default**: `64` on accelerated backends (Triton/torch.compile), `32` on the eager sweep
+- **Valid Values**: positive integer
+- **Usage**: Lower for faster searches at the cost of final resolution.
+
+```bash
+export AR_NEUQI_FINE=32
+```
+
+### AR_NEUQI_BACKEND
+- **Description**: Backend override for the NeUQI candidate sweep. `auto` picks the fastest available (extension Triton kernel, then torch.compile-fused, then eager) and latches down permanently on the first kernel failure. `eager`/`compile`/`triton` force one backend for debugging or benchmarking.
+- **Default**: `auto`
+- **Valid Values**: `auto`, `eager`, `compile`, `triton`
+- **Usage**: Escape hatch for driver/hardware incompatibilities; `auto` is recommended.
+
+```bash
+export AR_NEUQI_BACKEND=eager
+```
+
### AR_DYNAMO_CACHE_SIZE_LIMIT
- **Description**: Minimum value to which `torch._dynamo`'s `cache_size_limit`, `accumulated_cache_size_limit`, and `recompile_limit` are bumped when `torch.compile` is enabled (the default except on Windows). The same compiled quant function is reused across every linear layer in a transformer block (q/k/v/o_proj, gate/up/down_proj, ...) but each layer has a different weight shape, so per-layer static recompiles quickly exceed dynamo's default limit (8) and trigger a noisy fallback to eager. Raising the limit keeps static-shape compilation (best perf) and just allows more cache entries.
- **Default**: `16`
diff --git a/docs/environments_CN.md b/docs/environments_CN.md
index 2995f01a6..264c88fd7 100644
--- a/docs/environments_CN.md
+++ b/docs/environments_CN.md
@@ -130,6 +130,36 @@ export AR_ENABLE_ACT_MINMAX_TUNING=1
export AR_SEARCH_SCALE_RATIO=0.75
```
+### AR_NEUQI_COARSE
+- **描述**:NeUQI 网格搜索(`--enable_neuqi`,arXiv 2505.17595)粗阶段的候选数量。粗阶段扫描较宽的对数网格,细阶段在胜者附近细化(`AR_NEUQI_FINE`)。同时作用于非对称联合 (scale, zero-point) 搜索与对称两阶段 scale 搜索。
+- **默认值**:加速后端(Triton/torch.compile)为 `256`,eager 扫描为 `64` —— eager 路径每 group 需评估约 coarse x fine 个候选,因此未显式指定时使用更窄的默认网格(两种网格的实测质量持平)
+- **取值**:正整数
+- **用法**:调低可加快搜索,代价是网格覆盖范围变小。
+
+```bash
+export AR_NEUQI_COARSE=128
+```
+
+### AR_NEUQI_FINE
+- **描述**:NeUQI 网格搜索细阶段的候选数量,在粗阶段胜者附近细化。参见 `AR_NEUQI_COARSE`。
+- **默认值**:加速后端(Triton/torch.compile)为 `64`,eager 扫描为 `32`
+- **取值**:正整数
+- **用法**:调低可加快搜索,代价是最终分辨率下降。
+
+```bash
+export AR_NEUQI_FINE=32
+```
+
+### AR_NEUQI_BACKEND
+- **描述**:NeUQI 候选扫描的后端覆盖。`auto` 选择最快的可用后端(扩展 Triton 内核 → torch.compile 融合 → eager),并在内核首次失败时永久回退。`eager`/`compile`/`triton` 强制指定后端,用于调试或基准测试。
+- **默认值**:`auto`
+- **取值**:`auto`、`eager`、`compile`、`triton`
+- **用法**:驱动/硬件兼容性逃生口;推荐使用 `auto`。
+
+```bash
+export AR_NEUQI_BACKEND=eager
+```
+
### AR_DYNAMO_CACHE_SIZE_LIMIT
- **描述**:在开启 `torch.compile`(除 Windows 外默认开启)时,将 `torch._dynamo` 的 `cache_size_limit`、`accumulated_cache_size_limit` 与 `recompile_limit` 提升到的最小值。同一个被编译的量化函数会被 transformer block 内的所有 linear 层(q/k/v/o_proj、gate/up/down_proj 等)复用,但每层权重 shape 不同,按层的静态重编译会很快超过 dynamo 默认上限(8),导致打印告警并退回 eager。提高该上限可保留静态 shape 编译(性能最佳),仅增加缓存条目数。
- **默认值**:`16`
diff --git a/docs/neuqi_acc.md b/docs/neuqi_acc.md
new file mode 100644
index 000000000..09b8d03d9
--- /dev/null
+++ b/docs/neuqi_acc.md
@@ -0,0 +1,30 @@
+### NeUQI Accuracy Validation
+
+**NeUQI** ([arXiv 2505.17595](https://arxiv.org/abs/2505.17595)) is a calibration-free grid search for the optimized-RTN path, enabled with `--enable_neuqi`: asymmetric layers run a joint (scale, integer zero-point) search, symmetric layers a two-stage signed scale search; on the zero-shot path both are weighted by the activation imatrix (collected automatically), and with `iters > 0` the anchor uses the activation imatrix when the tuning path collects one (SignRoundV2's `enable_alg_ext`), running unweighted otherwise. With `iters > 0`, the search result anchors the SignRound tuning grid (frozen init).
+
+#### Protocol
+
+KL divergence between the BF16 base model and the dequantized W4A16 checkpoint over the **full vocabulary**, computed in **fp32** on openwebtext (8 sequences × 8192 tokens, 65,528 positions). Both passes run through the same evaluation stack, so the only difference is the quantization; **lower is better**. `top1` is the next-token agreement rate, `top5` the Jaccard similarity of the top-5 sets. Wall times are single-GPU runs on one RTX 3090 (24 GB) for Qwen3.8-27B (W4A16, group size 128).
+
+#### Results — iters=0 (calibration-free zero-shot)
+
+| recipe | mean | median | p90 | p99 | top1 | top5 | wall |
+|:--|--:|--:|--:|--:|--:|--:|--:|
+| asym, NeUQI | **0.02579** | 0.01361 | 0.05046 | 0.21753 | 0.9234 | 0.8514 | 11m |
+| sym, NeUQI | **0.02734** | 0.01468 | 0.05358 | 0.22014 | 0.9213 | 0.8463 | **9m** |
+| sym, incumbent OptRTN search | 0.02852 | 0.01533 | 0.05678 | 0.23453 | 0.9191 | 0.8429 | 13m19s |
+
+- The symmetric two-stage search **strictly dominates the incumbent search**: −4.1% mean KL, −32% wall time, and better agreement at every quantile.
+- There is no incumbent asymmetric search — `--enable_neuqi` adds the optimized-init capability for asymmetric layers; the quality ladder is flat from 64/32 to 256/64, so the [backend-aware default](./environments.md) narrows the eager sweep to 64/32 at no measured cost.
+
+#### Results — iters > 0 (SignRoundV2 tuning, `enable_alg_ext`)
+
+| recipe | mean | median | p90 | p99 | top1 | top5 | wall |
+|:--|--:|--:|--:|--:|--:|--:|--:|
+| asym, SignRoundV2 + NeUQI, iters=200 (64/32 grid) | **0.01921** | 0.00905 | 0.03591 | 0.17297 | 0.9370 | 0.8727 | 2h57m21s |
+| asym, SignRoundV2 + NeUQI | **0.02106** | 0.01050 | 0.04071 | 0.18038 | 0.9317 | 0.8661 | 49m07s |
+| sym, SignRoundV2 + NeUQI | **0.02191** | 0.01116 | 0.04237 | 0.18473 | 0.9304 | 0.8627 | 56m52s |
+| sym, SignRoundV2 without NeUQI | 0.02203 | 0.01112 | 0.04307 | 0.18239 | 0.9295 | 0.8630 | 55m40s |
+
+- At `iters=50` the NeUQI anchor is **quality-neutral on sym** (0.02191 vs 0.02203 without NeUQI): SignRoundV2's own tuning dominates once it runs, so the anchor's role is a no-cost initialization, not an accuracy lever. (An earlier no-NeUQI baseline of 0.02626 was not SignRoundV2-equivalent — most of its gap was the quantizer version, not NeUQI.)
+- Asymmetric tuning lands at 0.02106, on par with the best prior asymmetric recipe family at the same iteration budget; the dose-response continues to 0.01921 at 200 iterations. (No SignRoundV2 asym no-NeUQI baseline has been measured; the asym anchor claim is parity-with-prior, not an A/B.)
diff --git a/docs/step_by_step.md b/docs/step_by_step.md
index 628b3e24e..8a602b0b0 100755
--- a/docs/step_by_step.md
+++ b/docs/step_by_step.md
@@ -600,6 +600,8 @@ AutoRound also supports Optimized RTN (Round-To-Nearest) mode for fast, calibrat
For the GGUF format, we have optimized the RTN algorithm inspired by llamacpp. To use the original (pure) RTN algorithm instead, enable the `--disable_opt_rtn` option.
+Passing `--enable_neuqi` opts the optimized path into the **NeUQI** grid search ([arXiv 2505.17595](https://arxiv.org/abs/2505.17595)): asymmetric layers run a joint (scale, integer zero-point) search and symmetric layers a two-stage signed scale search, both weighted by the activation imatrix on the zero-shot path, which is collected automatically (with `iters > 0` the anchor uses the imatrix when the tuning path collects one, running unweighted otherwise). With `iters > 0`, the search result anchors the SignRound tuning grid (frozen init). Grid sizes are tunable via `AR_NEUQI_COARSE`/`AR_NEUQI_FINE` (see [environments](./environments.md)); the unpinned default is backend-aware (wide only on the Triton/torch.compile lanes). Accuracy and wall-time results are shown in [NeUQI accuracy validation](./neuqi_acc.md).
+
#### CLI Usage
Two dedicated CLI entry points are provided as shortcuts:
diff --git a/docs/step_by_step_CN.md b/docs/step_by_step_CN.md
index 6dde8eccf..e9bcbb7ee 100755
--- a/docs/step_by_step_CN.md
+++ b/docs/step_by_step_CN.md
@@ -590,6 +590,8 @@ AutoRound 还提供优化版 RTN(Round-To-Nearest,就近舍入)模式,
对于 GGUF 格式,我们参考 llamacpp 的思路,优化了 RTN 算法。若需使用原始(非优化)RTN 算法,开启 `--disable_opt_rtn` 即可。
+在优化路径上开启 `--enable_neuqi` 即可启用 **NeUQI** 网格搜索([arXiv 2505.17595](https://arxiv.org/abs/2505.17595)):非对称层执行联合 (scale, 整数 zero-point) 搜索,对称层执行两阶段带符号 scale 搜索,在零样本路径(`iters=0`)上二者均以激活 imatrix 加权(imatrix 会自动采集;`iters > 0` 时锚点在调优路径采集到 imatrix 时使用它,否则不加权)。当 `iters > 0` 时,搜索结果将作为 SignRound 调优网格的锚点(frozen init)。网格规模可通过 `AR_NEUQI_COARSE`/`AR_NEUQI_FINE` 调整(参见[《环境变量》](./environments_CN.md));未显式指定时的默认值与后端相关(仅在 Triton/torch.compile 路径使用宽网格)。精度与耗时结果详见[《NeUQI 精度验证》](./neuqi_acc.md)。
+
#### 命令行使用
我们提供了两个专用的 CLI 入口作为快捷方式:
diff --git a/test/unit/test_cpu/quantization/test_neuqi.py b/test/unit/test_cpu/quantization/test_neuqi.py
new file mode 100644
index 000000000..b4e4b63c3
--- /dev/null
+++ b/test/unit/test_cpu/quantization/test_neuqi.py
@@ -0,0 +1,1699 @@
+# Copyright (c) 2026 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+from auto_round.data_type.int import quant_tensor_asym
+from auto_round.data_type.neuqi import _log_search_engaged, neuqi_search_scale_zero, quant_tensor_opt_rtn_asym
+from auto_round.data_type.register import QUANT_FUNC_WITH_DTYPE
+from auto_round.data_type.utils import get_quant_func
+
+
+def _heavy_tailed_data(n_groups=64, group_size=128, seed=0):
+ gen = torch.Generator().manual_seed(seed)
+ data = torch.randn(n_groups, group_size, generator=gen)
+ # inject a few large outliers per group to make min/max suboptimal
+ data[:, :5] *= 120.0
+ return data
+
+
+def _mse(x, y):
+ return ((x.float() - y.float()) ** 2).mean().item()
+
+
+def _oracle_loss_fp64(data, qw, scale, zp, maxq):
+ """fp64 loss of one group under fixed (scale, zp) -- the tie-break oracle."""
+ r = torch.round(data.double() / scale.double()).clamp_(min=-maxq - 1, max=2 * maxq + 1)
+ q = (r + zp.double()).clamp_(0, maxq)
+ deq = scale.double() * (q - zp.double())
+ loss = (deq - data.double()) ** 2
+ if qw is not None:
+ loss = loss * qw.double()
+ return loss.sum(-1)
+
+
+def _assert_launch_latches(monkeypatch, N, attempt, impl_attr, checked_attr, broken_attr, gates, tail_args):
+ """A raising kernel/compile launch must return None, latch its broken flag,
+ and never retry (all three latch sites share this contract)."""
+ calls = {"n": 0}
+
+ def _boom(*a, **k):
+ calls["n"] += 1
+ raise RuntimeError("launch failed")
+
+ monkeypatch.setattr(N, impl_attr, _boom)
+ monkeypatch.setattr(N, checked_attr, True)
+ for gate_name, gate_ret in gates.items():
+ monkeypatch.setattr(N, gate_name, lambda *a, _r=gate_ret, **k: _r)
+ fake = SimpleNamespace(is_cuda=True, device=torch.device("cuda"))
+ out = getattr(N, attempt)(fake, *tail_args)
+ assert out is None
+ assert getattr(N, broken_attr)
+ out2 = getattr(N, attempt)(fake, *tail_args)
+ assert out2 is None and calls["n"] == 1 # latched: no second launch
+
+
+def _reset_neuqi_state(defaults):
+ """Reset neuqi module latch state (call before AND after each test)."""
+ import auto_round.data_type.neuqi as N
+
+ for attr, default in defaults.items():
+ if hasattr(N, attr):
+ setattr(N, attr, default)
+
+
+class TestNeuqiAsymSearch:
+ def test_search_output_contract(self):
+ """Validity/integrality/shape contract of the joint search outputs."""
+ import auto_round.data_type.neuqi as N
+
+ torch.manual_seed(5)
+ data = torch.randn(512, 64) * 0.05
+ qw = torch.rand(512, 64) + 0.1
+ scale, zp = N.neuqi_search_scale_zero(data, bits=4, qw=qw, coarse_n=4, fine_n=3)
+ assert scale.shape == (512, 1) and zp.shape == (512, 1)
+ assert torch.all(zp == zp.round())
+ assert (zp >= 0).all() and (zp <= 15).all()
+ assert torch.all(scale > 0)
+
+ data = _heavy_tailed_data(n_groups=32)
+ scale, zp = neuqi_search_scale_zero(data, bits=4)
+ assert torch.all(zp == zp.round())
+ assert torch.all(zp >= 0) and torch.all(zp <= 15)
+ assert torch.all(scale > 0)
+
+ gen = torch.Generator().manual_seed(2)
+ for shape in [(4, 33, 128), (17, 65), (3, 5, 7, 11)]:
+ tensor = torch.randn(*shape, generator=gen)
+ qdq, scale, zp = quant_tensor_opt_rtn_asym(tensor.clone(), bits=4, group_size=32)
+ assert qdq.shape == tensor.shape
+
+ def test_beats_minmax_on_heavy_tailed_data(self):
+ data = _heavy_tailed_data()
+ bits, group_size = 4, 128
+
+ minmax_qdq, _, _ = quant_tensor_asym(data.clone(), bits=bits, group_size=group_size)
+ neuqi_qdq, scale, zp = quant_tensor_opt_rtn_asym(data.clone(), bits=bits, group_size=group_size)
+
+ assert _mse(neuqi_qdq, data) < _mse(minmax_qdq, data)
+
+ def test_close_to_dense_exhaustive_reference(self):
+ """On a small case the search must approach a dense exhaustive grid."""
+ gen = torch.Generator().manual_seed(3)
+ n, g, bits = 8, 16, 2
+ maxq = 2**bits - 1
+ data = torch.randn(n, g, generator=gen)
+ data[:, 0] *= 50.0
+
+ scale, zp = neuqi_search_scale_zero(data, bits=bits)
+ r = torch.round(data / scale)
+ q = (r + zp).clamp(0, maxq)
+ our_loss = (((scale * (q - zp)) - data) ** 2).sum().item()
+
+ # brute force: dense additive scale grid per group x all integer zero points
+ best = torch.full((n,), float("inf"))
+ wmin = torch.clamp(data.min(-1).values, max=0)
+ wmax = torch.clamp(data.max(-1).values, min=0)
+ s0 = ((wmax - wmin) / maxq).clamp(min=1e-5)
+ for i in range(401):
+ frac = 0.05 + (1.0 - 0.05) * i / 400
+ s = (s0 * frac).unsqueeze(-1)
+ r = torch.round(data / s)
+ for z in range(maxq + 1):
+ q = (r + z).clamp(0, maxq)
+ loss = ((s * (q - z) - data) ** 2).sum(-1)
+ best = torch.minimum(best, loss)
+ exhaustive_loss = best.sum().item()
+ # within 1% of the (much denser) exhaustive grid
+ assert our_loss <= exhaustive_loss * 1.01 + 1e-9
+
+ def test_weighting_changes_solution(self):
+ """A per-element weight that de-emphasizes outliers must pull the scale down."""
+ gen = torch.Generator().manual_seed(1)
+ data = torch.randn(16, 64, generator=gen)
+ data[:, :4] *= 100.0
+
+ qw = torch.ones_like(data)
+ qw[:, :4] = 1e-4 # ignore outliers
+
+ # grid pinned to the measured 64/32: the scale comparison below is a
+ # heuristic (only loss dominance is a hard invariant) and finer grids
+ # shift near-tie selections between the two runs
+ scale_plain, _ = neuqi_search_scale_zero(data, bits=4, coarse_n=64, fine_n=32)
+ scale_weighted, _ = neuqi_search_scale_zero(data, bits=4, qw=qw, coarse_n=64, fine_n=32)
+ # ignoring outliers must never increase the scale and must shrink it somewhere
+ assert torch.all(scale_weighted <= scale_plain + 1e-6)
+ assert torch.any(scale_weighted < scale_plain)
+
+ def weighted_loss(scale):
+ maxq = 15
+ r = torch.round(data / scale).clamp(-maxq - 1, 2 * maxq + 1)
+ losses = []
+ for z in range(maxq + 1):
+ q = (r + z).clamp(0, maxq)
+ losses.append((((scale * (q - z) - data) ** 2) * qw).sum(-1))
+ return torch.stack(losses, -1).min(-1).values.sum()
+
+ assert weighted_loss(scale_weighted) < weighted_loss(scale_plain)
+
+ def test_constant_group_is_exact(self):
+ data = torch.full((4, 128), 0.25)
+ qdq, scale, zp = quant_tensor_opt_rtn_asym(data.clone(), bits=4, group_size=128)
+ # the fp16 scale cast limits exactness to one quantization step (0.25 / 15)
+ assert (qdq - data).abs().max() <= 0.25 / 15
+ assert torch.all(scale > 0)
+
+
+class TestNeuqiAsymIntegration:
+ """Asym joint search through the quant-func API: output validity across
+ bit widths and imatrix weighting that steers the solution."""
+
+ def test_output_validity_and_imatrix_weighting(self):
+ data = _heavy_tailed_data(n_groups=32)
+ imatrix = torch.rand(128) + 0.1
+
+ qdq_plain, _, _ = quant_tensor_opt_rtn_asym(data.clone(), bits=4, group_size=128)
+ qdq_im, _, _ = quant_tensor_opt_rtn_asym(data.clone(), bits=4, group_size=128, imatrix=imatrix)
+
+ qw = imatrix.expand(32, 128)
+ weighted = lambda q: (((q - data) ** 2) * qw).sum().item() # noqa: E731
+ assert weighted(qdq_im) < weighted(qdq_plain)
+
+ for bits in (2, 3, 4, 8):
+ d = _heavy_tailed_data(n_groups=8, group_size=64, seed=bits)
+ qdq, scale, zp = quant_tensor_opt_rtn_asym(d.clone(), bits=bits, group_size=64)
+ maxq = 2**bits - 1
+ assert torch.all(zp >= 0) and torch.all(zp <= maxq)
+ assert torch.all(scale > 0)
+ assert qdq.shape == d.shape
+
+
+class TestNeuqiAsymFusedSweep:
+ """The torch.compile-fused zero-point sweep must reproduce the eager reference.
+
+ ``AR_NEUQI_BACKEND=compile`` routes every chunk through the fused expression.
+ Selections must match the eager sweep; when compilation is unavailable in the
+ environment (e.g. no host C++ compiler for Inductor) the dispatch must fall
+ back to the eager expression and still return correct results."""
+
+ def _search_with_backend(self, monkeypatch, backend, data, bits, qw, **kwargs):
+ import auto_round.data_type.neuqi as N
+ from auto_round import envs
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", backend)
+ monkeypatch.setattr(N, "_fused_zp_fns", {})
+ monkeypatch.setattr(N, "_fused_zp_broken", False)
+ scale, zp = N.neuqi_search_scale_zero(data.clone(), bits, qw=qw.clone() if qw is not None else None, **kwargs)
+ return scale, zp, N
+
+ @pytest.mark.enable_torch_compile
+ def test_fused_matches_eager(self, monkeypatch):
+ """One setup, the structural combo set: bits=2 (smallest zp grid),
+ bits=4 (shipping default), bits=8 (zp grid of 256, largest sweep
+ tensors), and one weighted arm at the extreme -- the qw multiply is
+ size-independent, so a single weighted combo covers it. Selections
+ must match the eager sweep; a flip is acceptable only between
+ fp64-tied candidates."""
+ for bits, weighted in ((2, False), (4, False), (8, False), (8, True)):
+ gen = torch.Generator().manual_seed(bits * 10 + int(weighted))
+ n, g = 128, 64
+ data = torch.randn(n, g, generator=gen)
+ data[:, :5] *= 120.0 # heavy tails: near-tie zero points are common here
+ qw = torch.rand(n, g, generator=gen) + 0.1 if weighted else None
+
+ scale_e, zp_e, _ = self._search_with_backend(monkeypatch, "eager", data, bits, qw, coarse_n=8, fine_n=4)
+ scale_f, zp_f, N = self._search_with_backend(monkeypatch, "compile", data, bits, qw, coarse_n=8, fine_n=4)
+ if N._fused_zp_broken or not N._fused_zp_fns:
+ pytest.skip("torch.compile/inductor unavailable in this environment")
+
+ mismatch = (zp_e != zp_f).logical_or(scale_e != scale_f).nonzero().flatten()
+ for i_ in mismatch.tolist():
+ loss_e = _oracle_loss_fp64(
+ data[i_], qw[i_] if qw is not None else None, scale_e[i_], zp_e[i_], 2**bits - 1
+ )
+ loss_f = _oracle_loss_fp64(
+ data[i_], qw[i_] if qw is not None else None, scale_f[i_], zp_f[i_], 2**bits - 1
+ )
+ assert (loss_f - loss_e).abs().item() <= 1e-6 * loss_e.abs().item()
+
+ def test_last_layout_matches_mid_reference(self, monkeypatch):
+ """The CUDA layout (group axis last) must match the reference layout's
+ selections up to fp64 ties -- it runs on CUDA, its numerics are validated
+ here on CPU."""
+ import auto_round.data_type.neuqi as N
+ from auto_round import envs
+
+ gen = torch.Generator().manual_seed(31)
+ data = torch.randn(128, 64, generator=gen)
+ data[:, :4] *= 90.0
+ qw = torch.rand(128, 64, generator=gen) + 0.1
+
+ monkeypatch.setattr(N, "_fused_zp_fns", {})
+ monkeypatch.setattr(N, "_fused_zp_broken", False)
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")
+ monkeypatch.setattr(N, "_zp_expr_for", lambda dt: N._zp_expr_mid)
+ scale_m, zp_m = N.neuqi_search_scale_zero(data.clone(), bits=4, qw=qw, coarse_n=16, fine_n=4)
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")
+ monkeypatch.setattr(N, "_zp_expr_for", lambda dt: N._zp_expr_last)
+ scale_l, zp_l = N.neuqi_search_scale_zero(data.clone(), bits=4, qw=qw, coarse_n=16, fine_n=4)
+
+ if N._fused_zp_broken or not N._fused_zp_fns:
+ pytest.skip("torch.compile/inductor unavailable in this environment")
+
+ mismatch = (zp_m != zp_l).logical_or(scale_m != scale_l).nonzero().flatten()
+ maxq = 15
+ for i in mismatch.tolist():
+ loss_m = _oracle_loss_fp64(data[i], qw[i], scale_m[i], zp_m[i], maxq)
+ loss_l = _oracle_loss_fp64(data[i], qw[i], scale_l[i], zp_l[i], maxq)
+ assert (loss_l - loss_m).abs().item() <= 1e-6 * loss_m.abs().item()
+
+ def test_compile_failure_falls_back_to_eager(self, monkeypatch):
+ """A broken host toolchain must degrade to the eager sweep, not crash."""
+ import auto_round.data_type.neuqi as N
+ from auto_round import envs
+
+ data = _heavy_tailed_data(n_groups=16, seed=11)
+
+ def _boom(*args, **kwargs):
+ raise RuntimeError("simulated inductor failure")
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")
+ monkeypatch.setattr(N, "_get_fused_zp_fn", _boom)
+ monkeypatch.setattr(N, "_fused_zp_fns", {})
+ monkeypatch.setattr(N, "_fused_zp_broken", False)
+
+ scale_f, zp_f = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=4, fine_n=2)
+ assert N._fused_zp_broken is True # permanently latched after the first failure
+ monkeypatch.setattr(N, "_fused_zp_broken", False)
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")
+ scale_e, zp_e = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=4, fine_n=2)
+ torch.testing.assert_close(scale_f, scale_e)
+ torch.testing.assert_close(zp_f, zp_e)
+
+
+class TestNeuqiAsymBatchedSweep:
+ """The all-candidates batched sweep must reproduce the per-candidate search.
+
+ The batched sweep evaluates every coarse/fine candidate in one fused
+ call per group chunk ([C, K, Z, g], min over z fused in-kernel, then first-min
+ over candidates). Selections must match the per-candidate sweep; ties may
+ resolve differently only within fp64-oracle tolerance."""
+
+ def _search(self, monkeypatch, backend, batch, data, bits, qw, **kwargs):
+ import auto_round.data_type.neuqi as N
+ from auto_round import envs
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", backend)
+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: batch == "on")
+ monkeypatch.setattr(N, "_fused_zp_fns", {})
+ monkeypatch.setattr(N, "_fused_zp_broken", False)
+ monkeypatch.setattr(N, "_batched_broken", False)
+ scale, zp = N.neuqi_search_scale_zero(data.clone(), bits, qw=qw.clone() if qw is not None else None, **kwargs)
+ return scale, zp, N
+
+ @pytest.mark.enable_torch_compile
+ def test_batched_matches_per_candidate(self, monkeypatch):
+ """One setup, the structural combo set (see test_fused_matches_eager):
+ batched selections must match the per-candidate sweep up to
+ fp64-oracle ties."""
+ for bits, weighted in ((2, False), (4, False), (8, False), (8, True)):
+ gen = torch.Generator().manual_seed(bits * 7 + int(weighted))
+ n, g = 128, 64
+ data = torch.randn(n, g, generator=gen)
+ data[:, :4] *= 100.0
+ qw = torch.rand(n, g, generator=gen) + 0.1 if weighted else None
+
+ scale_ref, zp_ref, _ = self._search(monkeypatch, "eager", "off", data, bits, qw, coarse_n=8, fine_n=3)
+ scale_b, zp_b, N = self._search(monkeypatch, "compile", "on", data, bits, qw, coarse_n=8, fine_n=3)
+ if N._batched_broken or N._zp_expr_batched not in N._fused_zp_fns:
+ pytest.skip("torch.compile/inductor unavailable in this environment")
+
+ mismatch = (zp_ref != zp_b).logical_or(scale_ref != scale_b).nonzero().flatten()
+ maxq = 2**bits - 1
+ for i_ in mismatch.tolist():
+ loss_ref = _oracle_loss_fp64(
+ data[i_], qw[i_] if qw is not None else None, scale_ref[i_], zp_ref[i_], maxq
+ )
+ loss_b = _oracle_loss_fp64(data[i_], qw[i_] if qw is not None else None, scale_b[i_], zp_b[i_], maxq)
+ assert (loss_b - loss_ref).abs().item() <= 1e-6 * loss_ref.abs().item()
+
+ def test_batch_failure_falls_back_to_per_candidate(self, monkeypatch):
+ """A batched-call failure must degrade to the per-candidate sweep, not crash."""
+ import auto_round.data_type.neuqi as N
+ from auto_round import envs
+
+ data = _heavy_tailed_data(n_groups=24, group_size=64, seed=17)
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")
+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: True)
+ monkeypatch.setattr(N, "_fused_zp_fns", {})
+ monkeypatch.setattr(N, "_fused_zp_broken", False)
+ monkeypatch.setattr(N, "_batched_broken", False)
+ monkeypatch.setattr(N, "_eval_batched_chunk", lambda *a, **k: None) # simulate failure
+
+ scale_f, zp_f = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=6, fine_n=2)
+
+ # reference: per-candidate sweep (batching off never touches the patched fn)
+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: False)
+ scale_ref, zp_ref = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=6, fine_n=2)
+ torch.testing.assert_close(scale_f, scale_ref)
+ torch.testing.assert_close(zp_f, zp_ref)
+
+
+class TestNeuqiAsymTritonSweep:
+ """The extension Triton sweep integrates behind the batched dispatch.
+
+ Triton kernels cannot run on CPU hosts, so the plumbing (scale construction,
+ first-min bookkeeping, latch fallbacks, policy) is tested with a mock that
+ honors the kernel's contract; the kernel itself is exercised on CUDA by
+ ``test/unit/test_cuda/quantization/test_neuqi_triton.py`` and by the
+ self-contained GPU benchmark."""
+
+ def _mock_sweep_from(self, monkeypatch, impl):
+ import auto_round.data_type.neuqi as N
+
+ monkeypatch.setattr(N, "_triton_checked", True)
+ monkeypatch.setattr(N, "_triton_sweep", impl)
+ monkeypatch.setattr(N, "_triton_broken", False)
+
+ def test_triton_mock_matches_reference(self, monkeypatch):
+ """Driving the batched passes through a contract-honoring mock must
+ reproduce the per-candidate reference selections exactly."""
+ import auto_round.data_type.neuqi as N
+ from auto_round import envs
+
+ def mock(data, qw, scales, maxq): # same contract as neuqi_sweep_triton
+ zp_grid = torch.arange(0, maxq + 1, device=data.device, dtype=torch.float32)
+ return N._zp_expr_batched(data, qw, scales, zp_grid, maxq)
+
+ self._mock_sweep_from(monkeypatch, mock)
+ data = _heavy_tailed_data(n_groups=64, group_size=64, seed=23)
+ qw = torch.rand(64, 64) + 0.1
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "triton")
+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: True)
+ monkeypatch.setattr(N, "_fused_zp_fns", {})
+ monkeypatch.setattr(N, "_fused_zp_broken", False)
+ monkeypatch.setattr(N, "_batched_broken", False)
+ scale_t, zp_t = N.neuqi_search_scale_zero(data.clone(), bits=4, qw=qw, coarse_n=8, fine_n=3)
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")
+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: False)
+ scale_ref, zp_ref = N.neuqi_search_scale_zero(data.clone(), bits=4, qw=qw, coarse_n=8, fine_n=3)
+ torch.testing.assert_close(scale_t, scale_ref)
+ torch.testing.assert_close(zp_t, zp_ref)
+
+ def test_triton_failure_latches_to_compile_path(self, monkeypatch):
+ """A raising Triton sweep must latch off and finish via the torch paths."""
+ import auto_round.data_type.neuqi as N
+ from auto_round import envs
+
+ def boom(data, qw, scales, maxq):
+ raise RuntimeError("simulated triton failure")
+
+ self._mock_sweep_from(monkeypatch, boom)
+ data = _heavy_tailed_data(n_groups=32, group_size=64, seed=29)
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "triton")
+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: True)
+ monkeypatch.setattr(N, "_fused_zp_fns", {})
+ monkeypatch.setattr(N, "_fused_zp_broken", False)
+ monkeypatch.setattr(N, "_batched_broken", False)
+ scale_f, zp_f = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=6, fine_n=2)
+ assert N._triton_broken is True
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")
+ scale_ref, zp_ref = N.neuqi_search_scale_zero(data.clone(), bits=4, coarse_n=6, fine_n=2)
+ torch.testing.assert_close(scale_f, scale_ref)
+ torch.testing.assert_close(zp_f, zp_ref)
+
+ def test_resolver_logs_engagement(self, monkeypatch):
+ """A successful resolution announces itself: the Triton arm is silent
+ otherwise, and live runs must be able to tell which backend served."""
+ import sys
+ import types
+
+ import auto_round.data_type.neuqi as N
+
+ stub = types.ModuleType("auto_round_extension.triton.neuqi_sweep")
+ stub.neuqi_sweep_triton = object()
+ monkeypatch.setitem(sys.modules, "auto_round_extension.triton.neuqi_sweep", stub)
+ recorded = []
+ monkeypatch.setattr(N, "logger", types.SimpleNamespace(info=recorded.append, debug=recorded.append))
+ monkeypatch.setattr(N, "_triton_checked", False)
+ monkeypatch.setattr(N, "_triton_sweep", None)
+ assert N._triton_sweep_fn() is stub.neuqi_sweep_triton
+ assert any("Triton zero-point sweep engaged" in str(r) for r in recorded)
+
+
+class TestEnableNeuqiAPI:
+ """enable_neuqi (RTNConfig/SignRoundConfig kwarg + --enable_neuqi CLI flag):
+ opt-in gate for the NeUQI grid search on BOTH symmetry classes."""
+
+ def test_dispatch_table(self):
+ """Off: plain asym min/max + incumbent sym; opted in: joint search on asym."""
+ from auto_round.data_type.int import quant_tensor_asym as plain_asym
+
+ func, name = get_quant_func("int", 4, False, disable_opt_rtn=False, group_size=None, iters=0)
+ assert "opt_rtn" not in name and name.endswith("_asym") and func is plain_asym
+ func, name = get_quant_func("int", 4, False, disable_opt_rtn=False, group_size=None, iters=0, enable_neuqi=True)
+ assert name == "opt_rtn_int_asym" and func is QUANT_FUNC_WITH_DTYPE["opt_rtn_int_asym"]
+ _, sym_name = get_quant_func("int", 4, True, disable_opt_rtn=False, group_size=None, iters=0)
+ assert sym_name == "opt_rtn_int_sym"
+
+ def test_config_flag_semantics(self):
+ """enable_neuqi defaults off and accepts True on both symmetry classes."""
+ from auto_round.algorithms.quantization.rtn.config import RTNConfig
+
+ assert RTNConfig(bits=4, group_size=32, sym=False).enable_neuqi is False
+ on = RTNConfig(bits=4, group_size=32, sym=False, disable_opt_rtn=False, enable_neuqi=True)
+ assert on.enable_neuqi is True
+ sym_on = RTNConfig(bits=4, group_size=32, sym=True, disable_opt_rtn=False, enable_neuqi=True)
+ assert sym_on.enable_neuqi is True # engages the two-stage symmetric search
+
+ def test_neuqi_with_opt_rtn_disabled_raises(self):
+ from auto_round.algorithms.quantization.rtn.config import RTNConfig
+
+ with pytest.raises(ValueError, match="--enable_neuqi requires the optimized-RTN path"):
+ RTNConfig(bits=4, group_size=32, sym=False, disable_opt_rtn=True, enable_neuqi=True)
+
+ def test_default_is_off(self):
+ from auto_round.algorithms.quantization.rtn.config import RTNConfig
+
+ cfg = RTNConfig(bits=4, group_size=32, sym=False)
+ assert cfg.enable_neuqi is False
+
+
+class TestNeuqiGridHelpers:
+ """Shared coarse-grid helper invariants (both symmetry classes use _coarse_grid)."""
+
+ def test_coarse_grid_values_and_freshness(self):
+ """The eager coarse-grid helper must be numerically identical to the
+ previous inline torch.logspace, and a cached grid is a cross-stream
+ hazard in the expert fan-out (49aa6405: filled on the first caller's
+ stream, read from other threads' streams) -- it must be recreated per
+ call, never lru_cache'd."""
+ import math
+
+ import auto_round.data_type.neuqi as N
+
+ for lo in (0.03125, 0.25):
+ got = N._coarse_grid(lo, 64, torch.device("cpu"), torch.float32)
+ want = torch.logspace(math.log10(lo), 0.0, 64, dtype=torch.float32)
+ assert got.shape == want.shape
+ assert torch.equal(got, want)
+
+ args = (0.001, 8, torch.device("cpu"), torch.float32)
+ a = N._coarse_grid(*args)
+ b = N._coarse_grid(*args)
+ assert a is not b, "grid must not be shared state across calls"
+ assert not hasattr(N._coarse_grid, "cache_info"), "tensor-returning helper must not be lru_cache'd"
+ torch.testing.assert_close(a, b) # values stay deterministic across calls
+
+ def test_coarse_grid_is_dynamo_opaque(self):
+ """torch.logspace lowers to an fp64 libdevice.pow inside compiled
+ regions; inductor fused it into the batched sweep's index math where
+ Triton rejects fp64 ("unexpected type fp64") -- see the BPT worker
+ crash. The grid must be built eagerly and enter traced regions as a
+ plain tensor input (no logspace op in the traced graph)."""
+ import torch._dynamo as dynamo
+
+ from auto_round.data_type.neuqi import _coarse_grid
+
+ assert getattr(_coarse_grid, "_torchdynamo_disable", False) is True
+
+ def traced(x):
+ return _coarse_grid(0.25, 8, x.device, torch.float32) * x
+
+ ex = dynamo.explain(traced)(torch.ones(8))
+ op_names = [getattr(op, "__name__", str(op)) for ops in ex.ops_per_graph for op in ops]
+ assert not any("logspace" in n for n in op_names), f"logspace leaked into the traced graph: {op_names}"
+ assert any("mul" in n for n in op_names), "the sweep math itself should stay traced"
+
+
+class TestNeuqiBackendPolicy:
+ """Env x device gate matrices for every backend selector (pure policy:
+ no data, no kernels -- the selections themselves are covered by the
+ sweep-parity classes)."""
+
+ def test_env_device_matrix(self, monkeypatch):
+ import auto_round.data_type.neuqi as N
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "auto")
+ assert N._zp_wants_compile("cuda") is True
+ assert N._zp_wants_compile("cpu") is False and N._zp_wants_compile("hpu") is False
+ assert N._zp_batch_wanted("cuda") is True and N._zp_batch_wanted("cpu") is False
+ assert N._zp_wants_triton("cuda") is True and N._zp_wants_triton("cpu") is False
+ assert N._sym_shared_wants_triton("cuda") is True and N._sym_shared_wants_triton("cpu") is False
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")
+ assert N._zp_wants_compile("cpu") is True
+ assert N._sym_shared_wants_triton("cuda") is False
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")
+ assert N._zp_wants_compile("cuda") is False
+ assert N._zp_batch_wanted("cuda") is False # batched needs the fused backend
+ assert N._zp_wants_triton("cuda") is False
+ assert N._sym_shared_wants_triton("cuda") is False
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "triton")
+ assert N._zp_wants_triton("cpu") is True # forced (availability still checked)
+ assert N._sym_shared_wants_triton("cuda") is True
+ # unrecognized values behave as the reference sweep
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "nonsense")
+ assert N._zp_wants_compile("cuda") is False
+
+ def test_layout_policy(self):
+ import auto_round.data_type.neuqi as N
+
+ # device-automatic: the CUDA Triton fusion shape on cuda, the
+ # zero-point-last layout everywhere else
+ assert N._zp_expr_for("cuda") is N._zp_expr_last
+ assert N._zp_expr_for("cpu") is N._zp_expr_mid
+ assert N._zp_expr_for("hpu") is N._zp_expr_mid
+
+ def test_cpu_default_is_eager(self, monkeypatch):
+ """AR_NEUQI_BACKEND defaults to auto: CPU tensors never trigger a compile."""
+ import auto_round.data_type.neuqi as N
+ from auto_round import envs
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "auto")
+ monkeypatch.setattr(N, "_fused_zp_fns", {})
+ monkeypatch.setattr(N, "_fused_zp_broken", False)
+ N.neuqi_search_scale_zero(_heavy_tailed_data(n_groups=8, seed=13).clone(), bits=4, coarse_n=4, fine_n=2)
+ assert not N._fused_zp_fns # no compile attempt on CPU under auto
+
+
+class TestNeuqiItersGuard:
+ """--enable_neuqi with iters>0: the zero-shot search dispatch stays keyed
+ to iters=0 (the tuning path resolves its plain init here); the search
+ itself reaches SignRound as the wrapper's frozen initialization."""
+
+ def test_iters_dispatch(self):
+ from auto_round.data_type.utils import get_quant_func
+
+ fn, name = get_quant_func("int", 4, False, disable_opt_rtn=False, group_size=128, iters=20, enable_neuqi=True)
+ assert name == "int_asym" # tuning path initializes plain; the anchor owns the search
+ fn, name = get_quant_func("int", 4, False, disable_opt_rtn=False, group_size=128, iters=0, enable_neuqi=True)
+ assert name.startswith("opt_rtn_int_asym")
+
+
+class TestNeuqiSymSearch:
+ """Two-stage symmetric scale search (zero point fixed at 0)."""
+
+ def _skewed_data(self, n_groups=512, group_size=128, seed=0):
+ gen = torch.Generator().manual_seed(seed)
+ data = torch.randn(n_groups, group_size, generator=gen)
+ data[:, :6] *= 60.0 # outliers make the clip search matter
+ # half the groups skew negative so the mirrored clamp family is in play
+ data[::2] *= -1.0
+ return data
+
+ def _sym_loss(self, data, qw, scale):
+ """Literal loss of the shipped symmetric formula (signed scales allowed)."""
+ iq = (data / scale).round().clamp(-8, 7)
+ err = (scale * iq - data) ** 2
+ if qw is not None:
+ err = err * qw
+ return err.sum(-1)
+
+ def test_returns_signed_scales_with_magnitude_floor(self):
+ from auto_round.data_type.neuqi import neuqi_search_scale_sym
+
+ data = self._skewed_data()
+ qw = torch.rand(*data.shape) + 0.05
+ scale = neuqi_search_scale_sym(data, bits=4, qw=qw, coarse_n=32, fine_n=16)
+ assert scale.shape == (data.shape[0], 1)
+ assert (scale.abs() >= 1e-5).all()
+ # both clamp conventions must actually win somewhere on skewed data
+ assert (scale < 0).any() and (scale > 0).any()
+
+ def test_mirrored_family_matches_literal_negative_scale(self):
+ """The mirrored-clamp loss used inside the search must equal the literal
+ negative-scale evaluation of the shipped formula (guards the sign of the
+ mirrored error term)."""
+ from auto_round.data_type.neuqi import _sym_loss_chunk
+
+ torch.manual_seed(3)
+ data = torch.randn(64, 96)
+ qw = torch.rand(64, 96) + 0.1
+ scales = torch.rand(64, 8) * 0.05 + 0.005 # [C, K] positive candidates
+ loss, idx, mirror = _sym_loss_chunk(data, qw, scales, nmax=8)
+ for c in range(data.shape[0]):
+ k = idx[c].item()
+ s = scales[c, k]
+ signed = -s if mirror[c] else s
+ ref = self._sym_loss(data[c : c + 1], qw[c : c + 1], torch.tensor([[signed]]))[0]
+ assert torch.allclose(loss[c], ref, rtol=1e-4, atol=1e-3)
+
+ def test_signed_search_beats_positive_only_bruteforce_on_skewed_groups(self):
+ """Skewed groups: allowing the mirrored family must reach (near-)the signed
+ brute-force optimum, strictly below the positive-only optimum."""
+ from auto_round.data_type.neuqi import neuqi_search_scale_sym
+
+ data = self._skewed_data(128, 96, seed=7)
+ qw = torch.rand(*data.shape) + 0.05
+ scale = neuqi_search_scale_sym(data, bits=4, qw=qw, coarse_n=64, fine_n=32)
+ got = self._sym_loss(data, qw, scale)
+ # dense positive-only reference over the same anchor
+ s0 = data.abs().amax(dim=-1, keepdim=True) / 8
+ fracs = torch.logspace(-0.9, 0.3, 2000)
+ ref = torch.full_like(got, float("inf"))
+ for f in fracs:
+ l = self._sym_loss(data, qw, s0 * f)
+ ref = torch.minimum(ref, l)
+ # aggregate must win: the mirrored family is reachable, and the few
+ # two-stage basin misses (coarse grid skips a narrow basin) stay small
+ assert got.sum() <= ref.sum() * 1.001
+ # and on genuinely skewed groups it must beat the positive-only optimum
+ assert (got < ref - 1e-3).float().mean() > 0.25
+
+ def test_covers_incumbent_uniform_grid(self):
+ """Total weighted loss must not exceed search_scales (both searches see the
+ same data, weights and signed-scale families)."""
+ from auto_round.data_type.int import search_scales
+ from auto_round.data_type.neuqi import neuqi_search_scale_sym
+
+ data = self._skewed_data(512, 128, seed=11)
+ qw = torch.rand(*data.shape) + 0.05
+ s_neuqi = neuqi_search_scale_sym(data.clone(), bits=4, qw=qw.clone(), coarse_n=64, fine_n=32)
+ s_old = search_scales(data.clone(), bits=4, qw=qw.clone())
+ l_neuqi = self._sym_loss(data, qw, s_neuqi).sum().item()
+ l_old = self._sym_loss(data, qw, s_old).sum().item()
+ assert l_neuqi <= l_old * 1.001
+
+
+class TestNeuqiSymSharedCoarse:
+ """Shared-multiplier coarse pass: gating, latch-down, and driver math."""
+
+ _DEFAULTS = {"_sym_shared_triton": None, "_sym_shared_checked": False, "_sym_shared_broken": False}
+
+ @pytest.fixture(autouse=True)
+ def _reset_shared_state(self):
+ _reset_neuqi_state(self._DEFAULTS)
+ yield
+ _reset_neuqi_state(self._DEFAULTS)
+
+ def test_attempt_latches_on_failure(self, monkeypatch):
+ import auto_round.data_type.neuqi as N
+
+ _assert_launch_latches(
+ monkeypatch,
+ N,
+ attempt="_sym_shared_triton_attempt",
+ impl_attr="_sym_shared_triton",
+ checked_attr="_sym_shared_checked",
+ broken_attr="_sym_shared_broken",
+ gates={"_sym_shared_wants_triton": True},
+ tail_args=(None, None, None, 7),
+ )
+
+ def test_cpu_tensor_never_latches(self, monkeypatch):
+ import auto_round.data_type.neuqi as N
+
+ monkeypatch.delenv("AR_NEUQI_BACKEND", raising=False)
+ data = torch.randn(8, 16)
+ out = N._sym_shared_triton_attempt(data, None, None, None, 7)
+ assert out is None
+ assert not N._sym_shared_broken # gated before any launch
+
+ def test_coarse_pass_parity_and_fallback(self, monkeypatch):
+ """Driver math (dn normalization, s0^2 unnormalization, frac mapping) via an
+ eager stand-in mirrors the per-candidate path; launch None falls back."""
+ import auto_round.data_type.neuqi as N
+ from auto_round.data_type.neuqi import _sym_loss_chunk
+
+ torch.manual_seed(11)
+ n, g, bits = 700, 128, 4
+ nmax = int(2 ** (bits - 1))
+ data = torch.randn(n, g)
+ data[:, :3] *= 60.0
+ qw = torch.rand(n, g) + 0.1
+ s0 = (torch.abs(data).amax(dim=-1, keepdim=True) / nmax).clamp_(min=1e-5)
+ import math
+
+ coarse = torch.logspace(math.log10(0.25), math.log10(2.0), 96)
+
+ def eager_shared(dn, q, fracs, invf, nm):
+ # eager mirror of _sym_shared_kernel (normalized space)
+ d = dn.unsqueeze(1)
+ f = fracs.view(1, -1, 1)
+ r = torch.round(d * invf.view(1, -1, 1))
+ q_std = r.clamp(min=-nm, max=nm - 1)
+ q_mir = r.clamp(min=-(nm - 1), max=nm)
+ e_std = f * q_std - d
+ e_mir = f * q_mir - d
+ l_std = e_std * e_std
+ l_mir = e_mir * e_mir
+ if q is not None:
+ l_std = l_std * q.unsqueeze(1)
+ l_mir = l_mir * q.unsqueeze(1)
+ l_std = l_std.sum(-1)
+ l_mir = l_mir.sum(-1)
+ mirror = l_mir < l_std
+ loss = torch.where(mirror, l_mir, l_std)
+ best_loss, best_idx = loss.min(dim=-1)
+ return best_loss, best_idx, mirror.gather(1, best_idx.unsqueeze(1)).squeeze(1)
+
+ # force small chunks so the loop runs multiple iterations
+ monkeypatch.setattr(N, "_MAX_TMP_ELEMS", 2 * 96 * 128 * 4)
+ out = N._sym_coarse_pass_shared(data, qw, s0, coarse, nmax, launch=eager_shared)
+ assert out is not None
+ loss_s, frac_s, mirror_s = out
+
+ scales = s0 * coarse.view(1, -1)
+ loss_r, idx_r, mirror_r = _sym_loss_chunk(data, qw, scales, nmax)
+ torch.testing.assert_close(frac_s, coarse.index_select(0, idx_r), rtol=0, atol=0)
+ torch.testing.assert_close(mirror_s, mirror_r, rtol=0, atol=0)
+ torch.testing.assert_close(loss_s, loss_r, rtol=1e-4, atol=1e-5)
+
+ # weighted arm agrees too
+ out_w = N._sym_coarse_pass_shared(data, qw, s0, coarse, nmax, launch=eager_shared)
+ assert out_w is not None
+
+ # fallback: a launch that declines must produce None end to end
+ assert N._sym_coarse_pass_shared(data, qw, s0, coarse, nmax, launch=lambda *a: None) is None
+
+ def test_two_stage_core_uses_shared_launch(self, monkeypatch):
+ """The two-stage core routes the coarse pass through the shared helper."""
+ import auto_round.data_type.neuqi as N
+
+ torch.manual_seed(12)
+ data = torch.randn(64, 128)
+ qw = torch.rand(64, 128) + 0.1
+
+ def spy_launch(dn, q, fracs, invf, nm):
+ spy_launch.called = True
+ return None # decline -> per-candidate fallback
+
+ spy_launch.called = False
+ monkeypatch.setattr(N, "_sym_coarse_pass_shared", lambda *a, **k: (None if not spy_launch.called else None))
+ # direct wiring check: the core calls _sym_coarse_pass_shared at all
+ real = N._sym_coarse_pass_shared
+
+ def wrapper(data_, qw_, s0_, coarse_, nmax_):
+ wrapper.called = True
+ return real(data_, qw_, s0_, coarse_, nmax_)
+
+ wrapper.called = False
+ monkeypatch.setattr(N, "_sym_coarse_pass_shared", wrapper)
+ N.neuqi_search_scale_sym(data, 4, qw=qw, coarse_n=32, fine_n=8)
+ assert wrapper.called
+
+
+class TestNeuqiGridEnv:
+ """Grid resolution reaches both entries from the shared env pins. The
+ unpinned default is backend-aware (wide only on the accelerated lanes --
+ the eager sweep costs ~coarse x fine evaluations per group); env pins
+ override it everywhere and invalid pins raise at the read sites."""
+
+ @pytest.mark.parametrize("entry", ["asym", "sym"])
+ def test_env_pins_and_defaults(self, monkeypatch, entry):
+ import auto_round.data_type.neuqi as N
+
+ for k in ("AR_NEUQI_COARSE", "AR_NEUQI_FINE"):
+ monkeypatch.delenv(k, raising=False)
+ seen = {}
+ if entry == "asym":
+ monkeypatch.setattr(N, "_log_search_engaged", lambda c, f: seen.update(coarse=c, fine=f))
+ run = lambda: N.neuqi_search_scale_zero(torch.randn(4, 8), 4) # noqa: E731
+ else:
+
+ def spy(data, qw, bits, coarse, fine_n, q_scale_thresh):
+ seen.update(coarse=coarse.numel(), fine=fine_n)
+ return torch.ones(data.shape[0], 1), torch.zeros(data.shape[0])
+
+ monkeypatch.setattr(N, "_two_stage_sym_core", spy)
+ run = lambda: N.neuqi_search_scale_sym(torch.randn(4, 8), 4) # noqa: E731
+ run()
+ assert seen == {"coarse": 64, "fine": 32} # eager-lane default (CPU tensors)
+ monkeypatch.setenv("AR_NEUQI_COARSE", "96")
+ monkeypatch.setenv("AR_NEUQI_FINE", "24")
+ run()
+ assert seen == {"coarse": 96, "fine": 24} # pins override
+
+ def test_default_grid_is_backend_aware(self, monkeypatch):
+ import auto_round.data_type.neuqi as N
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "auto")
+ assert N._default_grid("cuda") == (256, 64) # accelerated lanes keep the wide grid
+ assert N._default_grid("cpu") == (64, 32) # eager lane narrows
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")
+ assert N._default_grid("cuda") == (64, 32) # forced eager narrows even on CUDA
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")
+ assert N._default_grid("cpu") == (256, 64) # forced compile keeps the wide grid
+
+ @pytest.mark.parametrize("entry", ["asym", "sym"])
+ def test_invalid_pins_raise(self, monkeypatch, entry):
+ import auto_round.data_type.neuqi as N
+
+ monkeypatch.setenv("AR_NEUQI_COARSE", "1")
+ with pytest.raises(ValueError, match="too small"):
+ if entry == "asym":
+ N.neuqi_search_scale_zero(torch.randn(4, 8), 4)
+ else:
+ N.neuqi_search_scale_sym(torch.randn(4, 8), 4)
+ monkeypatch.setenv("AR_NEUQI_COARSE", "8")
+ monkeypatch.setenv("AR_NEUQI_FINE", "1")
+ with pytest.raises(ValueError, match="too small"):
+ if entry == "asym":
+ N.neuqi_search_scale_zero(torch.randn(4, 8), 4)
+ else:
+ N.neuqi_search_scale_sym(torch.randn(4, 8), 4)
+
+
+class TestNeuqiAsymSharedCoarse:
+ """Shared-multiplier coarse pass (joint search): gating, latch, driver math."""
+
+ _DEFAULTS = {"_neuqi_shared_sweep": None, "_neuqi_shared_checked": False, "_neuqi_shared_broken": False}
+
+ @pytest.fixture(autouse=True)
+ def _reset_shared_state(self):
+ _reset_neuqi_state(self._DEFAULTS)
+ yield
+ _reset_neuqi_state(self._DEFAULTS)
+
+ def test_attempt_respects_backend_and_batch_gates(self, monkeypatch):
+ import auto_round.data_type.neuqi as N
+
+ fake = SimpleNamespace(is_cuda=True, device=torch.device("cuda"))
+ for wants, batched, should_call in ((False, True, False), (True, False, False), (True, True, True)):
+ calls = {"n": 0}
+
+ def _fn(*a, **k):
+ calls["n"] += 1
+ return "sentinel"
+
+ monkeypatch.setattr(N, "_zp_wants_triton", lambda dt: wants)
+ monkeypatch.setattr(N, "_zp_batch_wanted", lambda dt: batched)
+ monkeypatch.setattr(N, "_neuqi_shared_sweep", _fn)
+ monkeypatch.setattr(N, "_neuqi_shared_checked", True)
+ out = N._neuqi_shared_attempt(fake, None, None, None, 15)
+ if should_call:
+ assert out == "sentinel" and calls["n"] == 1
+ else:
+ assert out is None and calls["n"] == 0
+
+ def test_attempt_latches_on_failure(self, monkeypatch):
+ import auto_round.data_type.neuqi as N
+
+ _assert_launch_latches(
+ monkeypatch,
+ N,
+ attempt="_neuqi_shared_attempt",
+ impl_attr="_neuqi_shared_sweep",
+ checked_attr="_neuqi_shared_checked",
+ broken_attr="_neuqi_shared_broken",
+ gates={"_zp_wants_triton": True, "_zp_batch_wanted": True},
+ tail_args=(None, None, None, 15),
+ )
+
+ def test_cpu_tensor_never_latches(self):
+ import auto_round.data_type.neuqi as N
+
+ data = torch.randn(8, 16)
+ out = N._zp_coarse_pass_shared(data, None, torch.full((8, 1), 0.1), torch.tensor([0.5, 1.0]), 15)
+ assert out is None
+ assert not N._neuqi_shared_broken # gated before any launch
+
+ def test_coarse_pass_parity_and_fallback(self, monkeypatch):
+ """Driver math (dn normalization, s0^2 unnormalization, frac/zp mapping) via
+ an eager stand-in mirrors the per-candidate reference; launch None falls back."""
+ import auto_round.data_type.neuqi as N
+
+ torch.manual_seed(13)
+ n, g, bits = 600, 128, 4
+ maxq = 2**bits - 1
+ data = torch.randn(n, g)
+ data[:, :3] *= 70.0
+ qw = torch.rand(n, g) + 0.1
+ wmin = torch.clamp(data.min(dim=-1).values, max=0).unsqueeze(-1)
+ wmax = torch.clamp(data.max(dim=-1).values, min=0).unsqueeze(-1)
+ s0 = ((wmax - wmin) / maxq).clamp_(min=1e-5)
+ import math
+
+ coarse = torch.logspace(math.log10(0.25), 0.0, 80)
+
+ def eager_shared(dn, q, fracs, invf, mq):
+ # eager mirror of _neuqi_shared_kernel (normalized space)
+ d = dn.unsqueeze(-2).unsqueeze(-2) # [C, 1, 1, g]
+ f = fracs.view(1, -1, 1, 1) # [1, K, 1, 1]
+ zp = torch.arange(0, mq + 1, device=dn.device, dtype=dn.dtype).view(1, 1, -1, 1)
+ r = torch.round(d * invf.view(1, -1, 1, 1)).clamp_(min=-mq - 1, max=2 * mq + 1)
+ qd = (r + zp).clamp_(0, mq)
+ err = f * (qd - zp) - d
+ loss = err * err
+ if q is not None:
+ loss = loss * q.unsqueeze(-2).unsqueeze(-2)
+ loss = loss.sum(dim=-1) # [C, K, Z]
+ best_z, zp_arg = loss.min(dim=-1)
+ best, k_arg = best_z.min(dim=1) # first-minimum over k
+ zp_best = zp_arg.gather(1, k_arg.unsqueeze(1)).squeeze(1)
+ return best, k_arg, zp_best
+
+ # force small chunks so the loop runs multiple iterations
+ monkeypatch.setattr(N, "_MAX_TMP_ELEMS", 262144 * 3)
+ loss_s, frac_s, zp_s = N._zp_coarse_pass_shared(data, qw, s0, coarse, maxq, launch=eager_shared)
+
+ # per-candidate reference over the identical grid
+ scales = s0 * coarse.view(1, -1) # [C, K]
+ d = data.unsqueeze(-2).unsqueeze(-2)
+ zp = torch.arange(0, maxq + 1).view(1, 1, -1, 1)
+ r = torch.round(d / scales.unsqueeze(-1).unsqueeze(-1)).clamp_(min=-maxq - 1, max=2 * maxq + 1)
+ qd = (r + zp).clamp_(0, maxq)
+ err = scales.unsqueeze(-1).unsqueeze(-1) * (qd - zp) - d
+ loss = err * err * qw.unsqueeze(-2).unsqueeze(-2)
+ loss = loss.sum(dim=-1) # [C, K, Z]
+ best_z, zp_arg = loss.min(dim=-1)
+ loss_ref, k_ref = best_z.min(dim=1)
+ zp_ref = zp_arg.gather(1, k_ref.unsqueeze(1)).squeeze(1)
+
+ torch.testing.assert_close(frac_s, coarse.index_select(0, k_ref), rtol=0, atol=0)
+ torch.testing.assert_close(zp_s, zp_ref.to(torch.float32), rtol=0, atol=0)
+ torch.testing.assert_close(loss_s, loss_ref, rtol=1e-4, atol=1e-4)
+
+ # fallback: a launch that declines must produce None end to end
+ assert N._zp_coarse_pass_shared(data, qw, s0, coarse, maxq, launch=lambda *a: None) is None
+
+ def test_search_wires_shared_coarse(self, monkeypatch):
+ """neuqi_search_scale_zero routes its coarse pass through the shared helper."""
+ import auto_round.data_type.neuqi as N
+
+ torch.manual_seed(14)
+ data = torch.randn(64, 128)
+ qw = torch.rand(64, 128) + 0.1
+
+ def wrapper(data_, qw_, s0_, coarse_, maxq_):
+ wrapper.called = True
+ return None # decline -> batched fallback still correct
+
+ wrapper.called = False
+ monkeypatch.setattr(N, "_zp_coarse_pass_shared", wrapper)
+ N.neuqi_search_scale_zero(data, 4, qw=qw, coarse_n=32, fine_n=8)
+ assert wrapper.called
+
+
+class TestNeuqiSymTritonSweep:
+ """Triton sym-search dispatch: gating, latch-down, and parity where CUDA exists."""
+
+ @pytest.fixture(autouse=True)
+ def _reset_backend_state(self):
+ import auto_round.data_type.neuqi as N
+
+ for attr in ("_sym_triton", "_sym_triton_checked", "_sym_triton_broken"):
+ if hasattr(N, attr):
+ setattr(N, attr, None if attr != "_sym_triton_broken" else False)
+ N._sym_triton_checked = False
+ N._sym_search_warmed.clear()
+ N._sym_chunk_compiled = None
+ N._sym_compile_failed = False
+ yield
+ import auto_round.data_type.neuqi as N2
+
+ N2._sym_triton_broken = False
+ N2._sym_triton_checked = False
+ N2._sym_triton = None
+
+ def test_eval_on_cpu_matches_reference(self):
+ from auto_round.data_type.neuqi import _sym_loss_chunk, _sym_search_eval
+
+ torch.manual_seed(3)
+ data = torch.randn(96, 32)
+ scales = torch.rand(96, 7) + 0.5
+ qw = torch.rand(96, 32) + 0.1
+ for q in (qw, None):
+ ref = _sym_loss_chunk(data, q, scales, 7)
+ out = _sym_search_eval(data, q, scales, 7)
+ torch.testing.assert_close(out[0], ref[0], rtol=1e-5, atol=1e-6)
+ mism = (out[1] != ref[1]).float().mean().item()
+ assert mism < 5e-3, f"candidate index mismatch fraction {mism}"
+ mism_m = (out[2] != ref[2]).float().mean().item()
+ assert mism_m < 5e-2, f"convention mismatch fraction {mism_m}"
+
+ def test_attempt_skips_and_latches_on_failure(self, monkeypatch):
+ import auto_round.data_type.neuqi as N
+
+ # a CUDA-shaped stand-in: the gate only consults is_cuda/device before
+ # the launch attempt (real-CUDA parity is covered by the skipif test)
+ _assert_launch_latches(
+ monkeypatch,
+ N,
+ attempt="_sym_triton_attempt",
+ impl_attr="_sym_triton",
+ checked_attr="_sym_triton_checked",
+ broken_attr="_sym_triton_broken",
+ gates={"_sym_wants_triton": True},
+ tail_args=(None, None, 7),
+ )
+
+ def test_cpu_never_reaches_triton(self, monkeypatch):
+ import auto_round.data_type.neuqi as N
+
+ def _boom(*a, **k):
+ raise AssertionError("CPU tensors must never reach the Triton kernel")
+
+ monkeypatch.setattr(N, "_sym_triton", _boom)
+ monkeypatch.setattr(N, "_sym_triton_checked", True)
+ out = N._sym_search_eval(torch.randn(8, 32), None, torch.rand(8, 2) + 0.5, 7)
+ assert out[0].shape == (8,)
+
+ def test_wrapper_rejects_cpu_tensors(self):
+ pytest.importorskip("auto_round_extension.triton.neuqi_sweep")
+ from auto_round_extension.triton.neuqi_sweep import sym_search_triton
+
+ with pytest.raises(AssertionError):
+ sym_search_triton(torch.randn(4, 32), None, torch.rand(4, 2) + 0.5, 7)
+
+ @pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA for the Triton kernel")
+ def test_triton_matches_reference_on_cuda(self):
+ from auto_round.data_type.neuqi import _sym_loss_chunk
+ from auto_round_extension.triton.neuqi_sweep import sym_search_triton
+
+ torch.manual_seed(4)
+ dev = torch.device("cuda")
+ data = torch.randn(4096, 128, device=dev)
+ data[:, :5] *= 120.0 # outlier columns like real weights
+ scales = torch.rand(4096, 33, device=dev) + 0.25
+ qw = torch.rand(4096, 128, device=dev) + 0.1
+ for q in (qw, None):
+ ref = _sym_loss_chunk(data, q, scales, 7)
+ out = sym_search_triton(data.contiguous(), q.contiguous() if q is not None else None, scales, 7)
+ torch.testing.assert_close(out[0], ref[0], rtol=1e-4, atol=1e-5)
+ mism = (out[1] != ref[1]).float().mean().item()
+ assert mism < 1e-2, f"candidate index mismatch fraction {mism}"
+ mism_m = (out[2] != ref[2]).float().mean().item()
+ assert mism_m < 5e-2, f"convention mismatch fraction {mism_m}"
+
+
+class TestNeuqiSymCompiledCore:
+ """The compiled sym chunk core: parity with eager, backend gating, latch-down."""
+
+ @pytest.fixture(autouse=True)
+ def _reset_core_state(self):
+ import auto_round.data_type.neuqi as N
+
+ N._sym_chunk_compiled = None
+ N._sym_compile_failed = False
+ N._sym_compile_logged.clear()
+ yield
+ N._sym_chunk_compiled = None
+ N._sym_compile_failed = False
+
+ def test_compiled_matches_eager(self, monkeypatch):
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")
+ torch.manual_seed(0)
+ from auto_round.data_type.neuqi import _sym_loss_chunk, _sym_loss_chunk_eval
+
+ data = torch.randn(64, 32)
+ scales = torch.rand(64, 5) + 0.5
+ for qw in (torch.rand(64, 32) + 0.1, None):
+ out_e = _sym_loss_chunk(data, qw, scales, 7)
+ out_c = _sym_loss_chunk_eval(data, qw, scales, 7)
+ torch.testing.assert_close(out_c[0], out_e[0], rtol=1e-5, atol=1e-6)
+ assert torch.equal(out_c[1], out_e[1])
+ assert torch.equal(out_c[2], out_e[2])
+ # where the toolchain allows inductor codegen the compiled core must
+ # have ENGAGED (not silently latched down to eager); environments
+ # without a C++ compiler (e.g. Windows without MSVC) latch by design
+ import auto_round.data_type.neuqi as N
+
+ if not N._sym_compile_failed:
+ assert N._sym_chunk_compiled is not None
+
+ def test_eager_backend_never_compiles(self, monkeypatch):
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")
+ from auto_round.data_type import neuqi
+
+ def _boom(*a, **k):
+ raise AssertionError("torch.compile must not be called under backend=eager")
+
+ monkeypatch.setattr(neuqi.torch, "compile", _boom)
+ out = neuqi._sym_loss_chunk_eval(torch.randn(8, 32), None, torch.rand(8, 2) + 0.5, 7)
+ assert out[0].shape == (8,)
+
+ def test_failure_latches_down_to_eager(self, monkeypatch):
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")
+ from auto_round.data_type import neuqi
+
+ calls = {"n": 0}
+
+ def _flaky_compile(fn):
+ def _raise(*a, **k):
+ calls["n"] += 1
+ raise RuntimeError("inductor unavailable")
+
+ return _raise
+
+ monkeypatch.setattr(neuqi.torch, "compile", _flaky_compile)
+ data, scales = torch.randn(8, 32), torch.rand(8, 2) + 0.5
+ for _ in range(2): # second call must go straight to eager
+ out = neuqi._sym_loss_chunk_eval(data, None, scales, 7)
+ assert torch.isfinite(out[0]).all()
+ assert calls["n"] == 1
+ assert neuqi._sym_compile_failed
+
+ def test_search_end_to_end_matches_eager_backend(self, monkeypatch):
+ """The compiled core routes every chunk through; results must match an
+ eager-backend run."""
+ from auto_round.data_type.neuqi import neuqi_search_scale_sym
+
+ torch.manual_seed(1)
+ data = torch.randn(128, 32)
+
+ def _run():
+ return neuqi_search_scale_sym(data.clone(), 4, coarse_n=8, fine_n=4)
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "eager")
+ eager_plain = _run()
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "compile")
+ comp_plain = _run()
+ torch.testing.assert_close(comp_plain, eager_plain, rtol=0, atol=0)
+
+
+class TestNeuqiSymDynamoGuard:
+ def test_sym_search_never_traced_by_dynamo(self):
+ """The sym search is loop-heavy and eager-shaped; under
+ --enable_torch_compile it must be skipped by dynamo (tracing it unrolls
+ the chunk loops into giant graphs - minutes of CPU compile, GPU idle)."""
+ from auto_round.data_type.neuqi import _two_stage_sym_core, neuqi_search_scale_sym
+
+ for fn in (neuqi_search_scale_sym, _two_stage_sym_core):
+ assert getattr(fn, "_torchdynamo_disable", False), fn.__name__
+
+
+class TestNeuqiSymIntegration:
+ def test_sym_neuqi_dispatch(self):
+ fn, name = get_quant_func("int", 4, True, disable_opt_rtn=False, group_size=32, iters=0, enable_neuqi=True)
+ assert name == "opt_rtn_int_sym_neuqi"
+ assert "opt_rtn_int_sym_neuqi" in QUANT_FUNC_WITH_DTYPE
+
+ def test_wrapper_qdq_matches_returned_signed_scale(self):
+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_sym_neuqi
+
+ torch.manual_seed(13)
+ w = torch.randn(32, 256)
+ w[:, :20] *= -40.0 # negative-heavy columns exercise the mirrored family
+ qdq, scale, zp = quant_tensor_opt_rtn_sym_neuqi(w, bits=4, group_size=32)
+ assert zp == 8
+ assert scale.shape == (32 * 256 // 32, 1)
+ s2d = torch.repeat_interleave(scale.reshape(32, 8), 32, dim=1)
+ manual = (w / s2d).round().clamp(-8, 7) * s2d
+ assert torch.allclose(qdq, manual, rtol=1e-3, atol=1e-4)
+ assert (scale < 0).any() # mirrored family won somewhere
+
+ def test_sym_wrapper_supports_imatrix_off_1d_2d(self):
+ """qw=None (imatrix off), per-column 1-D and per-row 2-D weights all flow;
+ weighting must steer the search (lower weighted loss than the unweighted
+ solution under the same weights)."""
+ from auto_round.data_type.neuqi import neuqi_search_scale_sym, quant_tensor_opt_rtn_sym_neuqi
+
+ torch.manual_seed(17)
+ w = torch.randn(32, 256)
+ w[:, :24] *= -40.0
+ for im in (None, torch.rand(256) + 0.05, torch.rand(32, 256) + 0.05):
+ qdq, scale, _ = quant_tensor_opt_rtn_sym_neuqi(w.clone(), bits=4, group_size=32, imatrix=im)
+ assert torch.isfinite(qdq).all()
+
+ d = w.reshape(-1, 32)
+ qw = (torch.rand(32, 256) + 0.05).reshape(-1, 32)
+ s_q = neuqi_search_scale_sym(d.clone(), bits=4, qw=qw.clone())
+ s_p = neuqi_search_scale_sym(d.clone(), bits=4, qw=None)
+
+ def wloss(s):
+ iq = (d / s).round().clamp(-8, 7)
+ return ((s * iq - d) ** 2 * qw).sum().item()
+
+ assert wloss(s_q) < wloss(s_p)
+
+
+class _FakeQuantLayer(torch.nn.Module):
+ """Minimal stand-in for a quantisable layer (see also hadamard test_patch)."""
+
+ def __init__(self, out_features=8, in_features=32, bits=4, sym=False, group_size=32):
+ super().__init__()
+ self.bits = bits
+ self.sym = sym
+ self.group_size = group_size
+ self.data_type = "int"
+ self.act_bits = 16
+ self.act_data_type = "int"
+ self.act_sym = True
+ self.act_dynamic = True
+ self.iters = 200
+ self.tuning_device = "cpu"
+ torch.manual_seed(7)
+ self.weight = torch.nn.Parameter(torch.randn(out_features, in_features))
+ self.bias = None
+ self.imatrix = None
+
+
+class TestNeuqiFrozenInit:
+ """enable_neuqi on the tuning path: the search anchors the STE grid
+ once at wrapper init and the range margins stay pinned at 1.0."""
+
+ def _wrapper(self, enable_neuqi, sym=False, enable_round_tuning=True):
+ from auto_round.wrapper import WrapperLinear
+
+ layer = _FakeQuantLayer(sym=sym)
+ wrapper = WrapperLinear(
+ layer,
+ device="cpu",
+ enable_minmax_tuning=True,
+ enable_norm_bias_tuning=False,
+ enable_round_tuning=enable_round_tuning,
+ enable_torch_compile=False,
+ disable_opt_rtn=True,
+ enable_neuqi=enable_neuqi,
+ iters=5,
+ )
+ return wrapper, layer
+
+ def test_neuqi_anchor_replaces_minmax_asym(self):
+ from auto_round.data_type.neuqi import neuqi_search_scale_zero
+
+ wrapper, layer = self._wrapper(True, sym=False)
+ w = layer.weight.data.to(torch.float32).reshape(-1, layer.group_size)
+ s, zp = neuqi_search_scale_zero(w, layer.bits)
+ maxq = int(2**layer.bits) - 1
+ assert torch.allclose(wrapper.weight_min, -(zp * s).squeeze(-1), atol=1e-6)
+ assert torch.allclose(wrapper.weight_max, ((maxq - zp) * s).squeeze(-1), atol=1e-6)
+ # frozen margins: the searched grid IS the init; SignRound tunes values only
+ assert wrapper._neuqi_frozen_margins is True
+ # non-tunable params become plain tensors, not entries in params
+ assert "min_scale" not in wrapper.params and "max_scale" not in wrapper.params
+
+ def test_sym_anchor_uses_two_stage_search(self):
+ from auto_round.data_type.int import quant_tensor_sym
+ from auto_round.data_type.neuqi import neuqi_search_scale_sym, quant_tensor_opt_rtn_sym_neuqi
+
+ wrapper, layer = self._wrapper(True, sym=True)
+ w = layer.weight.data.to(torch.float32).reshape(-1, layer.group_size)
+ s_signed = neuqi_search_scale_sym(w, layer.bits)
+ nmax = int(2 ** (layer.bits - 1))
+ mag = (s_signed.abs() * nmax).squeeze(-1)
+ mirrored = (s_signed < 0).squeeze(-1)
+ # the STE tie-break maps wmax_abs < wmin_abs to the POSITIVE scale:
+ # the losing side is stored at HALF magnitude so the comparison
+ # resolves to the winner's convention in any storage dtype
+ assert torch.allclose(wrapper.weight_min, torch.where(mirrored, -mag / 2, -mag))
+ assert torch.allclose(wrapper.weight_max, torch.where(mirrored, mag, mag / 2))
+ assert wrapper._neuqi_frozen_margins is True
+ # R1-1/R2-1 regression guard: the anchored grid must reproduce the
+ # search's winning qdq exactly, per group, under BOTH clamp conventions
+ ref_qdq, _, _ = quant_tensor_opt_rtn_sym_neuqi(w, bits=layer.bits, group_size=layer.group_size)
+ ste_qdq, ste_scale, _ = quant_tensor_sym(
+ w,
+ bits=layer.bits,
+ group_size=layer.group_size,
+ min_scale=torch.tensor(1.0),
+ max_scale=torch.tensor(1.0),
+ tensor_min=wrapper.weight_min,
+ tensor_max=wrapper.weight_max,
+ )
+ assert torch.equal(ste_qdq.reshape(ref_qdq.shape), ref_qdq) or torch.allclose(
+ ste_qdq.reshape(ref_qdq.shape), ref_qdq, atol=1e-7
+ ), "anchored STE grid diverges from the search winner"
+
+ def test_sym_anchor_survives_low_precision_storage(self, monkeypatch):
+ # R2-1: the anchor is cast to the weight dtype; a sub-precision bias
+ # (2^-20) would be erased by bf16/fp16 rounding and flip every
+ # standard-convention group to the mirrored grid. The half-magnitude
+ # bias must reproduce the search qdq for bf16 and fp16 weights too.
+ import auto_round.wrapper as wrapper_mod
+ from auto_round.data_type.int import quant_tensor_sym
+ from auto_round.data_type.neuqi import neuqi_search_scale_sym, quant_tensor_opt_rtn_sym_neuqi
+
+ for dtype in (torch.bfloat16, torch.float16):
+ layer = _FakeQuantLayer(sym=True)
+ layer.weight.data = layer.weight.data.to(torch.float32).to(dtype)
+ wrapper = wrapper_mod.WrapperLinear(
+ layer,
+ device="cpu",
+ enable_minmax_tuning=True,
+ enable_round_tuning=True,
+ enable_torch_compile=False,
+ disable_opt_rtn=True,
+ enable_neuqi=True,
+ iters=5,
+ )
+ assert wrapper.weight_min.dtype == dtype
+ w32 = layer.weight.data.to(torch.float32).reshape(-1, layer.group_size)
+ s_signed = neuqi_search_scale_sym(w32, layer.bits)
+ ref_qdq, _, _ = quant_tensor_opt_rtn_sym_neuqi(w32, bits=layer.bits, group_size=layer.group_size)
+ ste_qdq, ste_scale, _ = quant_tensor_sym(
+ layer.weight.data,
+ bits=layer.bits,
+ group_size=layer.group_size,
+ min_scale=torch.tensor(1.0),
+ max_scale=torch.tensor(1.0),
+ tensor_min=wrapper.weight_min,
+ tensor_max=wrapper.weight_max,
+ )
+ ste_qdq = ste_qdq.to(torch.float32).reshape(ref_qdq.shape)
+ # (1) the R2-1 invariant: no group's clamp convention flips (the
+ # pre-fix failure was a systematic all-group sign flip)
+ ste_sign = torch.sign(ste_scale.reshape(-1))
+ want_sign = torch.sign(s_signed.reshape(-1))
+ assert torch.equal(
+ ste_sign, want_sign
+ ), f"{dtype}: convention flipped on {(ste_sign != want_sign).sum()} groups"
+ # (2) residual diffs are bf16/fp16 boundary-rounding jitter of the
+ # STE division on low-precision weights, bounded by one quantum
+ diff = (ste_qdq - ref_qdq.to(torch.float32)).abs()
+ quantum = s_signed.abs().reshape(-1, 1)
+ assert (diff <= quantum + 1e-6).all(), f"{dtype}: diff {(diff.max()):.3e} exceeds one quantum"
+
+ def test_zeroshot_asym_uses_joint_search(self):
+ """Zero-shot optimized path, asymmetric: the wrapper resolves the
+ joint (scale, zp) search quant func, and its qdq equals the direct
+ search result exactly (the search IS the quant function here)."""
+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_asym
+ from auto_round.wrapper import WrapperLinear
+
+ layer = _FakeQuantLayer(sym=False)
+ layer.iters = 0
+ wrapper = WrapperLinear(
+ layer,
+ device="cpu",
+ enable_minmax_tuning=False,
+ enable_norm_bias_tuning=False,
+ enable_round_tuning=False,
+ enable_torch_compile=False,
+ disable_opt_rtn=False,
+ enable_neuqi=True,
+ iters=0,
+ )
+ assert wrapper.data_type == "opt_rtn_int_asym"
+ w = layer.weight.data.to(torch.float32)
+ ref_qdq, _, _ = quant_tensor_opt_rtn_asym(w, bits=layer.bits, group_size=layer.group_size)
+ got_qdq, _, _ = wrapper.weight_quant_func(layer.weight.data)
+ assert torch.allclose(
+ got_qdq.to(torch.float32).reshape(ref_qdq.shape), ref_qdq.to(torch.float32), atol=1e-7
+ ), "zero-shot asym qdq diverges from the joint search result"
+
+ # fp16 weights: the division runs in the weight dtype (mirroring the
+ # search function), so parity must hold there too
+ layer_fp16 = _FakeQuantLayer(sym=False)
+ layer_fp16.iters = 0
+ layer_fp16.weight.data = layer_fp16.weight.data.to(torch.float16)
+ wrapper_fp16 = WrapperLinear(
+ layer_fp16,
+ device="cpu",
+ enable_minmax_tuning=False,
+ enable_norm_bias_tuning=False,
+ enable_round_tuning=False,
+ enable_torch_compile=False,
+ disable_opt_rtn=False,
+ enable_neuqi=True,
+ iters=0,
+ )
+ ref16, _, _ = quant_tensor_opt_rtn_asym(
+ layer_fp16.weight.data, bits=layer_fp16.bits, group_size=layer_fp16.group_size
+ )
+ got16, _, _ = wrapper_fp16.weight_quant_func(layer_fp16.weight.data)
+ assert torch.equal(got16, ref16), "fp16 zero-shot asym qdq diverges from the joint search result"
+
+ def test_zeroshot_sym_uses_neuqi_search(self):
+ """Zero-shot optimized path, symmetric: the wrapper resolves the
+ two-stage NeUQI scale search (not the uniform search_scales grid),
+ and its qdq equals quant_tensor_opt_rtn_sym_neuqi's result."""
+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_sym_neuqi
+ from auto_round.wrapper import WrapperLinear
+
+ layer = _FakeQuantLayer(sym=True)
+ layer.iters = 0
+ wrapper = WrapperLinear(
+ layer,
+ device="cpu",
+ enable_minmax_tuning=False,
+ enable_norm_bias_tuning=False,
+ enable_round_tuning=False,
+ enable_torch_compile=False,
+ disable_opt_rtn=False,
+ enable_neuqi=True,
+ iters=0,
+ )
+ assert wrapper.data_type == "opt_rtn_int_sym_neuqi"
+ w = layer.weight.data.to(torch.float32)
+ ref_qdq, ref_scale, _ = quant_tensor_opt_rtn_sym_neuqi(w, bits=layer.bits, group_size=layer.group_size)
+ got_qdq, _, _ = wrapper.weight_quant_func(layer.weight.data)
+ assert torch.allclose(
+ got_qdq.to(torch.float32).reshape(ref_qdq.shape), ref_qdq.to(torch.float32), atol=1e-7
+ ), "zero-shot sym qdq diverges from the two-stage NeUQI search"
+ assert ref_scale is not None
+ assert "min_scale" not in wrapper.params and "max_scale" not in wrapper.params
+
+ def test_anchor_stays_off_when_inappropriate(self):
+ """Flag off: plain min/max with tunable margins. Zero-shot path: the
+ search runs inside the quant-func dispatch, so the wrapper-side anchor
+ must not fire there either."""
+ wrapper, layer = self._wrapper(False, sym=False)
+ w = layer.weight.data.reshape(-1, layer.group_size)
+ assert torch.allclose(wrapper.weight_min, w.min(1)[0].clamp(max=0), atol=1e-7)
+ assert torch.allclose(wrapper.weight_max, w.max(1)[0].clamp(min=0), atol=1e-7)
+ assert wrapper._neuqi_frozen_margins is False
+ assert wrapper.params["min_scale"].requires_grad is True
+ assert wrapper.params["max_scale"].requires_grad is True
+
+ wrapper, _ = self._wrapper(True, enable_round_tuning=False)
+ assert wrapper.weight_min is None and wrapper.weight_max is None
+ assert wrapper._neuqi_frozen_margins is False
+
+
+class TestSignRoundEnableNeuqi:
+ """SignRoundConfig carries enable_neuqi; True = the frozen-init anchor."""
+
+ def test_config_flag(self):
+ from auto_round.algorithms.quantization.sign_round.config import SignRoundConfig
+
+ assert SignRoundConfig(bits=4, group_size=32).enable_neuqi is False
+ assert SignRoundConfig(bits=4, group_size=32, sym=False, enable_neuqi=True).enable_neuqi is True
+
+
+class TestExpertBatching:
+ """Same-shape expert projections quantize in one stacked search call
+ (row-independent: results identical to per-module calls)."""
+
+ @pytest.fixture(autouse=True)
+ def _tie_free_grid(self, monkeypatch):
+ # exact stacked-vs-per-module equality holds while the grid produces no
+ # fp-summation near-ties for the seeded data; the narrow eager default
+ # (64/32) hits ties that first-min resolves differently between the
+ # batched expression and the per-candidate expression (the sweep parity
+ # tests cover the tie semantics) -- pin the wide grid this exact-equality
+ # contract was verified against
+ monkeypatch.setenv("AR_NEUQI_COARSE", "256")
+ monkeypatch.setenv("AR_NEUQI_FINE", "64")
+
+ def teardown_method(self):
+ import auto_round.algorithms.quantization.rtn.quantizer as _qmod
+
+ if "model_context" in _qmod.OptimizedRTNQuantizer.__dict__:
+ del _qmod.OptimizedRTNQuantizer.model_context
+
+ def _expert(self, name, out=16, inn=32, bits=4, group_size=32, sym=False, seed=0):
+ m = torch.nn.Linear(inn, out, bias=False)
+ torch.manual_seed(seed)
+ with torch.no_grad():
+ m.weight.copy_(torch.randn(out, inn))
+ m.global_name = name
+ m.bits = bits
+ m.group_size = group_size
+ m.sym = sym
+ m.data_type = "int"
+ return m
+
+ def _quantizer(self, enable_neuqi=True, disable_opt_rtn=False, is_moe=True):
+ from auto_round.algorithms.quantization.rtn.config import OptimizedRTNConfig
+ from auto_round.algorithms.quantization.rtn.quantizer import OptimizedRTNQuantizer
+
+ cfg = OptimizedRTNConfig(bits=4, group_size=32, sym=False, enable_neuqi=enable_neuqi)
+ cfg.disable_opt_rtn = disable_opt_rtn
+ cfg.orig_disable_opt_rtn = False if not disable_opt_rtn else True
+ q = OptimizedRTNQuantizer(cfg)
+ if is_moe:
+ # model_context is a read-only property backed by the run ctx;
+ # patch the class property for the MoE-heuristic branch
+ import auto_round.algorithms.quantization.rtn.quantizer as _qmod
+
+ _qmod.OptimizedRTNQuantizer.model_context = property(lambda self: type("MC", (), {"is_moe_model": True})())
+ return q
+
+ def test_sym_and_asym_batched_and_equivalent(self):
+ # THE sym gap fix: sym modules join the batches under neuqi, and the
+ # stacked call matches per-module search outputs exactly
+ q = self._quantizer()
+ mods = [self._expert(f"blk.experts.{i}.gate_proj", sym=False, seed=i) for i in range(4)]
+ sym_mods = [self._expert(f"blk.experts.{i}.up_proj", sym=True, seed=100 + i) for i in range(4)]
+ batches, singles = q._split_expert_batches(mods + sym_mods)
+ assert len(batches) == 2 and not singles
+ assert {m.sym for m in batches[0]} == {False} and {m.sym for m in batches[1]} == {True}
+
+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_asym, quant_tensor_opt_rtn_sym_neuqi
+
+ assert q._quantize_expert_batch(batches[0], "cpu") == []
+ for m in mods:
+ ref_qdq, ref_scale, ref_zp = quant_tensor_opt_rtn_asym(
+ m.weight.data, bits=m.bits, group_size=m.group_size, v=0.0, q_scale_thresh=1e-5, imatrix=None
+ )
+ assert torch.equal(m.weight.data, ref_qdq)
+ assert torch.equal(m.scale.reshape(-1), ref_scale.reshape(-1))
+ assert torch.equal(m.zp.reshape(-1), ref_zp.reshape(-1))
+ assert m.q_scale_thresh == 1e-5
+
+ assert q._quantize_expert_batch(batches[1], "cpu") == []
+ for m in sym_mods:
+ ref_qdq, ref_scale, ref_zp = quant_tensor_opt_rtn_sym_neuqi(
+ m.weight.data, bits=m.bits, group_size=m.group_size, v=0.0, q_scale_thresh=1e-5, imatrix=None
+ )
+ assert torch.equal(m.weight.data, ref_qdq)
+ assert torch.equal(m.scale.reshape(-1), ref_scale.reshape(-1))
+ assert m.zp == ref_zp
+ # R2-5: the batch writeback must stamp the resolved data_type both classes
+ assert all(m.data_type == "opt_rtn_int_asym" for m in mods)
+ assert all(m.data_type == "opt_rtn_int_sym_neuqi" for m in sym_mods)
+
+ def test_imatrix_weighting_stacked_equivalence(self):
+ q = self._quantizer()
+ mods = [self._expert(f"blk.experts.{i}.down_proj", sym=False, seed=7 + i) for i in range(3)]
+ for i, m in enumerate(mods):
+ torch.manual_seed(50 + i)
+ m.imatrix = torch.rand(m.weight.shape[1])
+ batches, singles = q._split_expert_batches(mods)
+ assert len(batches) == 1 and not singles
+ assert q._quantize_expert_batch(batches[0], "cpu") == []
+ from auto_round.data_type.neuqi import quant_tensor_opt_rtn_asym
+
+ for m in mods:
+ ref_qdq, ref_scale, ref_zp = quant_tensor_opt_rtn_asym(
+ m.weight.data,
+ bits=m.bits,
+ group_size=m.group_size,
+ v=0.0,
+ q_scale_thresh=1e-5,
+ imatrix=m.imatrix,
+ )
+ assert torch.equal(m.weight.data, ref_qdq)
+ assert torch.equal(m.scale.reshape(-1), ref_scale.reshape(-1))
+
+ def test_batch_splitting_rules(self):
+ """One setup for every _split_expert_batches rule: gating, expert-name
+ filtering, shape/act-quant/symmetry segregation."""
+ q = self._quantizer()
+
+ # gating: off -> inactive; disable_opt_rtn -> inactive
+ assert self._quantizer(enable_neuqi=False)._expert_search_active() is False
+ assert self._quantizer(enable_neuqi=True, disable_opt_rtn=True)._expert_search_active() is False
+
+ # non-expert and odd-shaped modules stay single
+ dense = self._expert("blk.mlp.gate_proj")
+ odd = self._expert("blk.experts.0.gate_proj", inn=48, group_size=32) # in % g != 0 is fine for grouping
+ mixed_shape = self._expert("blk.experts.1.gate_proj", inn=64)
+ batches, singles = q._split_expert_batches([dense, odd, mixed_shape])
+ assert not batches and {dense, odd, mixed_shape} == set(singles)
+
+ # act quantization needs the per-module wrapper path
+ m = self._expert("blk.experts.0.gate_proj")
+ m.act_bits = 8
+ batches, singles = q._split_expert_batches([m])
+ assert not batches and singles == [m]
+
+ # same (parent, proj, shape, bits, gs) but different symmetry: not
+ # mergeable into one stacked call -> both stay single (group of 1)
+ a = self._expert("blk.experts.0.gate_proj", sym=False)
+ sym_m = self._expert("blk.experts.1.gate_proj", sym=True)
+ batches, singles = q._split_expert_batches([a, sym_m])
+ assert not batches and len(singles) == 2
+
+ def test_oom_returns_only_unwritten(self, monkeypatch):
+ # R2-2: a mid-batch OOM must leave already-written modules untouched
+ # and return ONLY the unwritten remainder (per-module fallback must
+ # never re-quantize a qdq'd weight)
+ import auto_round.algorithms.quantization.rtn.quantizer as Q
+ import auto_round.data_type.neuqi as N
+
+ q = self._quantizer()
+ mods = [self._expert(f"blk.experts.{i}.gate_proj", seed=i) for i in range(3)]
+ batches, singles = q._split_expert_batches(mods)
+ assert len(batches) == 1
+
+ calls = []
+ orig = N.quant_tensor_opt_rtn_asym
+
+ def oom_after_first(weights, *a, **kw):
+ calls.append(1)
+ if len(calls) > 1:
+ raise torch.OutOfMemoryError("simulated")
+ return orig(weights, *a, **kw)
+
+ monkeypatch.setattr(N, "quant_tensor_opt_rtn_asym", oom_after_first)
+ # one module per stacked call -> the failure hits mid-batch
+ monkeypatch.setattr(Q, "_EXPERT_BATCH_MAX_ELEMS", mods[0].weight.numel())
+ before = mods[0].weight.data.clone()
+ remaining = q._quantize_expert_batch(batches[0], "cpu")
+ assert len(remaining) == 2, "modules after the failed chunk must be returned unwritten"
+ assert remaining == mods[1:]
+ # the first chunk was written and must NOT be re-quantized later
+ assert not torch.equal(mods[0].weight.data, before)
+ assert hasattr(mods[0], "scale") and mods[0].data_type == "opt_rtn_int_asym"
+ # unwritten modules are untouched
+ assert not hasattr(mods[1], "scale") and not hasattr(mods[2], "scale")
+
+ def test_act_quant_modules_stay_single(self):
+ q = self._quantizer()
+ m = self._expert("blk.experts.0.gate_proj")
+ m.act_bits = 8 # act quantization needs the per-module wrapper path
+ batches, singles = q._split_expert_batches([m])
+ assert not batches and singles == [m]
+
+ def test_sym_key_separates_batches(self):
+ q = self._quantizer()
+ a = self._expert("blk.experts.0.gate_proj", sym=False)
+ s = self._expert("blk.experts.1.gate_proj", sym=True)
+ batches, singles = q._split_expert_batches([a, s])
+ # same (parent, proj, shape, bits, gs) but different symmetry: not
+ # mergeable into one stacked call -> both stay single (group of 1)
+ assert not batches and len(singles) == 2
+
+
+class TestImatrixAutoRule:
+ """asym + neuqi must auto-collect the imatrix (the search consumes qw);
+ plain asym keeps it off (min/max init ignores it)."""
+
+ def test_rule(self):
+ from auto_round.algorithms.quantization.rtn.config import OptimizedRTNConfig, RTNConfig
+ from auto_round.autoround import _select_rtn_compressor_base_cls
+
+ cfg = RTNConfig(bits=4, group_size=128, sym=False, enable_neuqi=True)
+ _select_rtn_compressor_base_cls(cfg, scheme=None, format="auto_round", base_kwargs={})
+ assert cfg.enable_imatrix is True and isinstance(cfg, OptimizedRTNConfig)
+
+ cfg = RTNConfig(bits=4, group_size=128, sym=False, enable_neuqi=False)
+ _select_rtn_compressor_base_cls(cfg, scheme=None, format="auto_round", base_kwargs={})
+ assert cfg.enable_imatrix is False
+
+
+class TestEnableNeuqiCliFlag:
+ """The CLI gate is --enable_neuqi (per the governing PR design); the
+ boolean enable_neuqi kwarg is the API surface."""
+
+ def test_cli_flag_registration(self):
+ from auto_round.algorithms.quantization.rtn.config import RTNConfig
+ from auto_round.algorithms.quantization.sign_round.config import SignRoundConfig
+
+ reg = RTNConfig.get_registered_args()
+ p = next(p for p in reg.parameters if "--enable_neuqi" in p.option_strings)
+ assert p.field == "enable_neuqi"
+ assert p.argparse_kwargs["action"] == "store_true"
+ assert any("--enable_neuqi" in q.option_strings for q in SignRoundConfig.get_registered_args().parameters)
diff --git a/test/unit/test_cuda/quantization/test_neuqi_e2e.py b/test/unit/test_cuda/quantization/test_neuqi_e2e.py
new file mode 100644
index 000000000..ca001b794
--- /dev/null
+++ b/test/unit/test_cuda/quantization/test_neuqi_e2e.py
@@ -0,0 +1,75 @@
+# Copyright (c) 2026 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""End-to-end NeUQI runs through the public AutoRound API on CUDA.
+
+The CPU tier covers the search math and dispatch tables; these runs verify
+the full wiring (config routing, imatrix collection, frozen-init anchor,
+layer_config output) with the REAL backend ladder engaged -- on CUDA the
+Triton/compile sweep paths serve the search instead of the eager fallback,
+so default grids (256/64) run in seconds. Hosts without CUDA skip."""
+
+import pytest
+import torch
+
+pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+_DS = ["local NeUQI calibration sample with enough tokens for quantization"] * 2
+
+
+def _autoround(model_path, sym, iters):
+ from auto_round import AutoRound
+
+ return AutoRound(
+ model_path,
+ bits=4,
+ group_size=32,
+ sym=sym,
+ enable_neuqi=True,
+ iters=iters,
+ nsamples=2,
+ seqlen=8,
+ batch_size=2,
+ dataset=list(_DS),
+ device_map="cuda:0",
+ enable_torch_compile=False,
+ )
+
+
+def test_zero_shot_neuqi_e2e(tiny_opt_model_path, tmp_path):
+ ar = _autoround(tiny_opt_model_path, sym=False, iters=0)
+ model, layer_config = ar.quantize()
+ assert model is not None and len(layer_config) > 0
+ for name, cfg in layer_config.items():
+ assert cfg["bits"] == 4, f"Layer {name} expected bits=4, got {cfg['bits']}"
+ assert cfg["data_type"] == "int", f"Layer {name} expected int, got {cfg.get('data_type')}"
+
+
[email protected]("sym", [False, True])
+def test_frozen_init_e2e(tiny_opt_model_path, monkeypatch, sym):
+ """Both symmetry classes share the tuning wiring; only the anchor's
+ internals (joint vs two-stage search) differ."""
+ import auto_round.data_type.neuqi as neuqi_mod
+
+ calls = []
+ orig_anchor = neuqi_mod.neuqi_anchor_margins
+
+ def spy(weight, *a, **kw):
+ calls.append(tuple(weight.shape))
+ return orig_anchor(weight, *a, **kw)
+
+ monkeypatch.setattr(neuqi_mod, "neuqi_anchor_margins", spy)
+ ar = _autoround(tiny_opt_model_path, sym=sym, iters=1)
+ model, layer_config = ar.quantize()
+ assert model is not None and len(layer_config) > 0
+ assert calls, f"the NeUQI frozen-init anchor never fired on the iters>0 path (sym={sym})"
diff --git a/test/unit/test_cuda/quantization/test_neuqi_triton.py b/test/unit/test_cuda/quantization/test_neuqi_triton.py
new file mode 100644
index 000000000..4450813a5
--- /dev/null
+++ b/test/unit/test_cuda/quantization/test_neuqi_triton.py
@@ -0,0 +1,76 @@
+# Copyright (c) 2026 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""CUDA parity test for the NeUQI Triton sweep kernel against the eager reference.
+
+The kernel rounds with round-half-to-even and keeps the first-minimum zero-point
+rule, so selections must match the eager per-candidate sweep up to the usual
+fp32 summation ties (fp64-oracle checked). Skipped on hosts without CUDA+triton.
+"""
+
+import pytest
+import torch
+
+from auto_round.data_type.neuqi import neuqi_search_scale_zero
+
+triton_sweep = pytest.importorskip("auto_round_extension.triton.neuqi_sweep").neuqi_sweep_triton
+
+pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+
+def _oracle_loss_fp64(data, qw, scale, zp, maxq):
+ r = torch.round(data.double() / scale.double()).clamp_(min=-maxq - 1, max=2 * maxq + 1)
+ q = (r + zp.double()).clamp_(0, maxq)
+ deq = scale.double() * (q - zp.double())
+ loss = (deq - data.double()) ** 2
+ if qw is not None:
+ loss = loss * qw.double()
+ return loss.sum(-1)
+
+
[email protected]("bits", [2, 3, 4, 8])
[email protected]("weighted", [False, True])
[email protected]("g", [32, 64, 100]) # 100: non-power-of-two group size (masked)
+def test_triton_sweep_matches_eager(bits, weighted, g, monkeypatch):
+ gen = torch.Generator().manual_seed(bits * 13 + int(weighted) * 7 + g)
+ n = 512
+ data = torch.randn(n, g, generator=gen)
+ data[:, :5] *= 90.0
+ qw = torch.rand(n, g, generator=gen) + 0.1 if weighted else None
+ data_c, qw_c = data.cuda(), (qw.cuda() if qw is not None else None)
+
+ scale_ref, zp_ref = neuqi_search_scale_zero(data.clone(), bits, qw=qw, coarse_n=8, fine_n=3)
+
+ import auto_round.data_type.neuqi as N
+ from auto_round import envs
+
+ monkeypatch.setenv("AR_NEUQI_BACKEND", "triton") # envs reads os.getenv per access; setenv auto-restores
+ saved_batch = N._zp_batch_wanted
+ N._zp_batch_wanted = lambda dev: True # batching is device-automatic now; force it for the parity run
+ try:
+ N._triton_checked, N._triton_sweep, N._triton_broken = True, triton_sweep, False
+ N._fused_zp_fns, N._fused_zp_broken, N._batched_broken = {}, False, False
+ scale_t, zp_t = neuqi_search_scale_zero(data_c, bits, qw=qw_c, coarse_n=8, fine_n=3)
+ finally:
+ N._zp_batch_wanted = saved_batch
+
+ assert not N._triton_broken, "kernel raised; fallback engaged instead"
+ scale_ref, zp_ref = scale_ref.cpu(), zp_ref.cpu()
+ scale_t, zp_t = scale_t.cpu(), zp_t.cpu()
+
+ mismatch = (zp_ref != zp_t).logical_or(scale_ref != scale_t).nonzero().flatten()
+ maxq = 2**bits - 1
+ for i in mismatch.tolist():
+ loss_ref = _oracle_loss_fp64(data[i], qw[i] if qw is not None else None, scale_ref[i], zp_ref[i], maxq)
+ loss_t = _oracle_loss_fp64(data[i], qw[i] if qw is not None else None, scale_t[i], zp_t[i], maxq)
+ assert (loss_t - loss_ref).abs().item() <= 1e-4 * loss_ref.abs().item()