Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

update ARK README (#1922)

Luo Yuf42a04216eb0
diff --git a/auto_round_extension/ark/README.md b/auto_round_extension/ark/README.md
index a9d8eb63a..6e7a7254c 100644
--- a/auto_round_extension/ark/README.md
+++ b/auto_round_extension/ark/README.md
@@ -1,246 +1,187 @@
-## What is AutoRound Kernel?
+## What is AutoRound Kernel (ARK)?
-AutoRound Kernel (ARK) is a low-bit acceleration library for Intel platform, providing high-performance kernels for LLM inference including weight-only quantization, flash attention (with [SageAttention](https://github.com/thu-ml/SageAttention) v1 support), and Mixture-of-Experts (MOE) computation.
+AutoRound Kernel (ARK) is a low-bit acceleration library for Intel platform, providing three categories of optimized operators for LLM inference.
-The kernels are optimized for the following CPUs:
-* Intel Xeon Scalable processor (formerly Sapphire Rapids, and Emerald Rapids)
-* Intel Xeon 6 processors (formerly Sierra Forest and Granite Rapids)
+| Operator Category | CPU | XPU (Battlemage) |
+|:------------------|:---:|:----------------:|
+| **Weight-Only Quantized Linear** (INT4/INT8/FP8/FP4) | ✅ | ✅ |
+| **MoE Grouped GEMM** | ❌ | ✅ |
+| **SageAttention** (SDPA / SAGE) | ❌ | ✅ |
-The kernels are optimized for the following GPUs:
-* Intel Arc B-Series Graphics and Intel Arc Pro B-Series Graphics
- (formerly Battlemage)
+**Validated CPU:** Intel Xeon Scalable (Sapphire Rapids / Emerald Rapids), Intel Xeon 6 (Sierra Forest / Granite Rapids)<br>
+**Validated GPU:** Intel Arc B-Series / Arc Pro B-Series (Battlemage)
-## Key Features
+### Highlights — Ecosystem Integration
-AutoRound Kernel provides the following computational capabilities for LLM inference:
+ARK kernels are integrated into the following projects:
-### Weight-Only Quantization (WOQ) Linear
-| Weight dtype | Compute dtype | Scale dtype | Algorithm<sup>[1]</sup> |
-|------------------|:--------------------------------:|:-----------------:|:-----------------------:|
-| INT8 | INT8<sup>[2]</sup> / BF16 / FP32 | BF16 / FP32 | sym / asym |
-| INT4 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |
-| INT3 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |
-| INT2 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |
-| INT5 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |
-| INT6 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |
-| INT7 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |
-| INT1 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |
-| FP8 (E4M3, E5M2) | BF16 / FP32 | FP32 / FP8 (E8M0) | NA |
-| FP4 (E2M1) | BF16 / FP32 | BF16 / FP32 | NA |
+| Project | Integration | Description |
+|---------|-------------|-------------|
+| [vllm](https://github.com/vllm-project/vllm) | [`inc_wna16_linear.py`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py#L379-L470) | `INCXPUARKLinearMethod` — weight-only quantized linear on XPU via `auto_round_kernel.qlinear.QuantLinear`. The goal is to support all AutoRound-quantized models. The plan is tracked in [Intel Quantization Support Roadmap](https://github.com/vllm-project/vllm/issues/37979) |
+| [vllm-omni](https://github.com/vllm-project/vllm-omni) | [`sage_attn.py`](https://github.com/vllm-project/vllm-omni/blob/f8340d078e4e9c3b793bd92d55d891b29703f0a8/vllm_omni/diffusion/attention/backends/sage_attn.py#L27) | `SageAttentionBackend` — diffusion model attention on XPU via `ARK.sagev1` |
+| [Transformers](https://github.com/huggingface/transformers) (via [auto-round](https://github.com/intel/auto-round)) | [`backend.py`](https://github.com/intel/auto-round/blob/main/auto_round/inference/backend.py#L531) | All models quantized by AutoRound automatically use ARK on CPU/XPU by default; no additional configuration required. 6 backends registered: `auto_round_kernel[_xpu]` (GPTQ no-zp), `auto_round_kernel_zp[_xpu]` (GPTQ +zp), `auto_round_kernel_awq[_xpu]` (AWQ).<br>**CPU:** INT2/INT4/INT8; **XPU:** INT4/INT8 |
-### XPU Weight-Only Quantization
-| Weight dtype | Compute dtype | Scale dtype | Algorithm |
-|------------------|:--------------:|:-----------------:|:----------:|
-| INT8 | INT8 / FP16 | FP16 | sym |
-| INT4 | INT8 / FP16 | FP16 | sym |
-| INT2 | INT8 / FP16 | FP16 | sym |
-| FP8 (E4M3, E5M2) | FP16 | FP16 / FP8 (E8M0) | NA |
+---
-<sup>[1]</sup>: Quantization algorithms for integer types: symmetric or asymmetric.
-<sup>[2]</sup>: Includes dynamic activation quantization; results are dequantized to floating-point formats.
+## 1. Linear (Weight-Only Quantized GEMM)
-### Flash Attention (XPU only)
+Low-bit weight-only linear for LLM inference. Both CPU and XPU are supported.
-ARK provides multiple attention backends for prefill and decode, with [SageAttention](https://github.com/thu-ml/SageAttention) v1 support — offering INT8-quantized attention variants for higher throughput on Intel Arc GPUs. (SageAttention v2/v3 are under development.)
+### API
-| Backend | Description | Q/K/V dtype | Head dim | Features |
-|---------|-------------|-------------|----------|----------|
-| `sdpa` | Standard flash attention (FP16/BF16) | FP16 / BF16 | 64, 96, 128, 192 | Causal mask, additive mask, GQA |
-| `sagev1` | **SageAttention v1** with INT8 Q/K quantization | FP16 / BF16 | 64, 128 | Block-wise INT8 QK, PV in half |
-| `sagev1_pvi8` | **SageAttention v1** with INT8 Q/K/V quantization | FP16 / BF16 | 64, 128 | Block-wise INT8 QK + INT8 PV |
-| `sage` | Low-level SageAttention with pre-quantized INT8 Q/K | INT8 (Q/K), FP16 (V) | 64, 128 | External Q/K scales |
-| `sage_pvi8` | Low-level SageAttention with pre-quantized INT8 Q/K/V | INT8 (Q/K/V) | 64, 128 | External Q/K/V scales |
-| `sage_dynquant` | SageAttention with fused dynamic INT8 quantization | FP16 | 64, 128 | Auto-quantizes Q/K internally |
+| API | Description | Platform |
+|-----|-------------|----------|
+| `QuantLinear` ([example ↓](#example)) | Unified PyTorch module (GPTQ/AWQ/raw quantized checkpoint) | CPU / XPU |
+| `QuantLinearGPTQ` | GPTQ-format checkpoint loader | CPU / XPU |
+| `QuantLinearAWQ` | AWQ-format checkpoint loader | CPU / XPU |
+| `QuantLinearFP8` | FP8 weight-only linear | CPU / XPU |
+| `woqgemm` | Low-level weight-only GEMM (packed format) | CPU / XPU |
+| `woqgemm_s8` | Low-level INT8-weight GEMM with scale | CPU / XPU |
+| `_repack_quantized_weight` | Repack raw qweight/qzero/scale → ARK format | CPU / XPU |
+| `_unpack_weight` | Unpack ARK-format weight back to full precision | CPU / XPU |
-All attention backends support both `HND` (`[B, H, N, D]`) and `NHD` (`[B, N, H, D]`) tensor layouts, as well as non-contiguous (sliced) input tensors.
+### Key Features
-### MOE GEMM (XPU only)
+> **W4A8 / W2A8 Rescale (QQQ-style)** — On XPU, ARK supports \~[QQQ](https://github.com/HandH1998/QQQ)-style compute: low-bit weights (INT2/INT4) are re-scaled to INT8 and computed via INT8 GEMM, avoiding FP16 dequantization for better throughput. Enabled automatically via environment variable `ARK_AUTO_S8`; see [xpu_wrapper.hpp](auto_round_kernel/wrapper/include/xpu_wrapper.hpp).
-Grouped GEMM for Mixture-of-Experts layers, supporting FP16/BF16 with variable token counts per expert.
+### Supported Data Types
-## Installation
-
-### 1. Install via pip
-```bash
-pip install auto-round-lib
-```
-
-### 2. Install from Source
-
-Requires a sourced oneAPI environment (2025.3+ recommended for SYCL TLA support).
-
-```bash
-# Source oneAPI environment
-source /opt/intel/oneapi/setvars.sh
-
-# Build and install
-pip install . --no-build-isolation
-# or
-python setup.py bdist_wheel; pip install dist/*
-```
+#### CPU
-The build system automatically detects the oneAPI version. SYCL TLA (Tensor Linear Algebra) kernels are enabled when oneAPI >= 2025.3.
+| Weight dtype | Compute dtype | Scale dtype | Algorithm |
+|--------------|:-------------:|:-----------:|:---------:|
+| INT1–INT8 | INT8<sup>[1]</sup> / BF16 / FP32 | BF16 / FP32 | sym / asym |
+| FP8 (E4M3, E5M2) | BF16 / FP32 | FP32 / FP8 (E8M0) | NA |
+| FP4 (E2M1) | BF16 / FP32 | BF16 / FP32 | NA |
-### Validated Hardware Environment
+#### XPU
-#### CPU based on [Intel 64 architecture or compatible processors](https://en.wikipedia.org/wiki/X86-64):
-* Intel Xeon Scalable processor (Granite Rapids)
+| Weight dtype | Compute dtype | Scale dtype | Algorithm |
+|--------------|:-------------:|:-----------:|:---------:|
+| INT4, INT8 | INT8 / FP16 | FP16 | sym |
+| FP8 (E4M3, E5M2) | FP16 | FP16 / FP8 (E8M0) | NA |
-#### GPU built on Intel's Xe architecture:
-* Intel Arc B-Series Graphics (Battlemage)
+<sup>[1]</sup> INT8 compute includes dynamic activation quantization; results are dequantized to floating-point.
-## API Reference
-
-### QuantLinear (Weight-Only Quantization)
-
-ARK exposes a unified weight-only linear interface through `QuantLinear`, `QuantLinearGPTQ`, `QuantLinearAWQ`, and `QuantLinearFP8`. Please refer to the [QLinear](auto_round_kernel/qlinear.py) for more integration details.
-
-The expected lifecycle is: create the module, load quantized tensors from the checkpoint, call `post_init()` once to repack weights into the ARK-friendly layout, and then call `forward()` during inference.
-
-Minimal usage:
-```python
-from auto_round_kernel.qlinear import QuantLinear
-
-qlinear = QuantLinear(
- bits=4,
- group_size=128,
- sym=True,
- in_features=in_features,
- out_features=out_features,
- bias=bias is not None,
- weight_dtype=weight_dtype,
-)
-# Load qweight, qzeros, scales, and bias from checkpoint.
-qlinear.post_init()
-
-# Run inference
-y = qlinear(x)
-```
-
-### Attention APIs
-
-#### `ark.sdpa` — Standard Flash Attention
+### Example
```python
import auto_round_kernel as ark
-output = ark.sdpa(
- query,
- key,
- value,
- attn_mask=None,
- dropout_p=0.0,
- is_causal=False,
- scale=None,
- tensor_layout="HND", # "HND" or "NHD"
+# Prepare quantized weight: qweight [K, N] int4/int2, scale [K/G, N] fp16/fp32, zp [K/G, N] int4/int2
+packw = ark.repack_quantized_weight(
+ qweight,
+ scale,
+ zp,
+ blocksize=128,
+ compute_type="fp16",
+ weight_type="int4",
+ scale_type="fp16",
+ asym=False,
)
-```
-- Q/K/V dtype: FP16 or BF16
-- Supported head dims: 64, 96, 128, 192
-- Supports GQA (grouped query attention) via different Hq/Hkv
-
-#### `ark.sagev1` — SAGE v1 Attention
+# Run weight-only quantized GEMM: activation [M, K] → output [M, N]
+output = ark.woqgemm(
+ activation, # [M, K] fp16/bf16
+ packw, # packed weight blob (INT8)
+ bias, # [1, N] optional bias
+ n, # output features
+ k, # input features
+ groupsize=128,
+ compute_type="fp16",
+ weight_type="int4",
+ scale_type="fp16",
+ asym=False,
+)
-```python
-output = ark.sagev1(
- query,
- key,
- value,
- attn_mask=None,
- dropout_p=0.0,
- is_causal=False,
- scale=None,
- enable_gqa=False,
- quant_block_size=64, # block size for INT8 QK quantization
- tensor_layout="HND",
+# Decompose back to full precision for verification
+decompressed = ark.unpack_weight(
+ packw,
+ dtype=torch.float16,
+ n=n,
+ k=k,
+ groupsize=128,
+ compute_type="fp16",
+ weight_type="int4",
+ scale_type="fp16",
+ asym=False,
)
```
-- Q/K/V dtype: FP16 or BF16
-- Supported head dims: 64, 128
-- Internally quantizes Q/K to INT8 per block; PV computed in half precision
-- Falls back to `ark.sdpa` when `quant_block_size <= 0`
+See [test_weightonly.py](test/test_weightonly.py) for an end-to-end example of weight repack, verification, and woqgemm execution on CPU and XPU.
-#### `ark.sagev1_pvi8` — SAGE v1 with INT8 PV
+---
-Same interface as `sagev1` but also quantizes V to INT8 internally for higher throughput.
+## 2. MoE (Mixture-of-Experts Grouped GEMM)
-#### `ark.sage_dynquant` — SAGE with Fused Dynamic Quantization
+Grouped GEMM for MoE layers where different experts process varying numbers of tokens.
-```python
-output = ark.sage_dynquant(
- query,
- key,
- value,
- attn_mask=None,
- dropout_p=0.0,
- is_causal=False,
- scale=None,
- enable_gqa=False,
- quant_block_size=64,
-)
-```
+### API
+
+| Function | Description | Platform | Activation Dtype | Weight Dtype |
+|----------|-------------|----------|:----------------:|:------------:|
+| `ark.moe_gemm(...)` ([example ↓](#example-1)) | Grouped GEMM across experts | XPU | FP16 / BF16 | FP16 / BF16 |
+| `ark.moe_gemm(...)` (WIP) | Grouped GEMM with INT4 weight | XPU | FP16 / BF16 | INT4 🚧 |
+| `ark.moe_gemm(...)` (WIP) | Grouped GEMM with INT2 weight | XPU | FP16 / BF16 | INT2 🚧 |
+| `ark.moe_gemm(...)` (WIP) | Grouped GEMM with INT8 weight | XPU | FP16 / BF16 | INT8 🚧 |
-- Takes FP16 Q/K/V, performs fused block-wise INT8 quantization of Q/K via SYCL kernel
-- Supports quant_block_size: 1 (per-token), 32, 64, 128, 256
-- Auto-pads sequence lengths for block alignment
+> 🚧 INT2 / INT4 / INT8 weight support is under active development. See [#PR](https://github.com/intel/auto-round/pull).
-#### `ark.sageattn` — SageAttention-Compatible Dispatcher
+### Details
-ARK provides a drop-in replacement for the `sageattention.sageattn` API, enabling seamless integration with existing SageAttention workflows:
+| Parameter | Shape | Dtype |
+|-----------|-------|-------|
+| activations | `[total_tokens, K]` | FP16 / BF16 |
+| weights | `[num_experts, K, N]` (row-major) | FP16 / BF16 |
+| num_tokens_per_expert | `[num_experts]` | INT32 |
+| scales (optional) | `[num_experts, N]` | FP16 / BF16 |
+| **output** | `[total_tokens, N]` | same as activations |
+
+### Example
```python
-import auto_round_kernel as ark
+# FP16/BF16 MoE
+output = ark.moe_gemm(activations, weights, num_tokens_per_expert)
-# Drop-in replacement for sageattention.sageattn
-output = ark.sageattn(
- q,
- k,
- v,
- tensor_layout="HND",
- is_causal=False,
- sm_scale=None,
- return_lse=False,
- kernel="v1_pvhalf", # or "v1_pvi8"
- **kwargs,
-)
+# INT4 MoE (coming soon)
+# output = ark.moe_gemm(activations, q4_weights, num_tokens_per_expert, scales=scales)
```
-This mirrors the [SageAttention](https://github.com/thu-ml/SageAttention) interface for drop-in compatibility, allowing models using SageAttention to run on Intel Arc GPUs without code changes.
+Build requirement: `ARK_SYCL_TLA=ON`. See [test_moe.py](test/test_moe.py).
-### MOE GEMM
+---
-```python
-output = ark.moe_gemm(
- activations, # [total_tokens, K] FP16/BF16
- weights, # [num_experts, K, N] FP16/BF16
- num_tokens_per_expert, # [num_experts] int32
- scales=None, # optional [num_experts, N] FP16/BF16
-)
-```
+## 3. SageAttention (XPU SDPA Acceleration)
-### Patching torch SDPA
+ARK provides a full family of scaled dot-product attention kernels on XPU, ranging from vanilla FP16 SDPA to INT8-quantized SageAttention variants.
-ARK can globally replace `torch.nn.functional.scaled_dot_product_attention` for evaluation, including SageAttention backends:
+### API Overview
-```python
-import auto_round_kernel as ark
-
-# Patch with standard SDPA backend
-ark.patch_torch_sdpa(backend="sdpa")
+| Function | Description | Q/K/V Input | PV Precision | Head Dim |
+|----------|-------------|-------------|:------------:|:--------:|
+| `ark.sdpa` ([example ↓](#drop-in-sdpa-replacement)) | FP16/BF16 SDPA (flash attention) | FP16 / BF16 | FP16 | 64, 96, 128, 192 |
+| `ark.sage` | Low-level INT8 SAGE (pre-quantized Q/K) | INT8 (Q/K), FP16 (V) | FP16 | 64, 128 |
+| `ark.sage_pvi8` | Low-level INT8 SAGE (pre-quantized Q/K/V) | INT8 | INT8 | 64, 128 |
+| `ark.sagev1` | High-level FP16 → internal Q/K quant → SAGE | FP16 / BF16 | FP16 | 64, 128 |
+| `ark.sagev1_pvi8` | High-level FP16 → internal Q/K/V quant → SAGE PV INT8 | FP16 / BF16 | INT8 | 64, 128 |
+| `ark.sageattn` | Dispatcher (sageattention-compatible API) | FP16 / BF16 | FP16 / INT8 | 64, 128 |
+| `ark.sage_dynquant` | Dynamic INT8 block-wise Q/K quant → SAGE (drop-in SDPA replacement) | FP16 / BF16 | FP16 | 64, 128 |
-# Patch with SageAttention v1 backend (INT8 QK, PV half)
-ark.patch_torch_sdpa(backend="sagev1", quant_block_size=64)
+### Comparison
-# Patch with SageAttention v1 + INT8 PV backend
-ark.patch_torch_sdpa(backend="sagev1_pvi8", quant_block_size=64)
+| Feature | `sdpa` | `sagev1` | `sagev1_pvi8` | `sage_dynquant` |
+|---------|:------:|:--------:|:-------------:|:---------------:|
+| Q/K quantization | None | Internal INT8 | Internal INT8 | Internal INT8 |
+| PV quantization | None | None | Internal INT8 | None |
+| quant_block_size | N/A | 1 / ≥32 | 1 / ≥32 | 1 / ≥32 |
+| Additive mask | ✅ [B,1,Sq,Skv] FP32 | ✅ | ✅ | ✅ |
+| Causal mask | ✅ | ✅ | ✅ | ✅ |
+| GQA | ✅ | ✅ | ✅ | ✅ |
+| Tensor layout | HND / NHD | HND / NHD | HND / NHD | HND |
-# Restore original
-ark.unpatch_torch_sdpa()
-```
+### Drop-in SDPA Replacement
-Or use the helper launcher for lm-eval:
+Replace `torch.nn.functional.scaled_dot_product_attention` globally for lm-eval:
```bash
cd /path/to/auto_round_extension/ark
@@ -248,46 +189,52 @@ PYTHONPATH=$PWD python tools/lm_eval_with_ark_sdpa.py \
--model hf \
--model_args pretrained=/path/to/model,trust_remote_code=True,dtype=bfloat16 \
--tasks hellaswag,piqa,winogrande \
- --device xpu:0 \
- --batch_size 1
+ --device xpu:0 --batch_size 1
```
-### Low-Level Matrix Operations
+The patching logic (in [`auto_round_kernel/torch_sdpa_patch.py`](auto_round_kernel/torch_sdpa_patch.py)) routes to ARK on XPU when constraints are met; otherwise falls back to PyTorch SDPA.
-```python
-# FP16/BF16 matrix multiply with bias
-C = ark.matmul(A, B, bias)
+### Constraints
+
+| Constraint | `sdpa` | `sagev1` / `sagev1_pvi8` / `sage_dynquant` |
+|------------|:------:|:------------------------------------------:|
+| Q/K/V dtype | FP16, BF16 | FP16, BF16 |
+| Head dim | 64, 96, 128, 192 | 64, 128 |
+| `dropout_p` | must be 0.0 | must be 0.0 |
+| Boolean mask | falls back to torch | falls back to torch |
+| Additive mask shape | `[B, 1, Sq, Skv]` FP32 | `[B, 1, Sq, Skv]` FP32 |
+| quant_block_size | N/A | 1 (per-token) or ≥32 |
-# INT8 matrix multiply (s8s8s32)
-C = ark.igemm_s8s8s32(A, B)
+---
+
+## Installation
-# Weight-only quantized GEMM with INT8 weights
-C = ark.woqgemm_s8(A, B, scaleB, bias)
+### Install via pip
+```bash
+pip install auto-round-lib
+```
-# General weight-only quantized GEMM
-C = ark.woqgemm(A, B, bias, n, k, groupsize, compute_type, weight_type, scale_type, asym)
+### Install from Source
+```bash
+pip install . --no-build-isolation
+# or
+python setup.py bdist_wheel; pip install dist/*
```
-## Testing
-
-Unit tests are available in the [test](test/) directory:
-
-| Test file | Description |
-|-----------|-------------|
-| `test_weightonly.py` | Weight-only quantized GEMM (CPU + XPU) |
-| `test_flash_attn.py` | Flash attention (sdpa) correctness |
-| `test_sdpa.py` | SDPA benchmark suite |
-| `test_sdpa_parity.py` | SDPA parity with non-contiguous inputs and layouts |
-| `test_sage_dynquant.py` | SageDynQuant block-wise benchmark |
-| `test_bench_bmg.py` | BMG comparison benchmark |
-| `test_matmul.py` | General matrix multiply |
-| `test_packq.py` | Weight packing/unpacking |
-| `test_moe.py` | MOE GEMM correctness |
-
-## Notes
-
-- The SDPA patch only routes calls to ARK on XPU when inputs match kernel constraints; otherwise it falls back to the original torch SDPA.
-- Supported Q/K/V dtypes for attention are FP16 and BF16 (except SAGE variants which may use INT8 internally).
-- `dropout_p` must be 0.0 for all ARK attention paths.
-- Additive masks are supported when they can be normalized to `[B, 1, Sq, Skv]`; boolean masks fall back to torch.
-- Non-contiguous (sliced) input tensors are supported for all attention backends.
+Build with MoE / SageAttention support requires `ARK_SYCL_TLA=ON`.
+
+---
+
+## Tests
+
+| Test | Description |
+|------|-------------|
+| [test_weightonly.py](test/test_weightonly.py) | WOQ GEMM pack/unpack/run on CPU & XPU |
+| [test_moe.py](test/test_moe.py) | MoE grouped GEMM |
+| [test_flash_attn.py](test/test_flash_attn.py) | SDPA (flash attention) prefill |
+| [test_sdpa.py](test/test_sdpa.py) | SDPA benchmark suite |
+| [test_sdpa_parity.py](test/test_sdpa_parity.py) | SDPA vs PyTorch parity check |
+| [test_sage_dynquant.py](test/test_sage_dynquant.py) | SageAttention dynamic INT8 quant benchmarks |
+| [test_bench_bmg.py](test/test_bench_bmg.py) | BMG SDPA / SageAttention benchmarking |
+| [test_matmul.py](test/test_matmul.py) | Low-level matmul |
+| [test_packq.py](test/test_packq.py) | Weight packing utilities |