Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

9.8 KB
Raw Blame

SVDQuant (Experimental)

SVDQuant splits a linear weight into a quantized residual branch and a small floating-point low-rank branch:

W ~= Q(R) + U @ V
Linear(x) ~= QuantizedLinear(x, Q(R)) + Linear(Linear(x, V), U)

This feature follows the decomposition proposed in SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models. AutoRound can combine the transform with RTN or SignRound and export Nunchaku-loadable MXFP4 FLUX and SDXL pipelines.

The architecture adapters are strict: FLUX is validated for FLUX.1-dev and the SDXL adapter targets stabilityai/stable-diffusion-xl-base-1.0. Stable Diffusion 1.x UNets are not handled by the SDXL adapter.

Quick start

Install AutoRound and its diffusion dependencies in an isolated environment:

pip install --no-build-isolation -e .
pip install -r test/test_cuda/requirements_diffusion.txt

Default RTN workflow

The recommended starting point uses rank 32, one residual iteration, no smooth search, and plain RTN:

CUDA_VISIBLE_DEVICES=0 auto-round-rtn \
  --model /path/to/FLUX.1-dev \
  --model_dtype bf16 \
  --scheme MXFP4 \
  --algorithm svdquant \
  --format svdquant_nunchaku \
  --device 0 \
  --output_dir ./flux-dev-mxfp4-svdquant-rtn

When svdquant is selected and --format is omitted, the CLI also defaults to svdquant_nunchaku.

SDXL base 1.0 RTN workflow

SDXL uses a UNet rather than a Diffusers transformer component. Auto detection selects the SDXL adapter from the UNet config; it can also be selected explicitly:

CUDA_VISIBLE_DEVICES=0 auto-round-rtn \
  --model /path/to/stable-diffusion-xl-base-1.0 \
  --model_dtype bf16 \
  --scheme MXFP4 \
  --algorithm svdquant \
  --svdquant-model-adapter sdxl \
  --format svdquant_nunchaku \
  --device 0 \
  --output_dir ./sdxl-base-1.0-mxfp4-svdquant-rtn

Only the projections replaced by Nunchaku are quantized. SDXL self-attention Q/K/V projections share one low-rank down factor and are exported as fused to_qkv. Cross-attention K/V, convolutions, normalization layers, and the remaining UNet state stay in BF16.

Quality results for SDXL base 1.0:

Format CLIP ↑ CLIP-IQA ↑ ImageReward ↑
BF16 26.8583 0.929027 0.812544
MXFP4 26.9006 0.895704 0.697131
MXFP4 (SVDQuant) 26.8088 0.927907 0.7967

Default SignRound workflow

Use SignRound as the terminal quantizer by selecting auto_round:

CUDA_VISIBLE_DEVICES=0 auto-round \
  --model /path/to/FLUX.1-dev \
  --model_dtype bf16 \
  --scheme MXFP4 \
  --algorithm svdquant,auto_round \
  --format svdquant_nunchaku \
  --dataset /path/to/coco2017-captions.tsv \
  --batch_size 1 \
  --device 0 \
  --low_gpu_mem_usage \
  --output_dir ./flux-dev-mxfp4-svdquant-signround

The caption dataset is a tab-separated file with id and caption columns:

id	caption
0	A photo of a cat
1	A city street at night

The COCO2017 calibration captions are available from changwangss/coco2017.

Optional quality settings

Smooth search and additional residual iterations are opt-in:

--enable-svdquant-smooth \
--svdquant-smooth-num-grids 20 \
--svdquant-smooth-max-calibration-calls 128 \
--svdquant-residual-iters 20 \
--enable-svdquant-residual-early-stop

For diffusion calibration, --nsamples controls prompts and --calib_num_inference_steps controls the number of inference steps requested when the scheduler builds its native short calibration schedule. Higher-order schedulers may expand these into more internal timesteps. Start small before running a full quality configuration.

Processing flow

SVDQuant is a structural preprocessor and must run before one terminal quantizer:

Original linear weights
  -> optional activation-aware smooth search
  -> group related projections
  -> low-rank decomposition and residual iteration
  -> replace nn.Linear with SVDQuantLinear
       -> quantized residual_linear
       -> floating-point lora_down
       -> floating-point lora_up
       -> optional input smooth factor
  -> RTN or SignRound
  -> Nunchaku export

FLUX and SDXL self-attention Q/K/V projections are grouped so they share one low-rank down factor and retain separate up factors, matching their runtime layouts.

Residual iteration

With the default residual_iters=1:

  1. Compute a rank-limited decomposition of the grouped weight W.
  2. Materialize U and V in low_rank_dtype.
  3. Form R = W - U @ V.
  4. Let the terminal RTN or SignRound stage quantize the residual linear.

With residual_iters > 1, each outer iteration alternates between fitting the low-rank branch to W - Q(R_previous) and applying deployment-compatible RTN QDQ to the new residual. Early stop retains the best candidate and stops when the objective becomes worse.

The objective is weight reconstruction error without smooth search and calibration output error with smooth search. Residual iterations are separate from SignRound --iters.

Smooth search

For G = --svdquant-smooth-num-grids, the candidates are:

(alpha, beta) = (0, 0)
(alpha, 0) for alpha in {1/G, ..., (G-1)/G}
(alpha, 1-alpha) for the same alpha values

The default G=20 produces 39 candidates. Each candidate constructs x_span**alpha / w_span**beta and is scored against floating-point group outputs. The lowest finite output error wins.

--svdquant-smooth-max-calibration-calls uses reservoir sampling to bound the retained calls per group. Calls that are not selected are not copied into the smooth CPU cache. This limit does not replace SignRound calibration samples.

Configuration

Python field CLI option Default Meaning
rank --svdquant-rank 32 Low-rank branch rank.
smooth_enabled --enable-svdquant-smooth False Enable Alpha/Beta search.
smooth_num_grids --svdquant-smooth-num-grids 20 Candidate grid resolution.
smooth_max_calibration_calls --svdquant-smooth-max-calibration-calls 128 Retained calls per group.
residual_iters --svdquant-residual-iters 1 Residual outer iterations.
residual_early_stop --enable-svdquant-residual-early-stop False Stop after the objective worsens.
low_rank_dtype --svdquant-low-rank-dtype bf16 BF16, FP16, or FP32 factors.
target_modules --svdquant-target-modules None Included module-name substrings.
exclude_modules --svdquant-exclude-modules None Excluded module-name substrings.
model_adapter --svdquant-model-adapter auto Runtime grouping/export adapter.

Python API

from auto_round import AutoRound
from auto_round.algorithms.quantization.rtn.config import RTNConfig
from auto_round.algorithms.transforms.svdquant import SVDQuantConfig

autoround = AutoRound(
    "/path/to/FLUX.1-dev",
    scheme="MXFP4",
    model_dtype="bf16",
    alg_configs=[
        SVDQuantConfig(rank=32, model_adapter="flux"),
        RTNConfig(disable_opt_rtn=True),
    ],
    device_map=0,
    low_gpu_mem_usage=True,
)
autoround.quantize_and_save(
    "./flux-dev-mxfp4-svdquant-rtn",
    format="svdquant_nunchaku",
)

Export and inference

The svdquant_nunchaku format currently requires MXFP4 E2M1 W4A4, group size 32, symmetric weights/activations, dynamic activation quantization, and a supported runtime adapter. Nunchaku is not imported during quantization; it is required only for inference.

Pre-quantized FLUX.1-dev checkpoints are available for smooth SVDQuant + SignRound and no-smooth SVDQuant + SignRound.

The output is a self-contained Diffusers pipeline. Load its Nunchaku transformer explicitly, then pass it to Diffusers:

import torch
from diffusers import FluxPipeline
from nunchaku import NunchakuFluxTransformer2dModel

model_dir = "./flux-dev-mxfp4-svdquant-signround"
transformer = NunchakuFluxTransformer2dModel.from_pretrained(
    f"{model_dir}/transformer/diffusion_pytorch_model.safetensors",
    torch_dtype=torch.bfloat16,
    precision="mxfp4",
    device="cuda:0",
)
pipe = FluxPipeline.from_pretrained(
    model_dir,
    transformer=transformer,
    torch_dtype=torch.bfloat16,
    local_files_only=True,
)
pipe.enable_model_cpu_offload()

image = pipe(
    "A cat holding a sign that says Hello world",
    num_inference_steps=20,
    guidance_scale=3.5,
    generator=torch.Generator(device="cuda").manual_seed(12345),
    height=512,
    width=512,
).images[0]
image.save("flux-svdquant-mxfp4.png")

For SDXL, load the exported Nunchaku UNet and inject it into the saved Diffusers pipeline:

import torch
from diffusers import StableDiffusionXLPipeline
from nunchaku.models.unets.unet_sdxl import NunchakuSDXLUNet2DConditionModel

model_dir = "./sdxl-base-1.0-mxfp4-svdquant-rtn"
unet = NunchakuSDXLUNet2DConditionModel.from_pretrained(
    f"{model_dir}/unet/diffusion_pytorch_model.safetensors",
    torch_dtype=torch.bfloat16,
    device="cuda:0",
)
pipe = StableDiffusionXLPipeline.from_pretrained(
    model_dir,
    unet=unet,
    torch_dtype=torch.bfloat16,
    local_files_only=True,
).to("cuda:0")

image = pipe(
    "A cinematic photograph of a red panda in a bamboo forest",
    num_inference_steps=20,
    guidance_scale=5.0,
    generator=torch.Generator(device="cuda").manual_seed(12345),
).images[0]
image.save("sdxl-svdquant-mxfp4.png")

Limitations

  • The SDXL adapter supports SDXL base 1.0, not Stable Diffusion 1.x UNets.
  • SVDQuant requires nblocks=1.
  • Smooth search replays every retained call for every Alpha/Beta candidate.
  • More residual iterations repeat decomposition and QDQ work.
  • A smoke image validates loading and numerical stability, not dataset-level generation quality.