PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
AutoRound Environment Variables Configuration
English | 简体中文
This document describes the environment variables used by AutoRound for configuration and their usage.
Overview
AutoRound uses a centralized environment variable management system through the envs.py module. This system provides lazy evaluation of environment variables and programmatic configuration capabilities.
Available Environment Variables
AR_LOG_LEVEL
- Description: Controls the default logging level for AutoRound
- Default:
"INFO" - Valid Values:
"TRACE","DEBUG","INFO","WARNING","ERROR","CRITICAL" - Usage: Set this to control the verbosity of AutoRound logs
export AR_LOG_LEVEL=DEBUG
AR_ENABLE_COMPILE_PACKING
- Description: Enables compile packing optimization
- Default:
False(equivalent to"0") - Valid Values:
"1","true","yes"(case-insensitive) for enabling; any other value for disabling - Usage: Enable this for performance optimizations during packing FP4 tensors into
uint8.
export AR_ENABLE_COMPILE_PACKING=1
AR_NVFP4_FUSED_LAYER_GLOBAL_SCALE
- Description: Makes fused NVFP4 weight projections use one shared weight global scale. This applies to
q_proj/k_proj/v_projandgate_proj/up_proj, as required by vLLM fused kernels. - Default:
True(equivalent to"1") - Valid Values:
"0","false","no", or"off"(case-insensitive) disable sharing; any other value enables it. - Usage: Disable only when exporting for a runtime that does not require fused projections to share a global scale.
export AR_NVFP4_FUSED_LAYER_GLOBAL_SCALE=0
AR_USE_MODELSCOPE
- Description: Controls whether to use ModelScope for model downloads
- Default:
False - Valid Values:
"1","true"(case-insensitive) for enabling; any other value for disabling - Usage: Enable this to use ModelScope instead of Hugging Face Hub for model downloads
export AR_USE_MODELSCOPE=true
AR_QUANTIZE_BAGEL_MOE_GEN
- Description: Enables quantization of BAGEL's
*_moe_genimage-generation experts. By default, AutoRound keeps these modules in BF16 because quantizing them has been observed to reduce image-generation quality. BAGEL's normal transformer attention and MLP layers are still quantized by default. - Default:
False(equivalent to"0") - Valid Values:
"1","true", or"yes"(case-insensitive) for enabling; any other value keeps*_moe_genin BF16 - Usage: Enable only when experimenting with the checkpoint-size and image-quality tradeoff
export AR_QUANTIZE_BAGEL_MOE_GEN=1
AR_WORK_SPACE
- Description: Sets the workspace directory for AutoRound operations
- Default:
"ar_work_space" - Usage: Specify a custom directory for AutoRound to store temporary files and outputs
export AR_WORK_SPACE=/path/to/custom/workspace
AR_DISABLE_OFFLOAD
- Description: Forcibly disables the weight offloading feature in
OffloadManager. Useful during development and debugging to skip all offload/reload overhead. - Default:
False(equivalent to"0") - Valid Values:
"1","true","yes"(case-insensitive) for disabling offload; any other value keeps the default behavior - Usage: Set this to bypass offloading entirely
export AR_DISABLE_OFFLOAD=1
AR_DISABLE_DATASET_SUBPROCESS
- Description: Only for research. Disables the use of a subprocess for dataset preprocessing. By default, AutoRound uses a subprocess to ensure all temporary memory is reclaimed by the OS.
- Default:
False - Valid Values:
"1","true"(case-insensitive) for disabling; any other value for enabling - Usage: Set this to run dataset preprocessing in the main process
export AR_DISABLE_DATASET_SUBPROCESS=true
AR_DISABLE_GGUF_MTP_EXPORT
- Description: Disables multi-token prediction (MTP) layers when exporting GGUF models. By default, AutoRound follows the llama.cpp converter and includes MTP layers when the model provides them.
- Default:
False(equivalent to"0") - Valid Values:
"1","true","yes", or"on"(case-insensitive) disable MTP export; any other value keeps MTP export enabled - Usage: Enable this when exporting a checkpoint that uses an MTP-capable architecture but does not contain MTP tensors
export AR_DISABLE_GGUF_MTP_EXPORT=1
AR_ACT_SCALE
- Description: Only for research. Controls the scaling factor applied to activation min/max values during activation quantization. A value less than 1.0 shrinks the clipping range, which can reduce outlier impact.
- Default:
1.0 - Valid Values: float>=0.0, e.g.
0.8,0.9,1.0 - Usage: Set this to adjust the activation clipping range
export AR_ACT_SCALE=0.9
AR_ENABLE_ACT_MINMAX_TUNING
- Description: Enables tuning of activation min/max scale parameters (
act_min_scale,act_max_scale) during quantization optimization. When enabled, these scales become tunable instead of remaining fixed at1.0. - Default:
False(equivalent to"0") - Valid Values:
"1","true","yes"(case-insensitive) for enabling tuning; any other value keeps tuning disabled - Usage: Set this to enable activation min-max scale tuning
export AR_ENABLE_ACT_MINMAX_TUNING=1
AR_SEARCH_SCALE_RATIO
- Description: Controls the search range ratio used by the symmetric INT scale search in
auto_round.data_type.int.search_scales. The search bound isnmax * AR_SEARCH_SCALE_RATIO, wherenmax = 2^(bits-1). Smaller values restrict the search to a tighter neighborhood around the initial scale (faster, less thorough); larger values broaden the search (slower, may improve accuracy on outlier-heavy weights). - Default: unset → falls back to the built-in default (
0.5, i.e.nmax/2). - Valid Values: positive float, e.g.
0.25,0.5,0.75,1.0 - Usage: Set this to override the default scale-search range
export AR_SEARCH_SCALE_RATIO=0.75
AR_NEUQI_COARSE
- Description: Number of coarse-stage candidates in the NeUQI grid search (
--enable_neuqi, arXiv 2505.17595). The coarse stage scans a wide log-spaced grid; the fine stage refines around the winner (AR_NEUQI_FINE). Applies to both the asymmetric joint (scale, zero-point) search and the symmetric two-stage scale search. - Default:
256on accelerated backends (Triton/torch.compile),64on the eager sweep — the eager path evaluates ~coarse x fine candidates per group, so it narrows the unpinned default (measured quality is flat between the two grids) - Valid Values: positive integer
- Usage: Lower for faster searches at the cost of grid coverage.
export AR_NEUQI_COARSE=128
AR_NEUQI_FINE
- Description: Number of fine-stage candidates in the NeUQI grid search, refined around the coarse winner. See
AR_NEUQI_COARSE. - Default:
64on accelerated backends (Triton/torch.compile),32on the eager sweep - Valid Values: positive integer
- Usage: Lower for faster searches at the cost of final resolution.
export AR_NEUQI_FINE=32
AR_NEUQI_BACKEND
- Description: Backend override for the NeUQI candidate sweep.
autopicks the fastest available (extension Triton kernel, then torch.compile-fused, then eager) and latches down permanently on the first kernel failure.eager/compile/tritonforce one backend for debugging or benchmarking. - Default:
auto - Valid Values:
auto,eager,compile,triton - Usage: Escape hatch for driver/hardware incompatibilities;
autois recommended.
export AR_NEUQI_BACKEND=eager
AR_DYNAMO_CACHE_SIZE_LIMIT
- Description: Minimum value to which
torch._dynamo'scache_size_limit,accumulated_cache_size_limit, andrecompile_limitare bumped whentorch.compileis enabled (the default except on Windows). The same compiled quant function is reused across every linear layer in a transformer block (q/k/v/o_proj, gate/up/down_proj, ...) but each layer has a different weight shape, so per-layer static recompiles quickly exceed dynamo's default limit (8) and trigger a noisy fallback to eager. Raising the limit keeps static-shape compilation (best perf) and just allows more cache entries. On multi-device runs the target is scaled by the number of engaged devices (per-layer shapes multiplied by per-device guard pins); single-device runs use the value as-is. - Default:
16(per engaged device on multi-device runs) - Valid Values: positive integer
- Usage: Increase if your model has more than 16 distinct linear-weight shapes per block per engaged device (rare).
export AR_DYNAMO_CACHE_SIZE_LIMIT=32
AR_MODEL_FREE_SHARD_PARALLELISM
- Description: Controls how many weight shards are processed concurrently during model-free quantization. Increasing this value improves resource utilization but consumes more RAM.
- Auto policy (when the variable is not set):
shard_count // 4, capped at 10, minimum 1. For example, 8 shards → 2 workers; 40 shards → 10 workers. - The effective parallelism is always capped to the actual number of shards.
- Auto policy (when the variable is not set):
- Default: unset → auto policy applies (
shard_count // 4, max 10, min 1) - Valid Values: any positive integer; these are not restricted to specific values — e.g.
2,4,6,8 - Usage: Set this to override the automatic parallelism selection
export AR_MODEL_FREE_SHARD_PARALLELISM=4
AR_AUTO_SCHEME_NSAMPLES
- Description: Controls the default number of calibration samples used by AutoScheme scoring when
AutoScheme.nsamplesis not explicitly set. - Default: unset → 16
- Valid Values: any positive integer, e.g.
8,16,32 - Usage: Set this to override the automatic sample-count selection for AutoScheme
export AR_AUTO_SCHEME_NSAMPLES=1 # set 1 for quick execution
AR_AUTO_SCHEME_BATCH_SIZE
- Description: Controls the default batch size used by AutoScheme scoring when
AutoScheme.batch_sizeis not explicitly set. - Default: unset → built-in heuristic applies (8 for low GPU memory mode, 1 for normal mode)
- Valid Values: any positive integer, e.g.
1,2,4 - Usage: Set this to override the default batch size for AutoScheme
export AR_AUTO_SCHEME_BATCH_SIZE=1
AR_AUTO_SCHEME_SEQLEN
- Description: Controls the default calibration sequence length used by AutoScheme scoring when
AutoScheme.seqlenis not explicitly set. - Default: unset → built-in heuristic applies (128 for MoE models, 256 otherwise)
- Valid Values: any positive integer, e.g.
256,512,1024 - Usage: Set this to override the default sequence length for AutoScheme (2-bit schemes usually benefit from
1024)
export AR_AUTO_SCHEME_SEQLEN=1024
AR_AUTO_SCHEME_NO_SERIAL_FALLBACK
- Description: Turn a parallel-scoring failure into a hard error instead of falling back to serial scoring. Useful when the serial pass is known to be unable to run (or would take workers-count times longer): completed schemes and batches are persisted in the per-scheme cache, so a rerun scores only the failed parts.
- Default: unset -> parallel scoring failure falls back to serial
- Valid Values:
1,true,yes - Usage: Set this to fail fast on parallel scoring errors
export AR_AUTO_SCHEME_NO_SERIAL_FALLBACK=1
AR_AUTO_SCHEME_CACHE
- Description: Stores persistent per-scheme AutoScheme scoring JSON files. This directory is independent of
AR_WORK_SPACE, which is reserved for temporary working data. - Default:
~/.cache/auto_round - Valid Values: any writable directory path
- Usage: Set this to place reusable AutoScheme scores in a different cache directory
export AR_AUTO_SCHEME_CACHE=/path/to/auto_scheme_cache
AR_ENABLE_AUTO_SCHEME_PARALLEL
- Description: Enables multiprocessing across AutoScheme candidates. It can be combined with
AR_DISK_STREAM_MODEL=1; each worker then builds its own meta-model skeleton and streams blocks independently. Disable it when concurrent workers could exhaust host RAM or device memory. - Default:
"1"(schemes are scored in parallel when multiprocessing requirements are met) - Valid Values:
"1","true","yes"(case-insensitive) enable parallel scoring; any other value disables parallel scoring - Usage: Set this to
0before running AutoScheme to force serial candidate scoring
export AR_ENABLE_AUTO_SCHEME_PARALLEL=0
AR_SCHEME_MEM_INVENTORY
- Description: When enabled, AutoScheme streaming scoring prints a live CUDA-tensor census (
[mem-inv]lines) at block boundaries -- tensors grouped by (shape, dtype), largest first -- which makes unreleased module weights or retained autograd graphs visible as they accumulate. Independently of this variable, a richer census (live tensors, the retaining containers, and the failing op's traceback) is always attached to scoring-worker CUDA OOM errors. - Default:
False - Valid Values:
"1","true","yes"(case-insensitive) for enabling; any other value for disabling - Usage: Enable when investigating VRAM growth during AutoScheme scoring
export AR_SCHEME_MEM_INVENTORY=1
AR_NVFP4_E5M3_CACHE_HP_WEIGHT
- Description: Controls whether
NVFP4E5M3QuantLinearcaches a dequantized high-precision weight after the first forward pass, instead of dequantizing the packed FP4 weight on every call. - Default:
False(equivalent to"0") - Valid Values:
"1","true","yes","on"(case-insensitive) enable caching; any other value disables caching - Usage: Enable this when repeated inference throughput matters more than memory footprint. The current implementation releases
weight_packedandweight_scaleafter materializing the cached high-precision weight, so steady-state memory usage increases and the cache cannot be cleared back to packed storage.
export AR_NVFP4_E5M3_CACHE_HP_WEIGHT=1
AR_DISK_STREAM_MODEL
- Description: When enabled,
AutoRound(model=<path>, ...)builds the model as a meta-device skeleton instead of fully materializing the checkpoint on CPU RAM up front, and streams each decoder block's real weights from the checkpoint's safetensors shards on demand -- materializing right before a block is used (calibration, tuning, orAutoSchemesensitivity scoring) and freeing it back to meta right after. This keeps peak CPU RAM roughly flat regardless of checkpoint size, instead of proportional to it. Non-block parameters (embeddings,lm_head, final norm) are still loaded up front, since they are typically small. Text-model AutoScheme scoring also supports combining this with parallel scoring, which is enabled by default; each worker streams its own block copy. - Default:
False - Valid Values:
"1","true","yes"(case-insensitive) for enabling; any other value for disabling - Usage: Enable this to quantize checkpoints larger than available CPU RAM + GPU VRAM combined. Only applies when
modelis a string (local directory) path; has no effect on already-loaded model objects.
export AR_DISK_STREAM_MODEL=1
AR_ALLOW_W8_ASYM
- Description: Allows 8-bit asymmetric weight quantization in every export format, skipping the default refusal for formats that cannot serve it (native
auto_round/auto_gptq/auto_awq/ marlin). Without it, 8-bit asym is allowed for thellm_compressorformat (compressed-tensors serving in vLLM) and refused or pinned back to symmetric elsewhere. Artifacts produced with it may not load in stock vLLM GPTQ-format serving. - Default:
0(disabled) - Valid Values:
0/1 - Usage: Opt-in escape hatch for serving stacks beyond stock vLLM.
AR_ALLOW_W8_ASYM=1 python -m auto_round --model ... --scheme W8A16 --asym --format auto_round
AR_DISABLE_BATCHED_SEARCH
- Description: Disables the batched quantization-search machinery: SignRoundV2 wrap-time init-scale searches (iters>0) and optimized-RTN scale searches (iters=0) then run fully serially (the GGUF DQ scale search always runs inline per module), one module at a time, on the device hosting each weight. Batching stacks modules sharing device/shape/config into one call with bit-identical per-module results; set this to fall back to the serial loop (e.g. for debugging or bisecting a suspected batching-related difference). Stacking also changes the transient VRAM envelope (see
AR_SEARCH_BATCH_GB). - Default:
0(batching enabled) - Valid Values:
0/1 - Usage: Kill switch for the batched/parallel search machinery.
AR_DISABLE_BATCHED_SEARCH=1 python -m auto_round --model ... --device_map 0,1,2,3
AR_PERF_COUNTERS
- Description: Emits one per-block
[perf] tune phasesINFO line with the wall time of each tuning phase (wrap / prepare / loop / tail) and, inside the loop, the sampler draw, best-params snapshots, gradient sync + optimizer step, rest, and the serial forward+loss+backward time. Intended for performance attribution of large multi-device quantization runs; adds negligible overhead. - Default:
0(disabled) - Valid Values:
0/1 - Usage: Set during benchmarking runs to attribute block time to wrap searches vs the tuning loop.
AR_PERF_COUNTERS=1 python -m auto_round --model ... --device_map 0,1,2,3
AR_SEARCH_BATCH_GB
- Description: Explicit override for the batched quantization searches' stacked-batch size, in GiB of stacked fp32 weights per batched call. Applies to ALL batched lanes: the iters=0 optimized-RTN search, the iters>0 wrapper init search, and the NeUQI grid anchors. When unset, batch sizes are capacity-driven: each batch is sized so the search's transient footprint (an fp32 copy chain over the batch, plus the NeUQI sweeps' capped internal temporaries when applicable) fits in half the probed free VRAM, and every executing device re-checks headroom at call time (an oversized batch halves instead of faulting). Set this only to cap batches below what the capacity planning would choose.
- Default: unset (capacity-driven sizing)
- Valid Values: positive float (GiB)
- Usage: Shrink batches on memory-tight multi-device runs.
AR_SEARCH_BATCH_GB=0.5 python -m auto_round --model ... --device_map 0,1
AR_MOE_EXPERTS_IMPL
- Description: Experts forward used during MoE tuning for models that were unfused into per-expert modules.
auto(default) predicts the per-device VRAM of the grouped-GEMM tuning path before the first MoE block and selectslinear_loopfor the whole run when grouped tuning cannot fit (qdq stack retention plus in-loop tuning state vs the free memory of each device); the decision and the per-device numbers are logged. Explicit values are never overridden byauto. - Default:
auto - Valid Values:
auto,linear_grouped,linear_grouped_sliced,linear_loop - Usage: Force a specific experts implementation for tuning, or keep
autoto let the memory model choose.
AR_MOE_EXPERTS_IMPL=linear_loop python -m auto_round --model ... --device_map 0,1,2,3
AR_DISABLE_CROSS_DEVICE_SEARCH
- Description: Restricts batched search chunks to the device that hosts each weight. By default both search lanes plan their chunks centrally: every chunk starts on its home device and chunks rebalance onto under-loaded viable devices, so the search spreads across every visible device with headroom (the searches read only their module's weight plus per-module statistics, never calibration activations; each executing device is owned by a single worker thread). Non-CUDA device families keep every chunk on its home device. Set to
1on rigs whose GPUs are shared with other jobs (a free-memory probe cannot see foreign reservations), when debugging cross-device behavior, or on hosts where the PCIe transfers outweigh the search cost. When no device fits, chunks stay on the weight's home device and rely on the per-chunk OOM fallback. - Default:
0(multi-GPU search enabled, probe-gated) - Valid Values:
0/1 - Usage: Pin search traffic to the weight-hosting devices.
AR_DISABLE_CROSS_DEVICE_SEARCH=1 python -m auto_round --model ... --iters 0
AR_RESUME_DIR
- Description: When set to a directory path, the per-block tuning loop checkpoints its progress there after each completed block, and resumes from the first not-yet-completed block on a fresh run against the same directory -- instead of restarting the whole tuning pass from block 0 after a crash or kill.
- Default: unset (no resumability)
- Valid Values: any writable directory path
- Usage: Set this for long-running quantization jobs on large checkpoints where a mid-run crash would otherwise be expensive to restart from scratch.
export AR_RESUME_DIR=/path/to/resume/state
Usage Examples
Setting Environment Variables
Using Shell Commands
# Set logging level to DEBUG
export AR_LOG_LEVEL=DEBUG
# Enable compile packing
export AR_ENABLE_COMPILE_PACKING=1
# Use ModelScope for downloads
export AR_USE_MODELSCOPE=true
# Set custom workspace
export AR_WORK_SPACE=/tmp/autoround_workspace
Using Python Code
from auto_round.envs import set_config
# Configure multiple environment variables at once
set_config(
AR_LOG_LEVEL="DEBUG",
AR_USE_MODELSCOPE=True,
AR_ENABLE_COMPILE_PACKING=True,
AR_WORK_SPACE="/tmp/autoround_workspace",
)
Checking Environment Variables
Using Python Code
from auto_round import envs
# Access environment variables (lazy evaluation)
log_level = envs.AR_LOG_LEVEL
use_modelscope = envs.AR_USE_MODELSCOPE
enable_packing = envs.AR_ENABLE_COMPILE_PACKING
workspace = envs.AR_WORK_SPACE
print(f"Log Level: {log_level}")
print(f"Use ModelScope: {use_modelscope}")
print(f"Enable Compile Packing: {enable_packing}")
print(f"Workspace: {workspace}")
Checking if Variables are Explicitly Set
from auto_round.envs import is_set
# Check if environment variables are explicitly set
if is_set("AR_LOG_LEVEL"):
print("AR_LOG_LEVEL is explicitly set")
else:
print("AR_LOG_LEVEL is using default value")
Configuration Best Practices
- Development Environment: Set
AR_LOG_LEVEL=TRACEorAR_LOG_LEVEL=DEBUGfor detailed logging during development - Production Environment: Use
AR_LOG_LEVEL=WARNINGorAR_LOG_LEVEL=ERRORto reduce log noise - Chinese Users: Consider setting
AR_USE_MODELSCOPE=truefor better model download performance - Performance Optimization: Enable
AR_ENABLE_COMPILE_PACKING=1if you have sufficient computational resources - Custom Workspace: Set
AR_WORK_SPACEto a directory with sufficient disk space for model processing
AR_TUNE_DDP_MAX_COLLECT_FORWARD_DEVICES
- Description: Caps the number of devices a hook-carrying sharded collection forward runs on concurrently (hook passes keep compiled runners python-bound between graph sections, which can convoy on the GIL). Only lower this on hosts where hook-carrying passes convoy; not reproduced on the validation rig up to world=8. Values above the engaged world are no-ops.
- Default:
0(no cap) - Valid Values: non-negative integer
- Usage: Limit concurrent shard-forward threads on weak-CPU hosts.
AR_TUNE_DDP_MAX_COLLECT_FORWARD_DEVICES=2 python -m auto_round --model ... --data_parallel 8
Notes
- Environment variables are evaluated lazily, meaning they are only read when first accessed
- The
set_config()function provides a convenient way to configure multiple variables programmatically - Boolean values for
AR_USE_MODELSCOPEare automatically converted to appropriate string representations - All environment variable names are case-sensitive
- Changes made through
set_config()will affect the current process and any child processes