The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.
Better documentation about the comfy quant format. (#17078)
comfyanonymous5a665890ded5
diff --git a/QUANTIZATION.md b/QUANTIZATION.mdindex b91bffe99..5d19de587 100644--- a/QUANTIZATION.md+++ b/QUANTIZATION.md@@ -1,167 +1,129 @@-# The Comfy guide to Quantization+# Comfy quantization format+Comfy quantized checkpoints are standard safetensors files containing quantized+weights, scale tensors, and per-module configuration. A checkpoint can mix+quantization formats and ordinary floating-point layers. ComfyUI selects native+kernels where supported and dequantizes weights for compute-dtype execution when+a format's fast path is disabled.-## How does quantization work?+## Checkpoint layout-Quantization aims to map a high-precision value x_f to a lower precision format with minimal loss in accuracy. These smaller formats then serve to reduce the models memory footprint and increase throughput by using specialized hardware.+Keep the model's normal state-dict names. Each quantized module adds a+`<module>.comfy_quant` tensor containing UTF-8 JSON as a one-dimensional `uint8`+array. For example, an FP8 linear layer contains:-When simply converting a value from FP16 to FP8 using the round-nearest method we might hit two issues:-- The dynamic range of FP16 (-65,504, 65,504) far exceeds FP8 formats like E4M3 (-448, 448) or E5M2 (-57,344, 57,344), potentially resulting in clipped values-- The original values are concentrated in a small range (e.g. -1,1) leaving many FP8-bits "unused"--By using a scaling factor, we aim to map these values into the quantized-dtype range, making use of the full spectrum. One of the easiest approaches, and common, is using per-tensor absolute-maximum scaling.--```-absmax = max(abs(tensor))-scale = amax / max_dynamic_range_low_precision--# Quantization-tensor_q = (tensor / scale).to(low_precision_dtype)--# De-Quantization-tensor_dq = tensor_q.to(fp16) * scale--tensor_dq ~ tensor-```--Given that additional information (scaling factor) is needed to "interpret" the quantized values, we describe those as derived datatypes.---## Quantization in Comfy--```-QuantizedTensor (torch.Tensor subclass)- ↓ __torch_dispatch__-Two-Level Registry (generic + layout handlers)- ↓-MixedPrecisionOps + Metadata Detection-```--### Representation--To represent these derived datatypes, ComfyUI uses a subclass of torch.Tensor to implements these using the `QuantizedTensor` class found in `comfy/quant_ops.py`--A `Layout` class defines how a specific quantization format behaves:-- Required parameters-- Quantize method-- De-Quantize method--```python-from comfy.quant_ops import QuantizedLayout--class MyLayout(QuantizedLayout):- @classmethod- def quantize(cls, tensor, **kwargs):- # Convert to quantized format- qdata = ...- params = {'scale': ..., 'orig_dtype': tensor.dtype}- return qdata, params-- @staticmethod- def dequantize(qdata, scale, orig_dtype, **kwargs):- return qdata.to(orig_dtype) * scale+```text+blocks.0.proj.weight # quantized weight+blocks.0.proj.weight_scale # weight scale+blocks.0.proj.bias # ordinary floating-point bias, if present+blocks.0.proj.comfy_quant # encoded {"format": "float8_e4m3fn"}```-To then run operations using these QuantizedTensors we use two registry systems to define supported operations.-The first is a **generic registry** that handles operations common to all quantized formats (e.g., `.to()`, `.clone()`, `.reshape()`).+To encode configuration in an existing state dict:-The second registry is layout-specific and allows to implement fast-paths like nn.Linear.```python-from comfy.quant_ops import register_layout_op--@register_layout_op(torch.ops.aten.linear.default, MyLayout)-def my_linear(func, args, kwargs):- # Extract tensors, call optimized kernel- ...-```-When `torch.nn.functional.linear()` is called with QuantizedTensor arguments, `__torch_dispatch__` automatically routes to the registered implementation.-For any unsupported operation, QuantizedTensor will fallback to call `dequantize` and dispatch using the high-precision implementation.---### Mixed Precision--The `MixedPrecisionOps` class (lines 542-648 in `comfy/ops.py`) enables per-layer quantization decisions, allowing different layers in a model to use different precisions. This is activated when a model config contains a `layer_quant_config` dictionary that specifies which layers should be quantized and how.--**Architecture:**--```python-class MixedPrecisionOps(disable_weight_init):- _layer_quant_config = {} # Maps layer names to quantization configs- _compute_dtype = torch.bfloat16 # Default compute / dequantize precision-```--**Key mechanism:**--The custom `Linear._load_from_state_dict()` method inspects each layer during model loading:-- If the layer name is **not** in `_layer_quant_config`: load weight as regular tensor in `_compute_dtype`-- If the layer name **is** in `_layer_quant_config`:- - Load weight as `QuantizedTensor` with the specified layout (e.g., `TensorCoreFP8Layout`)- - Load associated quantization parameters (scales, block_size, etc.)--**Why it's needed:**+import json+import torch-Not all layers tolerate quantization equally. Sensitive operations like final projections can be kept in higher precision, while compute-heavy matmuls are quantized. This provides most of the performance benefits while maintaining quality.--The system is selected in `pick_operations()` when `model_config.layer_quant_config` is present, making it the highest-priority operation mode.---## Checkpoint Format--Quantized checkpoints are stored as standard safetensors files with quantized weight tensors and associated scaling parameters, plus a `_quantization_metadata` JSON entry describing the quantization scheme.--The quantized checkpoint will contain the same layers as the original checkpoint but:-- The weights are stored as quantized values, sometimes using a different storage datatype. E.g. uint8 container for fp8.-- For each quantized weight a number of additional scaling parameters are stored alongside depending on the recipe.-- We store a metadata.json in the metadata of the final safetensor containing the `_quantization_metadata` describing which layers are quantized and what layout has been used.--### Scaling Parameters details-We define 4 possible scaling parameters that should cover most recipes in the near-future:-- **weight_scale**: quantization scalers for the weights-- **weight_scale_2**: global scalers in the context of double scaling-- **pre_quant_scale**: scalers used for smoothing salient weights-- **input_scale**: quantization scalers for the activations--| Format | Storage dtype | weight_scale | weight_scale_2 | pre_quant_scale | input_scale |-|--------|---------------|--------------|----------------|-----------------|-------------|-| float8_e4m3fn | float32 | float32 (scalar) | - | - | float32 (scalar) |--You can find the defined formats in `comfy/quant_ops.py` (QUANT_ALGOS).--### Quantization Metadata--The metadata stored alongside the checkpoint contains:-- **format_version**: String to define a version of the standard-- **layers**: A dictionary mapping layer names to their quantization format. The format string maps to the definitions found in `QUANT_ALGOS`.--Example:-```json-{- "_quantization_metadata": {- "format_version": "1.0",- "layers": {- "model.layers.0.mlp.up_proj": {"format": "float8_e4m3fn"},- "model.layers.0.mlp.down_proj": {"format": "float8_e4m3fn"},- "model.layers.1.mlp.up_proj": {"format": "float8_e4m3fn"}- }- }-}-```---## Creating Quantized Checkpoints--To create compatible checkpoints, use any quantization tool provided the output follows the checkpoint format described above and uses a layout defined in `QUANT_ALGOS`.--### Diffusion attention preferences--A diffusion attention module can have a `<module path>.comfy_attention.config` entry whose-uint8 tensor contains UTF-8 JSON. Use a single preference:--```json-{"attention": "comfy_kitchen_int8"}+config = {"format": "float8_e4m3fn", "full_precision_matrix_mult": True}+state_dict["blocks.0.proj.comfy_quant"] = torch.tensor(+ list(json.dumps(config).encode("utf-8")), dtype=torch.uint8+)```-Or an ordered list of preferences, each with its own options:+`format` is required. Modules without `comfy_quant` load normally. Logical weight+dimensions come from the model's module definition; compute dtype is selected at+runtime. No file-level layer map or `format_version` field is required. The older+`_quantization_metadata` safetensors header and `scaled_fp8` checkpoints are+converted during loading; new files should use per-module entries.++## Weight formats++All tensor names below have the same `<module>.` prefix. `N` is the output width+(or vocabulary size), `K` is the input width, and `G` is the quantization group+size. Scales are FP32 unless another dtype is specified.++| `format` | `weight` storage | Scale tensors |+| --- | --- | --- |+| `float8_e4m3fn` | FP8 E4M3FN, `[N, K]` | `weight_scale`: scalar |+| `float8_e5m2` | FP8 E5M2, `[N, K]` | `weight_scale`: scalar |+| `nvfp4` | FP4 E2M1 packed into `uint8`, `[N, K/2]` before padding | `weight_scale`: FP8 E4M3FN block scales, groups of 16; `weight_scale_2`: global scalar |+| `mxfp8` | FP8 E4M3FN, `[N, K]` before padding | `weight_scale`: E8M0 power-of-two block scales, groups of 32 |+| `int8_tensorwise` | `int8`, `[N, K]` | `weight_scale`: scalar or per-row `[N, 1]` |+| `convrot_w4a4` | Signed INT4 packed into `int8`, `[N, K/2]` | `weight_scale`: per-row `[N]` |+| `asym_w4a8_int8` | 4-bit codes packed into `int8`, `[N, K/2]` | `weight_s_rel`: FP8 E4M3FN or FP32 `[N, K/G]`; `weight_s_channel`: FP32 `[N]` |+| `w6a8_int8` | Uniform 6-bit codes packed into `int8`, `[N, 3K/4]` | `weight_s_rel` and `weight_s_channel`, as above |++NVFP4 and MXFP8 use Comfy Kitchen's blocked/swizzled scale layout, with weight+dimensions padded to multiples of 16 and 32 respectively. Their block scales,+and FP8 `weight_s_rel`, can also be stored as raw `uint8` bytes. Use the layout's+quantizer and `QuantizedTensor.state_dict(prefix)` to produce the correct packing,+padding, and scale keys; add `comfy_quant` separately.++### Format options++Options belong in the module's `comfy_quant` JSON:++| Option | Applies to | Behavior |+| --- | --- | --- |+| `full_precision_matrix_mult` | Linear and MoE weights | Default `false`. `true` keeps quantized storage but dequantizes for matrix multiplication in the compute dtype; it does not force FP32. |+| `convrot` | `int8_tensorwise` | Default `false`. Enables grouped Hadamard rotation; export rotated weights with per-row scales. |+| `convrot_groupsize` | Rotated INT8 and all packed integer formats | Rotation group size, default `256`. Packed integer formats always use rotation. |+| `linear_dtype` | `convrot_w4a4` | `"int4"` (default) or `"int8"` selects the multiplication path; stored weights remain INT4. The internal `quant_group_size` is fixed at `64`. |+| `group_size` | `asym_w4a8_int8`, `w6a8_int8` | Weight scale group size, default `16`. |+| `num_experts` | MoE banks | Expert count written when saving; the model definition supplies the dimensions when loading. |++Linear/MoE loaders also accept rotation, group-size, and `linear_dtype` options+inside a nested `params` object; top-level values take precedence. Embedding+rotation options must be top-level.++`asym_w4a8_int8` optionally stores `weight_codebook`, a floating-point `[16]`+lookup table; without it, codes use uniform levels. `w6a8_int8` has no codebook.+Export these formats with symmetric weights: the ComfyUI loader does not consume+Comfy Kitchen's optional `weight_correction` tensor.++Rotation groups must divide `K` and be powers of four. W4A4 also requires `K`+divisible by 64. For grouped W4A8, `K` must be divisible by 16 and `G`; `G` must+be at least 4 and either divide 16 or be a multiple of 16. W6A8 requires `K`+divisible by 32 and `G` a multiple of 16. Accelerated backends may impose further+shape restrictions.++## Activations and smoothing++- **FP8:** optional `input_scale` supplies a calibrated scalar activation scale.+ Without it, the scale is `1.0`.+- **NVFP4:** optional `input_scale` supplies the global activation scale;+ otherwise it is computed from the current input. Block scales are dynamic.+- **MXFP8:** activation block scales are computed dynamically; a stored+ `input_scale` is not used by the quantizer.+- **Integer formats:** accept floating-point inputs and quantize activations+ inside the selected kernel. They do not load `input_scale`.+- **NVFP4 smoothing:** optional `pre_quant_scale` multiplies the input before+ the linear operation. Export weights with the matching inverse adjustment.++Calibration is needed to choose representative fixed activation scales, not to+use formats with dynamic activation scaling. Biases remain floating point.++## Embeddings and MoE++Quantized embeddings support `float8_e4m3fn`, `float8_e5m2`, and+`int8_tensorwise`, including INT8 per-row scales and optional rotation. Lookup+dequantizes the selected rows; embedding scales default to `1.0` when omitted.++Models using `MoEExperts` can store an expert bank under one module prefix.+Weights and per-expert scales carry a leading expert dimension `E`; optional+bias is `[E, N]`. Scalar scales can be shared. Grouped W4A8/W6A8 and INT8 banks+with per-row scales can also be stored with expert/output rows flattened to+`E * N`. MoE banks do not load `input_scale` or `pre_quant_scale`.++Linear weights use ComfyUI's normal offloading and LoRA patching paths. Module+state-dict saving writes quantized tensors, scales, and supported configuration.++## Attention preferences++An attention module with a `ComfyAttention` child can store+`<module>.comfy_attention.config` using the same `uint8` JSON encoding. This is+independent of weight quantization. Supply one object or an ordered list:```json[@@ -170,40 +132,19 @@ Or an ordered list of preferences, each with its own options:]```-Use the module that performs attention, such as `transformer_blocks.0.attn` for-Qwen Image 2.1 or `blocks.0.attn` for MiniMax H3. The target must have a-`ComfyAttention` child module.--Only `comfy_kitchen_int8` and `comfy_kitchen_sol` are accepted. ComfyUI selects-the first available preference during loading, checking support for the primary-device. All other method names are skipped with a warning. Unavailable methods-are skipped. If none are available, or the list is empty, normal attention-selection remains in effect. Explicit attention overrides retain priority.--`comfy_kitchen_sol` supports only `tau` as an optional field alongside-`attention`. It defaults to `1.0` and sets the routing threshold; higher values-route fewer blocks exactly. Other SOL options use Comfy Kitchen's defaults.--Bake SOL preferences into modules with compatible inputs: matching FP16/BF16-Q/K/V tensors with a head dimension of 128. ComfyUI trusts the checkpoint's-choice; `low_precision_attention=False` does not disable SOL. Masked calls use-PyTorch attention.--The `ComfyAttention` child module loads and saves its metadata through normal-state-dict loading and saving, preserving the full preference list and options,-including unsupported entries. Preferences do not enable weight quantization.--### Weight Quantization--Weight quantization is straightforward - compute the scaling factor directly from the weight tensor using the absolute maximum method described earlier. Each layer's weights are quantized independently and stored with their corresponding `weight_scale` parameter.--### Calibration (for Activation Quantization)--Activation quantization (e.g., for FP8 Tensor Core operations) requires `input_scale` parameters that cannot be determined from static weights alone. Since activation values depend on actual inputs, we use **post-training calibration (PTQ)**:--1. **Collect statistics**: Run inference on N representative samples-2. **Track activations**: Record the absolute maximum (`amax`) of inputs to each quantized layer-3. **Compute scales**: Derive `input_scale` from collected statistics-4. **Store in checkpoint**: Save `input_scale` parameters alongside weights--The calibration dataset should be representative of your target use case. For diffusion models, this typically means a diverse set of prompts and generation parameters.+ComfyUI selects the first supported preference for the primary device during+loading. Unknown methods are skipped with a warning; unavailable methods are+skipped. An empty list or no available preference leaves normal attention+selection in effect. Explicit attention overrides take priority. Saving+preserves the full preference list, including unsupported entries.++Only the two methods above are accepted. SOL exposes `tau` (default `1.0`);+higher values route fewer blocks through exact attention. Other SOL options use+Comfy Kitchen defaults. SOL requires matching FP16/BF16 Q/K/V inputs with head+dimension 128. Masked calls use PyTorch attention; `low_precision_attention=False`+does not disable SOL.++Implementation: [format registry](comfy/quant_ops.py),+[loaders and serialization](comfy/ops.py),+[legacy conversion](comfy/utils.py),+[attention configuration](comfy/ldm/modules/attention.py).