Data-free 4-bit quantization for diffusion transformers

Quantize-on-load with no calibration data, scored as seed-matched PSNR/SSIM against a bf16 reference generated on the same GPU. Four models, three precisions, one recipe.

↑ higher is better  ·  ↓ lower is better  ·  →1 closest to 1.0 is best (ratios vs bf16). Every quality metric is measured against a bf16 reference.

Quality vs bf16

Each cell is PSNR in dB ↑ / SSIM ↑ against a bf16 reference from the same GPU, averaged over 5 seed-matched samples. Both precisions use the default recipe: rank 32, smoothing 0.5, no rotation. Per-sample spread is about ±3 dB, so treat differences under 0.5 dB as ties.

Memory, speed and load time

int4 speed is shown only where it ran on its target hardware, the A100. Qwen-Image-2.1 and FLUX.2 int4 were run on Blackwell, which has no native int4 tensor-core path for this kernel, so their timings there (2.2 to 3× slower than bf16) reflect the GPU, not int4 itself. Pre-quantized export's real value is load time, not quality.

Visual comparison

Video comparisons show every configuration of one seed side by side; pick a scene, then play whichever video you want to watch. Set speed to 0.25× to see flicker. Image comparisons are contact sheets with the same seed down each column. The int4 rows are compared with the A100 bf16 row and the nvfp4 rows with the RTX bf16 row, because the same seed gives a different trajectory on a different GPU.

LTX-2.3-Distilled: int4 default vs rank 128 vs nvfp4

What to look for: plain int4 keeps the scene but shifts pose, lighting and composition (the drummer gains headphones, the car moves lanes, the fireworks change count). int4 at rank 128 tracks its reference more closely. nvfp4 against its own RTX reference is the closest match. Every video is clean and sharp; the loss is trajectory drift, not artifacts.

MiniMax-H3: joint audio-video, 30.8B

nvfp4 is compared with the RTX reference and int4 with the A100 reference. Both keep the scene and the camera move; int4 drifts further in the drummer's framing and the dog's pose. With sound on, the quantized soundtracks are a different rendition of the same scene (envelope correlation about 0.55 to 0.6), not a failure.

FLUX.2-klein-9B: 4-step image model

Qwen-Image-2.1: text-to-image across CFG, and an edit

All Qwen-Image-2.1 sheets

Every sheet has rows bf16 / nvfp4 / int4 and columns CFG off / 2.5 / 4, with PSNR ↑ and LPIPS ↓ against bf16 on each quantized tile. Click any sheet to view it at full size.

Text-to-image, 5 prompts (2048², 40 steps)

Edits, 12 instructions (own edits 0–4, identity edit 5, blog Fidelity pairs 6–11)

Rank and smoothing

The data-free recipe has two settings: rank, the size of the low-rank branch that keeps the largest weight components in 16-bit, and smoothing strength, how far each input channel is rescaled before 4-bit rounding, from 0 (off) to 1 (fully flattened). The defaults are rank 32 and smoothing 0.5.

Smoothing barely matters at the default rank

Rank helps only when smoothing is retuned with it

• Up to +2.7 dB, but only at the right smoothing. Rank 128 reaches 15.99 dB at smoothing 0.25 and gains nothing at the default 0.5. The best smoothing value falls as rank grows, because the low-rank branch and smoothing absorb the same outliers.
• Cost. Transformer memory grows about 16% from rank 32 to 128. Inference time and quantize-on-load time do not change.
• Not worth it with rotation. Hadamard rotation at rank 32 reaches the same quality (15.42 dB) at the default memory, and rank 128 with rotation is worse (14.28 dB). See the Hadamard section.
• Other models. On MiniMax-H3, rank 64 cuts weight error by only about 3%; no end-to-end comparison was scored. Qwen-Image-2.1 and FLUX.2 were run at rank 32 only.

Calibrated vs data-free

The stock recipe above uses no data at all. Here it is set beside checkpoints that were calibrated on real activations, on the two models where such checkpoints exist, with the same seeds and the same bf16 references.

The calibrated checkpoints were quantized with rootonchair/diffuse-compressor, which runs the model on calibration data and chooses the smoothing values that minimize each block's error.

Weights used:

• LTX-2.3-Distilled int4: lite-infer/LTX-2.3-Distilled-v1.1-Diffusers-nunchaku-lite-int4-bnb4-text-encoder
• LTX-2.3-Distilled nvfp4: lite-infer/LTX-2.3-Distilled-v1.1-Diffusers-nunchaku-lite-nvfp4-bnb4-text-encoder
• MiniMax-H3 int4: rootonchair/MiniMax-H3-nunchaku-lite-int4/calibrated-8x20
• MiniMax-H3 nvfp4: rootonchair/MiniMax-H3-nunchaku-lite-nvfp4/calibrated-8x20

Per-layer error

• Calibration trades weight precision for activation precision. Its smoothing scales are much larger, so the weights themselves are rounded less exactly: 12.2% vs 8.2% relative weight error (nvfp4) and 12.6% vs 9.8% (int4) on LTX. In exchange, activation error drops sharply.
• The full layer comes out ahead. Scored on 512 real activation rows per layer through the whole 4-bit path, calibrated error is lower on 946 of 1344 layers (nvfp4): 6.9% vs 7.6% (nvfp4) and 9.3% vs 11.4% (int4).
• And far more even. Data-free error is spiky, reaching 80% activation error on some attention output layers. Calibration brings those to 20% or less, including the layer with the largest weight error (6.2% vs 9.7% full error).
• Caveat: the activations came from a data-free nvfp4 run because a bf16 capture did not fit on that machine, so the comparison is fair between the two recipes but absolute levels depend on that run.

Image quality

Every 16th frame of each video was scored with MUSIQ, a learned image-quality model (↑, roughly 0 to 100). It judges each frame on its own, so it captures artifacts such as streaks and grain directly.

• MiniMax-H3. On the rainy street, data-free int4 shows vertical and horizontal streaks across the buildings and dark areas, and calibrated int4 stays clean, close to bf16. On the ocean scene, data-free int4 water and sky are grainier. Calibrated nvfp4 also scores higher across all scenes.
• LTX-2.3-Distilled. Side-by-side crops of two scenes show no visible artifact difference between the two recipes.
• Why. The spiky per-layer errors of data-free are a plausible source of streaks and grain; calibration's flatter error avoids them.
• Limits: 5 videos per setting and one learned metric.

Closeness to bf16

PSNR and SSIM against a same-GPU bf16 reference, 5 seed-matched samples. These measure how closely each video follows the bf16 one, not how clean its frames look; quantized videos drift to slightly different compositions, which dominates these scores.

Visual comparison: LTX-2.3-Distilled

Both recipes keep each scene and drift from their reference in their own way: the drummer's pose, the car's position and the fireworks layout change from run to run.

Visual comparison: MiniMax-H3

Everything is compared with the RTX reference. On the rainy street, data-free int4 shows streak artifacts that calibrated int4 avoids; set speed to 0.25× and play each to compare. Calibrated int4 renders the drummer and fireworks with stronger contrast. Each video has its own volume control; both soundtracks are renditions of the same scene.

Hadamard rotation: the int4 improvement

Everything above uses the stock recipe. This section is the one change that moved int4 by more than trajectory noise: a data-free blockwise Hadamard rotation of the quantized residual, with the matching Walsh–Hadamard transform fused into the activation-quantization kernel. Validated on two video models and, as of 2026-10-09, on Qwen-Image-2.1.

Design

The problem it solves. 4-bit quantization gives each group of 64 input channels one shared scale. If one channel in a group carries a value ten times larger than its neighbours, the scale has to stretch to fit it, and every other channel in that group is rounded on a coarse grid. Diffusion transformers have exactly this: a handful of activation channels are consistently much larger than the rest, in a different place for every token. The low-rank branch of SVDQuant^[1] absorbs the largest weight outliers, and smoothing^[2] moves some activation range into the weights. But smoothing scales are computed from the weights, so they cannot follow outliers that change from token to token, and those remain the main error source.

The idea: rotate before rounding. Multiplying a vector by an orthonormal matrix changes its coordinates but not its length, and multiplying both sides of a matrix product by the same rotation leaves the result unchanged. A Hadamard matrix is a particularly convenient rotation: every entry is ±1/√n, so each output coordinate is an even mix of all inputs. One large channel gets spread across the whole block, and the rotated values look close to Gaussian, which a uniform 4-bit grid represents far better. This "incoherence processing" was introduced for weight quantization by QuIP^[3] with random orthogonal matrices. QuIP#^[4] switched to randomized Hadamard transforms because they can be applied fast. QuaRot^[5] extended it to activations to run LLMs fully in 4 bits, and SpinQuant^[6] learns the rotation instead of fixing it.

The math, step by step.

1. A linear layer maps an input vector to an output vector:

   \[ y = W x, \qquad W \in \mathbb{R}^{m \times n},\; x \in \mathbb{R}^{n} \]

2. Smoothing moves range from the activations into the weights with one scale per input channel. The product is unchanged:

   \[ y = \underbrace{\big(W\,\mathrm{diag}(s)\big)}_{\hat W}\;\underbrace{\big(\mathrm{diag}(s)^{-1} x\big)}_{\hat x} \]

3. SVDQuant splits the smoothed weight into a rank-32 part, kept in 16-bit, and a residual \(R\):

   \[ \hat W = L_2 L_1 + R, \qquad L_1 \in \mathbb{R}^{32 \times n}\ \text{(LoRA down)},\quad L_2 \in \mathbb{R}^{m \times 32}\ \text{(LoRA up)} \]

4. Only the residual path is quantized, both its weights and its activations, to 4 bits:

   \[ y \approx \underbrace{L_2\,(L_1 \hat x)}_{\text{16-bit LoRA}} \;+\; \underbrace{Q(R)\;Q(\hat x)}_{\text{W4A4 GEMM}} \]

5. Insert an orthonormal matrix \(H\) (so \(H H^\top = I\)) between the residual and its input. This is exact:

   \[ R\,\hat x \;=\; R\,H H^\top \hat x \;=\; (R H)\,(H^\top \hat x) \]

6. Now quantize the rotated operands instead. \(Q(RH)\) is computed once at load time, and \(H^\top \hat x\) per token inside the kernel; a Hadamard matrix is symmetric, so \(H^\top = H\):

   \[ y \approx L_2\,(L_1 \hat x) \;+\; Q(R H)\;Q(H \hat x) \]

7. \(H\) is block-diagonal, one \(64 \times 64\) Hadamard block per 4-bit quantization group, so every group keeps its own scale. Each block is applied with a fast Walsh–Hadamard transform in \(\log_2 64 = 6\) add/subtract stages:

   \[ H = \mathrm{diag}\big(H_{64}, \dots, H_{64}\big), \qquad H_{64} = \tfrac{1}{8} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}^{\otimes 6} \]

How it is applied here.

• Weights, at load time. The residual is rotated before rounding, as Q(R · H). H is block-diagonal with one 64×64 Hadamard block per quantization group, so each group's scale still covers only its own 64 channels. H is a fixed matrix, so no data is needed and the method stays data-free.
• Activations, at run time. The matching rotation is applied to each token inside the kernel that quantizes activations, as a fast Walsh–Hadamard transform in registers (64 inputs in log₂64 = 6 add/subtract stages). It runs after the smoothing division and after the low-rank projection, so the low-rank branch still sees the original input. This is the same fused online-rotation pattern QuaRot^[5] uses for LLMs.
• Why it beats smoothing here. Smoothing has to choose one scale per channel ahead of time. The rotation needs no statistics: it spreads whatever outlier a token has at run time, which is what per-token outliers require.

Why it does not help nvfp4. nvfp4 already gives every 16 channels their own fine-grained scale, and its 4-bit floating-point grid is built for heavy-tailed values. A 64-wide rotation mixes four of those small groups together and removes the tails the format handles well. Egiazarian et al.^[7] show that nvfp4's small groups neutralize classic outlier treatments, and they use smaller block-wise rotations tuned to the format instead. That matches the −2.2 dB measured here.

End-to-end numbers

Flicker ↓: on LTX it is mean |Laplacian| of frame differences (bf16 itself scores 6.39). MiniMax flicker is the period-4 power fraction, shown as mean / worst scene. Rotated int4 at rank 32 matches the old rank-128 best within noise at the default memory footprint. Qwen timings are not comparable across rows: the rotated run had the GPU to itself, the others shared it with other jobs.

Visual comparison: LTX-2.3-Distilled, same kernel, rotation off vs on

The control uses the same kernel build and seeds with the rotation off; H64 changes nothing else. With rotation the drummer loses the headphones and regains bf16's framing, the car stays on the crossing with pedestrians instead of drifting into traffic, the dog's pose and the tree line move back toward bf16, and the fireworks count matches. The second row repeats H64 next to rank 128 + H64: adding rank 128 on top drifts again, because the two treatments compete for the same outlier energy.

Visual comparison: MiniMax-H3

Rotation pulls int4's drummer framing, dog pose and fireworks shape back toward the A100 reference. The second row is plain nvfp4, without rotation, against its RTX reference: rotated int4 (15.6 dB) lands at about the same distance from its reference as nvfp4 does (15.9 dB). On the ocean scene at 0.25×, int4's frame-to-frame pulsing is the easiest thing to see; plain nvfp4 is already close to bf16 there.

Visual comparison: Qwen-Image-2.1, text-to-image across CFG

All Qwen-Image-2.1 sheets with the rotation row

Every sheet has rows bf16 / nvfp4 / int4 / int4 + H64 and columns CFG off / 2.5 / 4, with nvfp4 in its plain form (no rotation), with PSNR ↑ and LPIPS ↓ against bf16 on each quantized tile. Click any sheet on this page to view it at full size; click again or press Escape to close.

Text-to-image, 5 prompts (2048², 40 steps)

Edits, 12 instructions (own edits 0–4, identity edit 5, blog Fidelity pairs 6–11)

Rules learned

• int4 only. On nvfp4 it costs 2.2 dB and doubles flicker. The kernel option is validated for int4 and must stay off for nvfp4.
• Rank 32, smoothing off. Rotation and rank do not stack (r128 + H64 is 1.1 dB below r32 + H64), and smoothing on top of rotation lowers activation SNR. Recommended config: rank=32, hadamard=True, smooth_exponent=0.
• Pre-Blackwell hardware reaches nvfp4-class quality. A100 int4 + H64 (15.64) ≈ RTX nvfp4 (15.87) on MiniMax, at the same runtime as plain int4. On Qwen-Image-2.1 rotated int4 ties nvfp4 at CFG off (22.8 vs 22.7) and sits 0.8 dB below it under CFG, within 0.04 LPIPS.
• It removes most of int4's CFG sensitivity on Qwen. Plain int4 lost 3.5 dB from CFG off to CFG 4; rotated int4 loses 2.6, about what nvfp4 loses. The compositions int4 forked come back.
• Not a flicker cure. Mean flicker improves; the worst scene does not.
• Variants already tested with no extra gain: sign-flipped H64, H128. Random permutation is the only untested piece of the OrbitQuant family.

References

1. Muyang Li et al., 2024. SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models. arXiv:2411.05007
2. Guangxuan Xiao et al., 2022. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv:2211.10438
3. Jerry Chee et al., 2023. QuIP: 2-Bit Quantization of Large Language Models With Guarantees. arXiv:2307.13304
4. Albert Tseng et al., 2024. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. arXiv:2402.04396
5. Saleh Ashkboos et al., 2024. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. arXiv:2404.00456
6. Zechun Liu et al., 2024. SpinQuant: LLM quantization with learned rotations. arXiv:2405.16406
7. Vage Egiazarian et al., 2025. Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization. arXiv:2509.23202

Tested and negative

• LoftQ alternating SVD/quantize iteration: 4.6 to 11.1% reconstruction gain, below the pre-registered gate.
• Excluding the 288 worst layers (+3.5 GB, no effect; error is systemic). Decode-timestep dither (does not mask flicker).
• Activation scales derived from AdaLN modulation weights (helps only cross-modal to_q, hurts attn1).
• Frame-shared activation scales (+2.5 to 5 pts error, +75% jitter). DCT on all linears (harmful for nvfp4).
• AdaLN is not an error source on MiniMax (see the calibrated section).

Recommended configs

Code samples

Each sample is a complete script: it quantizes the transformer while it loads, with no calibration data, then generates one image. Quantizing takes roughly 5 to 20 minutes depending on model size. The examples use Qwen-Image-2.1; other models work the same way with their own transformer and pipeline classes.

Setup

# Data-free quantize-on-load lives on branch feature/data-free-quantization of rootonchair/diffusers.
# Qwen-Image-2.1 also needs upstream main merged in.
pip install "git+https://github.com/rootonchair/diffusers@feature/data-free-quantization" kernels
export DIFFUSERS_TRUST_REMOTE_KERNELS=true   # loads rootonchair/nunchaku-lite-kernels from the Hub

nvfp4

# nvfp4: Blackwell GPUs only. Best quality, no tuning needed.
import torch
from diffusers import NunchakuLiteQuantizationConfig, QwenImage21Pipeline, QwenImage21Transformer2DModel

model_id = "Qwen/Qwen-Image-2.1"

# Quantize the transformer while it loads. pre_quantized=False means data-free:
# no calibration data, the 4-bit weights are computed from the bf16 checkpoint.
quantization_config = NunchakuLiteQuantizationConfig(
    svdq_w4a4={
        "precision": "nvfp4",
        "group_size": 16,  # fixed for nvfp4
        "rank": 32,        # size of the 16-bit low-rank branch
    },
    pre_quantized=False,
)
transformer = QwenImage21Transformer2DModel.from_pretrained(
    model_id,
    subfolder="transformer",
    quantization_config=quantization_config,
    dtype=torch.bfloat16,
    device_map="cuda",
)

# The rest of the pipeline (text encoder, VAE) stays in bf16.
pipe = QwenImage21Pipeline.from_pretrained(model_id, transformer=transformer, dtype=torch.bfloat16)
pipe.to("cuda")

image = pipe(
    prompt="A street musician playing saxophone under a lamppost, holding a cardboard sign that says OPEN MIC, warm evening light.",
    num_inference_steps=40,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("qwen_nvfp4.png")

int4

# int4: runs on A100 and older GPUs.
import torch
from diffusers import NunchakuLiteQuantizationConfig, QwenImage21Pipeline, QwenImage21Transformer2DModel

model_id = "Qwen/Qwen-Image-2.1"

# Quantize the transformer while it loads. pre_quantized=False means data-free:
# no calibration data, the 4-bit weights are computed from the bf16 checkpoint.
quantization_config = NunchakuLiteQuantizationConfig(
    svdq_w4a4={
        "precision": "int4",
        "group_size": 64,  # fixed for int4
        "rank": 32,        # size of the 16-bit low-rank branch
    },
    pre_quantized=False,
)
transformer = QwenImage21Transformer2DModel.from_pretrained(
    model_id,
    subfolder="transformer",
    quantization_config=quantization_config,
    dtype=torch.bfloat16,
    device_map="cuda",
)

# The rest of the pipeline (text encoder, VAE) stays in bf16.
pipe = QwenImage21Pipeline.from_pretrained(model_id, transformer=transformer, dtype=torch.bfloat16)
pipe.to("cuda")

image = pipe(
    prompt="A street musician playing saxophone under a lamppost, holding a cardboard sign that says OPEN MIC, warm evening light.",
    num_inference_steps=40,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("qwen_int4.png")

int4 with Hadamard rotation

# Hadamard rotation needs branch feature/hadamard-rotation and a kernel build with the online transform
# (kernel branch "hadamard" of rootonchair/nunchaku_lite; builds exist for A100 and Blackwell).
pip install "git+https://github.com/rootonchair/diffusers@feature/hadamard-rotation"
pip install nunchaku_lite_kernels-0.1.1+cu12.8torch2.11-cp312-cp312-linux_x86_64.whl
export NUNCHAKU_LITE_LOCAL_KERNELS=1   # use the installed kernel instead of the Hub build

# int4 + Hadamard rotation: the recommended int4 setting. Never use rotation with nvfp4.
import torch
from diffusers import NunchakuLiteQuantizationConfig, QwenImage21Pipeline, QwenImage21Transformer2DModel

model_id = "Qwen/Qwen-Image-2.1"

# Quantize the transformer while it loads. pre_quantized=False means data-free:
# no calibration data, the 4-bit weights are computed from the bf16 checkpoint.
quantization_config = NunchakuLiteQuantizationConfig(
    svdq_w4a4={
        "precision": "int4",
        "group_size": 64,
        "rank": 32,
        "hadamard": True,        # rotate the residual and the activations in 64-channel blocks
        "smooth_exponent": 0.0,  # smoothing off; rotation replaces it
    },
    pre_quantized=False,
)
transformer = QwenImage21Transformer2DModel.from_pretrained(
    model_id,
    subfolder="transformer",
    quantization_config=quantization_config,
    dtype=torch.bfloat16,
    device_map="cuda",
)

# The rest of the pipeline (text encoder, VAE) stays in bf16.
pipe = QwenImage21Pipeline.from_pretrained(model_id, transformer=transformer, dtype=torch.bfloat16)
pipe.to("cuda")

image = pipe(
    prompt="A street musician playing saxophone under a lamppost, holding a cardboard sign that says OPEN MIC, warm evening light.",
    num_inference_steps=40,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("qwen_int4_hadamard.png")

Changing rank and smoothing

# int4 with a larger rank and retuned smoothing (best LTX int4 setting without rotation: +2.7 dB, +16% memory).
import torch
from diffusers import NunchakuLiteQuantizationConfig, QwenImage21Pipeline, QwenImage21Transformer2DModel

model_id = "Qwen/Qwen-Image-2.1"

# Quantize the transformer while it loads. pre_quantized=False means data-free:
# no calibration data, the 4-bit weights are computed from the bf16 checkpoint.
quantization_config = NunchakuLiteQuantizationConfig(
    svdq_w4a4={
        "precision": "int4",
        "group_size": 64,
        "rank": 128,              # default 32; larger = more memory, same speed
        "smooth_exponent": 0.25,  # default 0.5; 0 = off, 1 = fully flattened
    },
    pre_quantized=False,
)
transformer = QwenImage21Transformer2DModel.from_pretrained(
    model_id,
    subfolder="transformer",
    quantization_config=quantization_config,
    dtype=torch.bfloat16,
    device_map="cuda",
)

# The rest of the pipeline (text encoder, VAE) stays in bf16.
pipe = QwenImage21Pipeline.from_pretrained(model_id, transformer=transformer, dtype=torch.bfloat16)
pipe.to("cuda")

image = pipe(
    prompt="A street musician playing saxophone under a lamppost, holding a cardboard sign that says OPEN MIC, warm evening light.",
    num_inference_steps=40,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("qwen_int4_rank128.png")

• group_size is fixed by precision: 16 for nvfp4, 64 for int4. Any other value is rejected.
• rank is required. smooth_exponent (0 to 1, default 0.5) and hadamard are optional and apply only when pre_quantized=False.
• At rank 32, smoothing makes no measurable difference, so "smooth_exponent": 0.0 or leaving it out are equivalent in quality.
• Quantized layers are chosen automatically. To keep specific layers in bf16, pass exclude_targets=[...] to the config.