4-bit diffusion transformers on Apple silicon

Nunchaku Lite runs SVDQuant checkpoints on M5 Macs through PyTorch MPS and MLX, with Metal kernels for the GPU's Neural Accelerators

By Pham Hong Vinh

The same 4-bit SVDQuant checkpoints that run on CUDA now run on an M5 Mac, through PyTorch MPS (Diffusers) and MLX (mflux). New Metal kernels run the 4-bit layers as int8 matrix multiplies on the GPU's Neural Accelerators. All numbers come from an M5 with 24 GB, generating at 1024².

↑ higher is better  ·  ↓ lower is better. Speed-ups are per denoising step against mflux q4, the existing 4-bit option on MLX.

FLUX.1-dev, MLX
1.47× faster
7.70 s vs 11.28 s per step against mflux q4, 1024², 20 steps
FLUX.2 Klein 9B, MLX
1.61× faster
4.39 s vs 7.09 s per step, int4, peak 10.4 vs 10.3 GB
Quality vs bf16, Klein 4B
on par
PSNR ↑ 17.8 dB (Nunchaku) vs 17.5 dB (mflux q4) against the bf16 transformer, mean of 3 prompts; the gap is within the per-prompt spread
Image editing, Klein 4B, MLX
1.62× faster
4.66 s vs 7.54 s per step against mflux q4; the same checkpoint loads into mflux's edit pipeline

Two runtimes

Nunchaku Lite loads Nunchaku SVDQuant checkpoints into existing pipelines instead of shipping its own. On a Mac there are two pipeline ecosystems, so there are two runtimes.

PyTorch MPSMLX
PipelinesDiffusersmflux
Model adaptersthe CUDA adapters, unchangedwritten for mflux's modules (FLUX.1, FLUX.2 Klein, generic manifest)
Quantized layersMetal kernels in nunchaku-lite-kernelsMetal kernels through mx.fast.metal_kernel
Installpip install "nunchaku-lite[torch]" (Metal kernels download from the Hugging Face Hub)pip install "nunchaku-lite[mlx]" mflux
Best forDiffusers features: LoRA, CPU offload, every Diffusers pipelinespeed on the Klein models, and comparison with mflux q4

Both read the same checkpoints, including the lite-infer repos with a runtime manifest and older lite checkpoints that list their quantized layers in config.json.

How the 4-bit layers run on M5

The M5 Neural Accelerators

Every GPU core in the M5 has a Neural Accelerator, a unit built for matrix multiplication. Metal 4 shaders drive it through tensor operations (matmul2d in Metal Performance Primitives); earlier Apple GPUs run matrix multiplies on the ordinary shader cores. The accelerators are at their fastest with integers. Peak throughput of a bare matrix multiply on the M5:

operands (accumulator)throughput ↑
int8 × int8 (int32)~30 TOPS
int8 × int4 (int32)~30 TOPS
fp16 or bf16 × anything~15 TFLOPS

Two consequences shape the design. Int8 is twice as fast as bf16, so a layer gains by running its matmul in int8. And a 4-bit operand runs no faster than an 8-bit one (int8 × int4 matches int8 × int8), so 4-bit activations would add error without adding speed. So the layers run as W4A8 (4-bit weights, int8 activations) or W8A8 (the same 4-bit values stored expanded to int8), not as Nunchaku's CUDA W4A4.

The fused kernels come close to that ceiling. Here they are against a bf16 matmul at the projection shapes of a FLUX block for a 1024² image (4,608 tokens), with the scales, bias and low-rank branch all inside the int8 kernels:

projection (K → N)bf16 matmulW8A8 kernelW4A8 kernel
QKV 3072 → 921616.6 ms · 15.7 T9.1 ms · 28.6 T11.6 ms · 22.4 T
MLP in 3072 → 1228822.0 ms · 15.8 T12.1 ms · 28.8 T15.5 ms · 22.5 T
MLP out 12288 → 307232.1 ms · 10.8 T12.7 ms · 27.5 T17.0 ms · 20.4 T
attention out 3072 → 30725.7 ms · 15.3 T3.2 ms · 27.2 T4.0 ms · 21.5 T

T is trillions of operations per second, with a multiply-add counted as two. The W4A8 kernel is slower than W8A8 because it expands each 4-bit weight to its int8 value in the shader before the accelerator sees it. As whole layers, with activation quantization included, the four projections of a FLUX block take 47.2 ms with int8 and 58.4 ms with int4, against 81.5 ms in bf16: 1.73× and 1.40× faster. MLX's own 4-bit format, which mflux q4 uses, takes 78.1 ms: it stores 4-bit weights but does its arithmetic in bf16, so it saves memory but not time.

SVDQuant layers on the accelerators

An SVDQuant layer splits each weight into a small low-rank branch kept in 16-bit and a 4-bit residual with one scale per 64 input channels. On CUDA, the residual runs as a 4-bit × 4-bit matmul. Apple GPUs have no 4-bit matmul, but the M5's Neural Accelerators run int8 matmuls, reachable from Metal 4 shaders.

  • Activations are smoothed and quantized to int8 per token, a finer grid than the 4-bit activations the CUDA kernel uses.
  • Weights keep their 4-bit values. Each per-group scale is split into a per-channel scale and an integer-domain multiplier, so the int32 accumulator can run over the whole input dimension in one pass.
  • The epilogue applies the activation and channel scales, the bias, and the low-rank branch once, in the same kernel.

Two weight backends share that kernel. int8 stores the weights pre-expanded to int8, twice the memory of int4, and is the fastest. int4 keeps them 4-bit and expands them inside the kernel, about 20% slower per layer. backend="auto" picks int8 when the model fits in half of the GPU's memory budget, otherwise int4. On a 24 GB Mac that means int8 for Klein 4B and int4 for Klein 9B and FLUX.1-dev. Macs without Neural Accelerators fall back to a dequantized path.

The AdaNorm modulation layers are AWQ W4A16. On MLX they map losslessly onto MLX's own 4-bit format and its built-in kernel; on PyTorch MPS they use PyTorch's int4 matmul for MPS.

Attention

With the projections running in int8, attention and the small operations around it become a large share of each step. Two changes keep them on the accelerators as well.

Before attention, FLUX normalizes each head of the queries and keys (RMSNorm) and applies rotary position embeddings. Both runtimes ran this as a chain of float32 elementwise operations with reshapes in between, which costs about as much as attention itself on Klein. It is now a single kernel that reads q, k and v straight out of the projection's output and writes them in the layout attention expects. The same kernel source is built by MLX and by PyTorch.

Attention itself runs fast on MLX: its built-in kernel uses the Neural Accelerators and reaches about 13 TFLOPS of the 15 available in bf16. PyTorch's attention on MPS runs at about 3.5 TFLOPS on the M5. MLX's kernel source is MIT-licensed, so the PyTorch runtime compiles a copy of it at load time and calls it on PyTorch tensors, without MLX installed. At Klein's shape (24 heads, 4,608 tokens) one attention call goes from 74 ms to 20 ms, and its error against a float32 reference is about a third of PyTorch's.

Latency and memory

M5, 24 GB, on AC power. 1024², seed 42, one warm-up image, then the median step time over 3 prompts (2 for Klein 9B and FLUX.1-dev). Klein runs 4 steps, FLUX.1-dev 20 steps at guidance 3.5. Every MLX configuration uses the same text encoder and VAE (mflux's q4) and the same 1 GB cap on MLX's buffer cache.

MLX

modeltransformerstep ↓vs mflux q4 ↑peak memory ↓
FLUX.2 Klein 4Bbf163.68 s0.95×11.1 GB
FLUX.2 Klein 4Bmflux q43.50 s—5.5 GB
FLUX.2 Klein 4BNunchaku int81.81 s1.93×7.5 GB
FLUX.2 Klein 4BNunchaku int42.08 s1.69×5.5 GB
FLUX.2 Klein 9Bmflux q47.09 s—10.3 GB
FLUX.2 Klein 9BNunchaku int84.73 s1.50×15.7 GB
FLUX.2 Klein 9BNunchaku int44.39 s1.61×10.4 GB
FLUX.1-devmflux q411.28 s—21.9 GB
FLUX.1-devNunchaku int47.70 s1.47×22.7 GB

Not every model has every row because some configurations don't fit in 24 GB. A bf16 transformer is 18 GB for Klein 9B and 24 GB for FLUX.1-dev, too large to run next to the text encoder. FLUX.1-dev int8 would need about 7 GB more than int4 for the expanded weights, roughly 29 GB at peak, so it isn't shown; auto picks int4 for it.

Peak memory is the whole pipeline, text encoder and VAE included. On FLUX.1-dev both peaks come from the VAE decode at 1024², which adds about 12 GB in both pipelines; during denoising, Nunchaku's transformer step adds less than mflux's. Klein 9B int8 is slower than int4 on 24 GB because its int8 weights push the pipeline into swap, which is why auto picks int4 there.

PyTorch MPS

modeltransformerstep ↓vs bf16 ↑allocated ↓MPS driver ↓
FLUX.2 Klein 4Bbf164.94 s—10.8 GB17.1 GB
FLUX.2 Klein 4BNunchaku int82.22 s2.23×7.5 GB14.5 GB
FLUX.2 Klein 4BNunchaku int42.42 s2.04×5.5 GB11.8 GB

"Allocated" is memory held by tensors during denoising, including the text encoder (4-bit NF4 in the lite-infer repo) and VAE. "MPS driver" adds PyTorch's MPS allocator cache, which keeps freed buffers for reuse. On Klein 4B, MLX is about 1.2× faster than PyTorch MPS with the same backend.

Visual comparison

Each cell is PSNR in dB ↑ and SSIM ↑ against the bf16 transformer of the same runtime: same prompt, seed 42, text encoder and VAE, so only the transformer differs. The last column lists PSNR per prompt.

MLX (mflux pipeline, q4 text encoder)

transformerPSNR ↑SSIM ↑per prompt (dB)
mflux q417.530.76915.2 / 20.5 / 16.9
Nunchaku int8 = int4 (MLX)17.800.77814.6 / 18.6 / 20.2

PyTorch MPS (Diffusers pipeline)

transformerPSNR ↑SSIM ↑per prompt (dB)
Nunchaku int8 = int4 (MPS)17.320.74914.2 / 20.4 / 17.3

Nunchaku and mflux q4 are on par: the gap in mean PSNR is smaller than the spread between prompts, and each wins on some prompts. Absolute PSNR is modest for every 4-bit option because a 4-step distilled model turns small numerical differences into visible changes in pose and layout, as in the CUDA results of the previous post. The int8 and int4 backends produce bit-identical images: the int4 kernel expands the 4-bit weights to exactly the int8 values, so the two differ only in memory and speed. The grids below show one Nunchaku column for both.

FLUX.2 Klein 4B on MLX: bf16, mflux q4, Nunchaku
FLUX.2 Klein 4B on MLX: bf16 transformer, mflux q4, Nunchaku (int8 and int4 give the same image). Same prompt, seed, text encoder and VAE in every column. Both 4-bit options keep the sign text and both move the frisbee out of the dog's mouth; Nunchaku keeps the bf16 lace and table layout in the still life, mflux q4 spreads the lace across the table. Click to open at full size.
Klein 4B sign crop on MLX
Prompt 1 on MLX, 2× crop of the sign: bf16, mflux q4, Nunchaku. All three render the text correctly.
FLUX.2 Klein 4B on PyTorch MPS
FLUX.2 Klein 4B on PyTorch MPS with the Diffusers pipeline: bf16 and Nunchaku (int8 and int4 give the same image).

Compare images within one runtime. MLX and PyTorch MPS draw the initial noise differently and use different text encoders (mflux's q4 and the repo's NF4), so the same seed gives different compositions across runtimes.

Klein 9B and FLUX.1-dev have no bf16 column: their bf16 transformers (18 and 24 GB) do not fit a 24 GB Mac next to the text encoder.

FLUX.2 Klein 9B on MLX
FLUX.2 Klein 9B on MLX: mflux q4 and Nunchaku (int8 and int4 give the same image).
FLUX.1-dev on MLX
FLUX.1-dev on MLX, 20 steps: mflux q4 and Nunchaku int4.

Image editing

FLUX.2 Klein also edits images: the input image is encoded by the VAE and its tokens are appended to the output image's tokens, so the transformer attends to both. The same Nunchaku checkpoint loads into mflux's edit pipeline (Flux2KleinEdit) with no changes. Each edit below starts from one of the bf16 text-to-image results above, with seed 42, 4 steps and a 1024² output:

  1. Change the text on the cardboard sign to say JAZZ NIGHT.
  2. Replace the oranges with green apples.
  3. Make it a snowy winter day, with snow covering the meadow.
Klein 4B image edits on MLX
FLUX.2 Klein 4B image editing on MLX. Rows: the three instructions above. Columns: input image, bf16 transformer, mflux q4, Nunchaku (int8 and int4 give the same image). Click to open at full size.
transformerstep ↓vs mflux q4 ↑peak memory ↓
bf167.78 s0.97×11.9 GB
mflux q47.54 s—7.6 GB
Nunchaku int84.66 s1.62×11.9 GB
Nunchaku int45.25 s1.44×9.9 GB

Median step time over the three edits, after a warm-up edit. An edit step carries the input image's tokens as well as the output's, twice the image tokens of text-to-image, and attention cost grows faster than that: 5.25 s against 2.08 s per step for Nunchaku int4. The table's peak memory comes from the VAE encoding the 1024² input image, which needs about 5.7 GB in both pipelines, not from denoising: during a step the two peak within 0.1 GB of each other. The 2.3 GB difference is the transformer's weights. mflux loads its weights lazily from the checkpoint file and they are not resident while the VAE runs; Nunchaku converts its weights to the kernel layout once and keeps them in memory between images.

transformerPSNR ↑SSIM ↑per edit (dB)
mflux q431.600.97733.1 / 35.9 / 25.8
Nunchaku int8 = int4 (MLX)28.460.96330.3 / 35.6 / 19.5

PSNR and SSIM against the bf16 edit. All three transformers apply every edit and keep the rest of the input image. Here mflux q4 stays closer to the bf16 result than Nunchaku, 31.6 dB against 28.5 dB, and most of the gap is the snow edit, which repaints the whole background. The int8 and int4 backends again produce bit-identical images, so the grid shows one Nunchaku column.

Try it

MLX

# pip install "nunchaku-lite[mlx]" mflux
import mlx.core as mx
from mflux.models.common.config.model_config import ModelConfig
from mflux.models.flux2.variants import Flux2Klein
from nunchaku_lite.mlx import load_nunchaku_pipeline

mx.set_cache_limit(1 << 30)  # keep MLX's buffer cache from pushing a 24 GB Mac into swap
model = load_nunchaku_pipeline(
    "mflux-community/flux2-klein-9b-mflux-q4",  # text encoder, VAE, scheduler
    pipeline_cls=Flux2Klein,
    checkpoint="lite-infer/flux.2-klein-9b-nunchaku-lite-int4-r32",  # only transformer/ is downloaded
    backend="auto",  # int8 or int4 from available memory
    model_config=ModelConfig.flux2_klein_9b(),
)
image = model.generate_image(seed=42, prompt="A golden retriever mid-leap catching a frisbee",
                             num_inference_steps=4, height=1024, width=1024, guidance=1.0)
image.image.save("klein9b.png")

For editing, load the same checkpoint into mflux's edit pipeline and pass the input image:

from mflux.models.flux2.variants.edit.flux2_klein_edit import Flux2KleinEdit

model = load_nunchaku_pipeline(
    "mflux-community/flux2-klein-4b-mflux-q4",
    pipeline_cls=Flux2KleinEdit,
    checkpoint="lite-infer/flux.2-klein-4b-nunchaku-lite-int4-r32",
    backend="auto",
    model_config=ModelConfig.flux2_klein_4b(),
)
image = model.generate_image(seed=42, prompt="Change the text on the cardboard sign to say JAZZ NIGHT.",
                             image_paths=["musician.png"], num_inference_steps=4, height=1024, width=1024, guidance=1.0)
image.image.save("musician_edit.png")

PyTorch MPS

# pip install "nunchaku-lite[torch]"   # the Metal kernels download from the Hugging Face Hub on first use
import torch
from diffusers import Flux2KleinPipeline
from nunchaku_lite import load_nunchaku_pipeline

repo = "lite-infer/flux.2-klein-4b-nunchaku-lite-int4-r32"
pipe = load_nunchaku_pipeline(
    repo,
    pipeline_cls=Flux2KleinPipeline,
    checkpoint="lite-infer/flux.2-klein-4b-nunchaku-lite-int4-r32/transformer/diffusion_pytorch_model.safetensors",
    torch_dtype=torch.bfloat16,
    device="mps",
    backend="auto",
)
pipe.to("mps")
image = pipe("A golden retriever mid-leap catching a frisbee", num_inference_steps=4, guidance_scale=1.0).images[0]
image.save("klein4b.png")

Limitations and next steps

  • The int8 and int4 backends need an M5-class GPU (Neural Accelerators) and macOS 26. Earlier Apple GPUs use the dequantized fallback.
  • INT4 checkpoints only; NVFP4 is not supported on Apple silicon yet.
  • The prebuilt Metal kernels on the Hugging Face Hub cover PyTorch 2.10 to 2.15; other PyTorch versions need a local build of nunchaku-lite-kernels.
  • MLX adapters exist for FLUX.1 and FLUX.2 Klein; other families can use the generic manifest adapter.
  • Attention runs in bf16. Running its query-key product in int8, as SageAttention does on CUDA, would use the accelerators' faster integer path.
Share: X (Twitter) Facebook LinkedIn