The same 4-bit SVDQuant checkpoints that run on CUDA now run on an M5 Mac, through PyTorch MPS (Diffusers) and MLX (mflux). New Metal kernels run the 4-bit layers as int8 matrix multiplies on the GPU's Neural Accelerators. All numbers come from an M5 with 24 GB, generating at 1024².
↑ higher is better · ↓ lower is better. Speed-ups are per denoising step against mflux q4, the existing 4-bit option on MLX.
Two runtimes
Nunchaku Lite loads Nunchaku SVDQuant checkpoints into existing pipelines instead of shipping its own. On a Mac there are two pipeline ecosystems, so there are two runtimes.
| PyTorch MPS | MLX | |
|---|---|---|
| Pipelines | Diffusers | mflux |
| Model adapters | the CUDA adapters, unchanged | written for mflux's modules (FLUX.1, FLUX.2 Klein, generic manifest) |
| Quantized layers | Metal kernels in nunchaku-lite-kernels | Metal kernels through mx.fast.metal_kernel |
| Install | pip install "nunchaku-lite[torch]" (Metal kernels download from the Hugging Face Hub) | pip install "nunchaku-lite[mlx]" mflux |
| Best for | Diffusers features: LoRA, CPU offload, every Diffusers pipeline | speed on the Klein models, and comparison with mflux q4 |
Both read the same checkpoints, including the lite-infer repos with a runtime manifest and older lite checkpoints that list their quantized layers in config.json.
How the 4-bit layers run on M5
The M5 Neural Accelerators
Every GPU core in the M5 has a Neural Accelerator, a unit built for matrix multiplication. Metal 4 shaders drive it through tensor operations (matmul2d in Metal Performance Primitives); earlier Apple GPUs run matrix multiplies on the ordinary shader cores. The accelerators are at their fastest with integers. Peak throughput of a bare matrix multiply on the M5:
| operands (accumulator) | throughput ↑ |
|---|---|
| int8 × int8 (int32) | ~30 TOPS |
| int8 × int4 (int32) | ~30 TOPS |
| fp16 or bf16 × anything | ~15 TFLOPS |
Two consequences shape the design. Int8 is twice as fast as bf16, so a layer gains by running its matmul in int8. And a 4-bit operand runs no faster than an 8-bit one (int8 × int4 matches int8 × int8), so 4-bit activations would add error without adding speed. So the layers run as W4A8 (4-bit weights, int8 activations) or W8A8 (the same 4-bit values stored expanded to int8), not as Nunchaku's CUDA W4A4.
The fused kernels come close to that ceiling. Here they are against a bf16 matmul at the projection shapes of a FLUX block for a 1024² image (4,608 tokens), with the scales, bias and low-rank branch all inside the int8 kernels:
| projection (K → N) | bf16 matmul | W8A8 kernel | W4A8 kernel |
|---|---|---|---|
| QKV 3072 → 9216 | 16.6 ms · 15.7 T | 9.1 ms · 28.6 T | 11.6 ms · 22.4 T |
| MLP in 3072 → 12288 | 22.0 ms · 15.8 T | 12.1 ms · 28.8 T | 15.5 ms · 22.5 T |
| MLP out 12288 → 3072 | 32.1 ms · 10.8 T | 12.7 ms · 27.5 T | 17.0 ms · 20.4 T |
| attention out 3072 → 3072 | 5.7 ms · 15.3 T | 3.2 ms · 27.2 T | 4.0 ms · 21.5 T |
T is trillions of operations per second, with a multiply-add counted as two. The W4A8 kernel is slower than W8A8 because it expands each 4-bit weight to its int8 value in the shader before the accelerator sees it. As whole layers, with activation quantization included, the four projections of a FLUX block take 47.2 ms with int8 and 58.4 ms with int4, against 81.5 ms in bf16: 1.73× and 1.40× faster. MLX's own 4-bit format, which mflux q4 uses, takes 78.1 ms: it stores 4-bit weights but does its arithmetic in bf16, so it saves memory but not time.
SVDQuant layers on the accelerators
An SVDQuant layer splits each weight into a small low-rank branch kept in 16-bit and a 4-bit residual with one scale per 64 input channels. On CUDA, the residual runs as a 4-bit × 4-bit matmul. Apple GPUs have no 4-bit matmul, but the M5's Neural Accelerators run int8 matmuls, reachable from Metal 4 shaders.
- Activations are smoothed and quantized to int8 per token, a finer grid than the 4-bit activations the CUDA kernel uses.
- Weights keep their 4-bit values. Each per-group scale is split into a per-channel scale and an integer-domain multiplier, so the int32 accumulator can run over the whole input dimension in one pass.
- The epilogue applies the activation and channel scales, the bias, and the low-rank branch once, in the same kernel.
Two weight backends share that kernel. int8 stores the weights pre-expanded to int8, twice the memory of int4, and is the fastest. int4 keeps them 4-bit and expands them inside the kernel, about 20% slower per layer. backend="auto" picks int8 when the model fits in half of the GPU's memory budget, otherwise int4. On a 24 GB Mac that means int8 for Klein 4B and int4 for Klein 9B and FLUX.1-dev. Macs without Neural Accelerators fall back to a dequantized path.
The AdaNorm modulation layers are AWQ W4A16. On MLX they map losslessly onto MLX's own 4-bit format and its built-in kernel; on PyTorch MPS they use PyTorch's int4 matmul for MPS.
Attention
With the projections running in int8, attention and the small operations around it become a large share of each step. Two changes keep them on the accelerators as well.
Before attention, FLUX normalizes each head of the queries and keys (RMSNorm) and applies rotary position embeddings. Both runtimes ran this as a chain of float32 elementwise operations with reshapes in between, which costs about as much as attention itself on Klein. It is now a single kernel that reads q, k and v straight out of the projection's output and writes them in the layout attention expects. The same kernel source is built by MLX and by PyTorch.
Attention itself runs fast on MLX: its built-in kernel uses the Neural Accelerators and reaches about 13 TFLOPS of the 15 available in bf16. PyTorch's attention on MPS runs at about 3.5 TFLOPS on the M5. MLX's kernel source is MIT-licensed, so the PyTorch runtime compiles a copy of it at load time and calls it on PyTorch tensors, without MLX installed. At Klein's shape (24 heads, 4,608 tokens) one attention call goes from 74 ms to 20 ms, and its error against a float32 reference is about a third of PyTorch's.
Latency and memory
M5, 24 GB, on AC power. 1024², seed 42, one warm-up image, then the median step time over 3 prompts (2 for Klein 9B and FLUX.1-dev). Klein runs 4 steps, FLUX.1-dev 20 steps at guidance 3.5. Every MLX configuration uses the same text encoder and VAE (mflux's q4) and the same 1 GB cap on MLX's buffer cache.
MLX
| model | transformer | step ↓ | vs mflux q4 ↑ | peak memory ↓ |
|---|---|---|---|---|
| FLUX.2 Klein 4B | bf16 | 3.68 s | 0.95× | 11.1 GB |
| FLUX.2 Klein 4B | mflux q4 | 3.50 s | — | 5.5 GB |
| FLUX.2 Klein 4B | Nunchaku int8 | 1.81 s | 1.93× | 7.5 GB |
| FLUX.2 Klein 4B | Nunchaku int4 | 2.08 s | 1.69× | 5.5 GB |
| FLUX.2 Klein 9B | mflux q4 | 7.09 s | — | 10.3 GB |
| FLUX.2 Klein 9B | Nunchaku int8 | 4.73 s | 1.50× | 15.7 GB |
| FLUX.2 Klein 9B | Nunchaku int4 | 4.39 s | 1.61× | 10.4 GB |
| FLUX.1-dev | mflux q4 | 11.28 s | — | 21.9 GB |
| FLUX.1-dev | Nunchaku int4 | 7.70 s | 1.47× | 22.7 GB |
Not every model has every row because some configurations don't fit in 24 GB. A bf16 transformer is 18 GB for Klein 9B and 24 GB for FLUX.1-dev, too large to run next to the text encoder. FLUX.1-dev int8 would need about 7 GB more than int4 for the expanded weights, roughly 29 GB at peak, so it isn't shown; auto picks int4 for it.
Peak memory is the whole pipeline, text encoder and VAE included. On FLUX.1-dev both peaks come from the VAE decode at 1024², which adds about 12 GB in both pipelines; during denoising, Nunchaku's transformer step adds less than mflux's. Klein 9B int8 is slower than int4 on 24 GB because its int8 weights push the pipeline into swap, which is why auto picks int4 there.
PyTorch MPS
| model | transformer | step ↓ | vs bf16 ↑ | allocated ↓ | MPS driver ↓ |
|---|---|---|---|---|---|
| FLUX.2 Klein 4B | bf16 | 4.94 s | — | 10.8 GB | 17.1 GB |
| FLUX.2 Klein 4B | Nunchaku int8 | 2.22 s | 2.23× | 7.5 GB | 14.5 GB |
| FLUX.2 Klein 4B | Nunchaku int4 | 2.42 s | 2.04× | 5.5 GB | 11.8 GB |
"Allocated" is memory held by tensors during denoising, including the text encoder (4-bit NF4 in the lite-infer repo) and VAE. "MPS driver" adds PyTorch's MPS allocator cache, which keeps freed buffers for reuse. On Klein 4B, MLX is about 1.2× faster than PyTorch MPS with the same backend.
Visual comparison
Each cell is PSNR in dB ↑ and SSIM ↑ against the bf16 transformer of the same runtime: same prompt, seed 42, text encoder and VAE, so only the transformer differs. The last column lists PSNR per prompt.
MLX (mflux pipeline, q4 text encoder)
| transformer | PSNR ↑ | SSIM ↑ | per prompt (dB) |
|---|---|---|---|
| mflux q4 | 17.53 | 0.769 | 15.2 / 20.5 / 16.9 |
| Nunchaku int8 = int4 (MLX) | 17.80 | 0.778 | 14.6 / 18.6 / 20.2 |
PyTorch MPS (Diffusers pipeline)
| transformer | PSNR ↑ | SSIM ↑ | per prompt (dB) |
|---|---|---|---|
| Nunchaku int8 = int4 (MPS) | 17.32 | 0.749 | 14.2 / 20.4 / 17.3 |
Nunchaku and mflux q4 are on par: the gap in mean PSNR is smaller than the spread between prompts, and each wins on some prompts. Absolute PSNR is modest for every 4-bit option because a 4-step distilled model turns small numerical differences into visible changes in pose and layout, as in the CUDA results of the previous post. The int8 and int4 backends produce bit-identical images: the int4 kernel expands the 4-bit weights to exactly the int8 values, so the two differ only in memory and speed. The grids below show one Nunchaku column for both.



Compare images within one runtime. MLX and PyTorch MPS draw the initial noise differently and use different text encoders (mflux's q4 and the repo's NF4), so the same seed gives different compositions across runtimes.
Klein 9B and FLUX.1-dev have no bf16 column: their bf16 transformers (18 and 24 GB) do not fit a 24 GB Mac next to the text encoder.


Image editing
FLUX.2 Klein also edits images: the input image is encoded by the VAE and its tokens are appended to the output image's tokens, so the transformer attends to both. The same Nunchaku checkpoint loads into mflux's edit pipeline (Flux2KleinEdit) with no changes. Each edit below starts from one of the bf16 text-to-image results above, with seed 42, 4 steps and a 1024² output:
- Change the text on the cardboard sign to say JAZZ NIGHT.
- Replace the oranges with green apples.
- Make it a snowy winter day, with snow covering the meadow.

| transformer | step ↓ | vs mflux q4 ↑ | peak memory ↓ |
|---|---|---|---|
| bf16 | 7.78 s | 0.97× | 11.9 GB |
| mflux q4 | 7.54 s | — | 7.6 GB |
| Nunchaku int8 | 4.66 s | 1.62× | 11.9 GB |
| Nunchaku int4 | 5.25 s | 1.44× | 9.9 GB |
Median step time over the three edits, after a warm-up edit. An edit step carries the input image's tokens as well as the output's, twice the image tokens of text-to-image, and attention cost grows faster than that: 5.25 s against 2.08 s per step for Nunchaku int4. The table's peak memory comes from the VAE encoding the 1024² input image, which needs about 5.7 GB in both pipelines, not from denoising: during a step the two peak within 0.1 GB of each other. The 2.3 GB difference is the transformer's weights. mflux loads its weights lazily from the checkpoint file and they are not resident while the VAE runs; Nunchaku converts its weights to the kernel layout once and keeps them in memory between images.
| transformer | PSNR ↑ | SSIM ↑ | per edit (dB) |
|---|---|---|---|
| mflux q4 | 31.60 | 0.977 | 33.1 / 35.9 / 25.8 |
| Nunchaku int8 = int4 (MLX) | 28.46 | 0.963 | 30.3 / 35.6 / 19.5 |
PSNR and SSIM against the bf16 edit. All three transformers apply every edit and keep the rest of the input image. Here mflux q4 stays closer to the bf16 result than Nunchaku, 31.6 dB against 28.5 dB, and most of the gap is the snow edit, which repaints the whole background. The int8 and int4 backends again produce bit-identical images, so the grid shows one Nunchaku column.
Try it
MLX
# pip install "nunchaku-lite[mlx]" mflux
import mlx.core as mx
from mflux.models.common.config.model_config import ModelConfig
from mflux.models.flux2.variants import Flux2Klein
from nunchaku_lite.mlx import load_nunchaku_pipeline
mx.set_cache_limit(1 << 30) # keep MLX's buffer cache from pushing a 24 GB Mac into swap
model = load_nunchaku_pipeline(
"mflux-community/flux2-klein-9b-mflux-q4", # text encoder, VAE, scheduler
pipeline_cls=Flux2Klein,
checkpoint="lite-infer/flux.2-klein-9b-nunchaku-lite-int4-r32", # only transformer/ is downloaded
backend="auto", # int8 or int4 from available memory
model_config=ModelConfig.flux2_klein_9b(),
)
image = model.generate_image(seed=42, prompt="A golden retriever mid-leap catching a frisbee",
num_inference_steps=4, height=1024, width=1024, guidance=1.0)
image.image.save("klein9b.png")For editing, load the same checkpoint into mflux's edit pipeline and pass the input image:
from mflux.models.flux2.variants.edit.flux2_klein_edit import Flux2KleinEdit
model = load_nunchaku_pipeline(
"mflux-community/flux2-klein-4b-mflux-q4",
pipeline_cls=Flux2KleinEdit,
checkpoint="lite-infer/flux.2-klein-4b-nunchaku-lite-int4-r32",
backend="auto",
model_config=ModelConfig.flux2_klein_4b(),
)
image = model.generate_image(seed=42, prompt="Change the text on the cardboard sign to say JAZZ NIGHT.",
image_paths=["musician.png"], num_inference_steps=4, height=1024, width=1024, guidance=1.0)
image.image.save("musician_edit.png")PyTorch MPS
# pip install "nunchaku-lite[torch]" # the Metal kernels download from the Hugging Face Hub on first use
import torch
from diffusers import Flux2KleinPipeline
from nunchaku_lite import load_nunchaku_pipeline
repo = "lite-infer/flux.2-klein-4b-nunchaku-lite-int4-r32"
pipe = load_nunchaku_pipeline(
repo,
pipeline_cls=Flux2KleinPipeline,
checkpoint="lite-infer/flux.2-klein-4b-nunchaku-lite-int4-r32/transformer/diffusion_pytorch_model.safetensors",
torch_dtype=torch.bfloat16,
device="mps",
backend="auto",
)
pipe.to("mps")
image = pipe("A golden retriever mid-leap catching a frisbee", num_inference_steps=4, guidance_scale=1.0).images[0]
image.save("klein4b.png")Limitations and next steps
- The int8 and int4 backends need an M5-class GPU (Neural Accelerators) and macOS 26. Earlier Apple GPUs use the dequantized fallback.
- INT4 checkpoints only; NVFP4 is not supported on Apple silicon yet.
- The prebuilt Metal kernels on the Hugging Face Hub cover PyTorch 2.10 to 2.15; other PyTorch versions need a local build of
nunchaku-lite-kernels. - MLX adapters exist for FLUX.1 and FLUX.2 Klein; other families can use the generic manifest adapter.
- Attention runs in bf16. Running its query-key product in int8, as SageAttention does on CUDA, would use the accelerators' faster integer path.