Qwen-Image NVFP4 on a 16 GB Ada GPU: NVFP4 vs NF4, weight error, speed and the NF4 grain

Playing summary

NVIDIA released Qwen-Image in NVFP4, its 4-bit format for Blackwell; the card dates the release October 12. The model card lists a single compatible piece of hardware, the GB300. I have a 16 GB RTX 4070 Ti SUPER, an Ada card without FP4 tensor cores, and I use it to generate this blog’s covers with Qwen-Image in NF4. I wanted to know two things: whether those weights are any use on my card, and whether they beat the NF4 I already use.

Narrated: a loupe goes down from a grainy Qwen-Image NF4 photo. The obvious suspect, the 112 modulation matrices, turns out innocent: switching them to NVFP4 leaves the grain. The culprit is 8 matrices in the edge blocks; keeping them in bf16 lowers noise in 19 of 21 images at the same time per image.Watch it in the reel viewer →

What NVFP4 is and how it differs from NF4

Both store each weight in 4 bits, and both need scales to get back to the real value. The difference is the block size and what the 16 codes stand for.

NVFP4 stores a 4-bit floating-point number (E2M1: sign, two exponent bits and one mantissa bit, values from 0 to 6). Each block of 16 weights has its own FP8 scale, and each matrix one FP32 global scale. On Blackwell, the tensor cores multiply in FP4 directly, with weights and activations in 4 bits.
NF4, the QLoRA format bitsandbytes uses, puts its 16 levels on the quantiles of a normal distribution, and scales each block of 64 weights by its absolute maximum.

This is what 64 real weights of a modulation layer in block 30 look like, taken from both files:

64 real weights of transformer_blocks.30.img_mod.1 (row 0): bf16 base against what NVFP4 and NF4 give back

▮ bf16 base · ● NVFP4 expanded (error on these 64: 9.1 %) ·◆ NF4 expanded (error on these 64: 8.7 %)

First NVFP4 block, as stored in the file

0xdccode 1100-2.0code 1101-3.0
0x93code 00111.5code 1001-0.5
0x5fcode 1111-6.0code 01013.0
0x57code 01116.0code 01013.0
0x29code 1001-0.5code 00101.0
0x4bcode 1011-1.5code 01002.0
0x01code 00010.5code 00000.0
0xf9code 1001-0.5code 1111-6.0

weight = E2M1 × block scale × global scale → -2.0 × 4.5 × 0.0017439 = -0.01570 (base -0.01941)

NVFP4 uses four scales where NF4 uses one, so each group of 16 fits its own range. In this segment the two land close, and NF4 is even a little closer (8.7 % against 9.1 %). The differences show up when looking at whole matrices, further down. In memory they cost the same: 4 bits per weight plus 8 bits of scale every 16 in NVFP4, or 32 every 64 in NF4 without double quantization (the way diffusers loads it here). That is 4.5 bits per weight in both.

The official route, on a 16 GB card

I ran the model card’s container as is (TensorRT-LLM VisualGen 1.3.0rc28 with trtllm-serve), first with the card’s configuration and then with cpu_offload_config enabled, which changed nothing visible in the load. Both times it ran out of memory at the same point, with 14.00 GiB already allocated, while loading the text encoder after the transformer. That encoder ships in bf16 and its file weighs 15.4 GiB, almost the whole card.

Reading the container’s code I found something curious: it ships a Marlin kernel for NVFP4 that declares support for sm_89 to sm_99 and for sm_120 and sm_121, a range that includes mine, although it is only used if the layer explicitly allows it. With the pipeline never loaded, I never got to use it. The lines are in the gist.

Running the NVFP4 weights on Ada

If the hardware does not multiply in FP4, the packed matrix can stay on the GPU and be expanded to bf16 right before each multiplication. Activations stay in bf16, so this is weight-only NVFP4 (W4A16), not the W4A4 Blackwell runs. I wrote an expander of about 50 lines: a 256-entry table turns each byte into its two E2M1 values, and each block’s scale multiplies a 16-by-16 view. On a 12,288×3,072 matrix it takes 0.96 ms, against 4.15 ms for the direct version.

What my expander runs before every multiply: 0.96 ms for a 12,288×3,072 matrix, in plain PyTorch

Before generating, two controls. Reading the nibbles in the wrong order raises the error from 9.47 % to 141 %, so the order matters and mine is the right one. And the text encoder in NVIDIA’s repository is bit-for-bit identical to the base model’s (459 of 459 tensors in the first shard), so both variants get the same encoded text.

With the text encoder and the transformer in the same process, NF4 ran out of memory even after I freed the encoder. So I encode the prompts in a separate process that saves the embeddings and exits before the transformer loads.

Memory on a 15.57 GiB RTX 4070 Ti SUPER, and seconds per 1344×768 image

Expanded NVFP4 takes 43 s more per image than NF4: without FP4 tensor cores, the format only serves to store the weights, and the expansion is paid on every pass. The extra 2 GiB, on the other hand, are hardly the format’s. NVIDIA left blocks 0, 1, 58 and 59 in bf16, and that alone adds about 1.8 GiB against having them in 4 bits. The bitsandbytes NF4 copy quantizes 846 matrices: NVIDIA’s 784, those of the four blocks, and six more outside the blocks.

Quality: the error of each matrix

Before generating a single image I measured how far each format lands from the original weights, on the 784 matrices NVIDIA quantized (14 per block, blocks 2 to 57). The error is relative: the norm of the difference divided by the norm of the original.

Metric	NVFP4	NF4
Median error	9.47 %	9.33 %
90th percentile (nearest rank)	9.52 %	10.15 %
Maximum	9.59 %	12.89 %
Matrices where it lands closer	254	530

At the median NF4 wins. In the worst case NVFP4 wins, and it is very even: all its matrices fall between 6.3 and 9.6 %. NF4’s worst cases are not spread at random.

784 quantized matrices: the 112 modulation ones hold 33 % of the weights and a kurtosis ~240× higher; that is where NF4 falls behind

share of quantized weights

■ modulation (img_mod.1 / txt_mod.1): 6.34 B · median error: NF4 11.3 % · NVFP4 9.3 %
● the other 672: 12.68 B · median error: NF4 9.3 % · NVFP4 9.5 %

The modulation layers (img_mod.1 and txt_mod.1) compute, from the diffusion step, the shift, scale and gate applied to each block, as in DiT’s adaLN. They are 112 of the 784 matrices, but each one has 18,432×3,072 weights: together they add up to 6.34 of the 19.02 billion quantized weights, a third. Their weights have very heavy tails: the median kurtosis is 157.6, against 0.66 for the rest. On those layers NF4 has a median error of 11.3 % and NVFP4 9.3 %. On the rest, NF4 does better (9.27 % against 9.48 %). Within the 112, NF4’s error rises with kurtosis (rank correlation 0.75) and with depth: its median is 9.4 % in blocks 2 to 13, where NF4 still beats NVFP4, 10.8 % in blocks 14 to 29 and 12.1 % from 30 to 57.

NF4 is designed for normally distributed weights, and these are not. I do not have the exact mechanism: the ratio of the maximum to the standard deviation in each block of 64 is almost the same in the modulation layers and in the rest (2.65 against 2.61), so it is not one extreme value inside each block. What sets these layers apart is the shape of the whole matrix. As shown further down, this worst error is not what produces the grain.

Quality: the text inside the image

The same six Spanish-text prompts as the previous post, with seeds 7, 21 and 88, at 1344×768, with the steps (50), CFG (4) and negative prompt from NVIDIA’s card. I checked every image at full size with the same criterion: ✓ if every character came out as asked, ≈ if only a space is off, ✗ if there is an extra, missing or wrong character.

NF4 (bitsandbytes), base, poster, seed 7 — Requested text: NF4 against expanded NVFP4, the same base Qwen-Image on the same card

NVIDIA NVFP4, expanded, poster, seed 7 — Requested text: NF4 against expanded NVFP4, the same base Qwen-Image on the same card

NVFP4 wrote 6 of 18 and NF4 5. The rule I set in the measurement sheet before measuring said a difference of one image or less is a tie. Both fail on the same words: «rapido» without the accent, «modeles de decición», a mangled «PostgreSQL». Note: this is base Qwen-Image with 50 steps, not the 8-step 2.1-Turbo of my covers, which wrote 13 of 18 in the previous post.

What the rule did not measure: the grain

Looking at the images at full size shows something the rule did not measure: the NF4 ones are grainy, like a high-ISO photo, mostly on walls, chalkboards and blurred backgrounds. To go beyond the impression, I measured each image’s noise floor: subtract from the luminance its 3×3 mean, take the standard deviation of that residual in 16×16 patches and keep the 10th percentile, which corresponds to the flattest areas. I compare each image with the one with the same seed and prompt in the other variant. To the bench’s 18 I added the 3 sample prompts from NVIDIA’s card, which run at 512×512 and 1024×1024. NF4 comes out noisier than NVFP4 in 19 of 21 pairs, with a median of 0.557 against 0.420.

The first hypothesis, and why it was badly framed

The grain and NF4’s worst error showed up together, so I added an amendment to the measurement sheet before running the test, but after having seen the images: it is exploratory. The hypothesis was that the grain came from the modulation layers in NF4. The test took the modulation layers from NVIDIA’s file and lowered the noise in 19 of 21 pairs, with a median of 0.408. It met the rule.

The external audit found that this test changed two things at once. In blocks 2 to 57, 112 matrices went from NF4 to NVFP4. In blocks 0, 1, 58 and 59, which NVIDIA left unquantized, 8 matrices went from NF4 to bf16. Either change could explain the result.

The control

I registered a second amendment before running it, with two variants and a rule for each possible outcome: one with only the 112 layers in NVFP4 and another with only the 8 in bf16. Same 21 jobs, same embeddings.

Where each format sits: the pipeline on 16 GB, and the four variants block by block

NF4, etiqueta_s7, 256×256, 1:1 — The same 256 px of three images, at 1:1 (the window that is flattest in all four variants), and the noise floor of all 21

NF4 + 112 NVFP4 layers, etiqueta_s7, 256×256, 1:1 — The same 256 px of three images, at 1:1 (the window that is flattest in all four variants), and the noise floor of all 21

The 112 layers in NVFP4 do nothing: the noise drops against NF4 in 7 of 21 pairs and the median stays at 0.558, the same as NF4. The 8 matrices in bf16, on this bench, explain almost all of it: it drops in 19 of 21 and the median lands at 0.412, closer to NVFP4 than to NF4 in 20 of 21. The two images where it does not drop are the ones from NVIDIA’s card: the Chinese text one, which is almost all strokes that the metric cannot tell from grain, and the teapot photo, where they come out practically equal (0.411 and 0.412).

A third amendment, also registered before running it, split the output from the input: only the 4 matrices of blocks 58 and 59 in bf16. By the letter of its rule, the grain would be attributed to the final blocks, because the noise drops in 16 of 21 pairs (the threshold was 15). But it drops very little: the median lands at 0.548 against 0.557 for NF4, and in no image does it land closer to NVFP4 than to NF4. The rule counted direction and not size, and writing it that way was a mistake. Most of the effect needs blocks 0 and 1, alone or together with the final ones; I did not run the variant with only those two.

Those 8 matrices do not have a particularly high NF4 error: between 9.2 % and 11.4 %, less than the ones in the middle of the network, which reach 12.9 %. Their kurtosis ranges from 0.5 to 6,935. What sets them apart is where they are: in the first and last blocks. The error of each matrix on its own does not predict the damage to the image, and the weight-error table, which seemed to point at the culprit, was pointing at something else. NVIDIA left those four blocks unquantized in its file. The card does not say why, and this measurement suggests it made sense.

With those 8 matrices in bf16, the rest in NF4 takes 102.6 s per image (102.2 s NF4) and peaks at 11.7 GiB (11.1 GiB). I checked the text of its 18 images with the same criterion: 7 right (in one, «PORTÁTIL» has a tiny separate ® mark next to it, which I counted as right; read strictly it would be 6), against 5 for NF4 and 6 for NVFP4. That difference is not pre-registered and with 18 images I do not take it as real.

NF4 + 8 edge matrices in bf16, poster, seed 7 — NF4 against NF4 with the 8 edge matrices in bf16: same requested text, same seeds

Threats to validity

The three amendments are exploratory: I decided on them after seeing the grain. The second and third were registered before running them, with their rule written. They show which change removes the grain on this bench, with one model, one metric and 21 images.
The bitsandbytes NF4 copy quantizes 846 matrices and NVIDIA’s, 784. Six of the ones only NF4 quantizes sit outside the blocks (among them the diffusion-step embedding, with a kurtosis of 4,243), and they stay in NF4 in all my variants: they are not what causes the grain, but I did not measure whether they affect something else.
I did the text review alone, by eye. The 54 gallery images and the sheet for all 72 are published.
The noise metric is mine. I check it against what the 1:1 crops show, but it confuses dense text with grain, as in the Chinese image.
My covers use Qwen-Image 2.1-Turbo with 8 steps. I did not measure whether its NF4 version has the same grain.
NVFP4’s speed here says nothing about Blackwell, which runs it in native FP4.

Verdict

On a 16 GB Ada card, NVIDIA’s Qwen-Image NVFP4 is not worth it as shipped: the official route does not load, and expanded it runs slower than NF4 without writing better. What is useful in its file is not the 4 bits but a decision NVIDIA made: what to leave unquantized. If I use NF4 and keep in bf16 the same 8 matrices NVIDIA left out, the image comes out as clean as NVFP4, at less than half a second more per image and 0.6 GiB more. My first hypothesis, the one about the layers with the worst error, was wrong, and without the audit I would have published it. Next is measuring whether the 2.1-Turbo of my covers has the same grain and, if it does, applying the same recipe there.

Frequently asked questions

Does Qwen-Image NVFP4 work on a GPU that is not Blackwell?

The official route does not. NVIDIA's model card lists only Blackwell GB300 and TensorRT-LLM VisualGen 1.3.0rc28. On an RTX 4070 Ti SUPER (Ada, sm_89, 15.57 GiB) the official server ran out of memory loading the pipeline twice, also with CPU offload: the bf16 text encoder alone weighs 15.4 GiB. The weights can be used: I expanded them to bf16 on every pass inside diffusers and the image comes out fine.

How fast does NVFP4 run on a card without FP4 tensor cores?

Slower than NF4. At 1344×768 with 50 steps, expanded NVFP4 took 145.4 s per image with a 13.1 GiB peak; bitsandbytes NF4 took 102.2 s and 11.1 GiB (median of 18 images each). Without FP4 in hardware, NVFP4 is only a storage format and the expansion is paid on every pass.

Is NVFP4 more accurate than NF4?

Not at the median: over the 784 matrices NVIDIA quantized, the median relative error is 9.47 % for NVFP4 and 9.33 % for NF4, and NVFP4 lands closer to the original in only 254. The difference is in the worst case: NVFP4 never goes above 9.59 % while NF4 reaches 12.89 %, and those worst cases are the modulation layers.

Where does the grain in the NF4 images come from?

From 8 matrices: the modulation layers (img_mod.1 and txt_mod.1) of blocks 0, 1, 58 and 59, which NVIDIA left unquantized and bitsandbytes does quantize. With only those 8 in bf16, the noise in flat areas dropped against NF4 in 19 of 21 images. With the other 112 modulation layers in NVFP4, where NF4 has its worst error, it dropped in 7. These are exploratory tests, decided after seeing the grain.

Which one should I use on a 16 GB card?

In my tests, NF4 with those 8 matrices in bf16: 102.6 s per image and an 11.7 GiB peak, against 102.2 s and 11.1 GiB for NF4, and without its grain. Expanded NVFP4 adds nothing on Ada over that and is slower. On text, no variant pulled away from the others: between 5 and 7 of 18.

Can I reproduce it?

Yes. The NVFP4 expander, the generation script, the error, tail and noise computations, the image-by-image review and the timings are in a gist linked in the sources.

Sources

Qwen-Image-NVFP4, NVIDIA’s model card, with the supported hardware, the 784 matrices and the TensorRT-LLM recipe.
Introducing NVFP4, NVIDIA’s technical announcement, and the OCP Microscaling specification, which defines E2M1.
QLoRA, the paper that defines NF4 for normally distributed weights.
The Qwen-Image technical report and DiT, where adaLN modulation comes from.
TensorRT-LLM, the engine of the official route.
The expander, the scripts, the image-by-image review, the errors, the tails, the noise, the timings and the versions.

Measured on October 15, 2026 on a 16 GB RTX 4070 Ti SUPER (driver 595.71.05). Qwen/Qwen-Image revision 75e0b4be and nvidia/Qwen-Image-NVFP4 (ModelOpt 0.47.0); torch 2.14.1+cu130, diffusers 0.41.0.dev0, transformers 5.19.0, bitsandbytes 0.50.2; TensorRT-LLM 1.3.0rc28 for the official route. 1344×768, 50 steps, CFG 4, seeds 7, 21 and 88.

NVFP4 without Blackwell: I ran NVIDIA's Qwen-Image on an RTX 4070 Ti SUPER and the NF4 grain was in 8 matrices