NVFP4 vs FP8 vs GGUF on a 16 GB GPU without Blackwell: NVFP4 generates 2.3 times faster than BF16 and ties with BF16 on the full 1319-question GSM8K

When I type «nvfp4 vs», Google completes it with «fp8», «gguf», «q4_k_m» and «bf16». «mxfp8 vs» gets «fp8» and «nvfp4 vs bf16». It is the question anyone with a 16 GB card asks: which format to download. It is almost always answered by comparing different models, quantized by different people.

On October 16 NVIDIA released Nemotron Labs Dory 1.0, a 4-billion-parameter model, in four versions at once: BF16, NVFP4, MXFP8 and GGUF. The same model, quantized by its author. I measured it on an RTX 4070 Ti SUPER, an Ada card without FP4 hardware, for speed and for quality.

What I measured, and under which rules

Before the first run I wrote down a measurement sheet with the variants, the metrics and the decision rule. Dory has a «looped» architecture: according to its config.json, of 72 layers (16 input, 40 recurrent and 16 output, counting attention and MLP separately; NVIDIA’s card counts them as blocks of 8, 20 and 8), the 40 in the middle repeat. I left it at two loops, the value NVIDIA recommends, in every variant.

• In vLLM (from the PR that adds Dory, version 0.32.0rc1.dev319): BF16 as the reference, NVIDIA’s NVFP4 and MXFP8, and FP8 that vLLM quantizes while loading the BF16 weights (per-tensor weights, activations quantized on the fly).
• In llama.cpp (from the commit NVIDIA’s card points to): the official GGUF Q5_K_M.
• Speed: tokens per second generating 256 tokens at batch 1, and reading a 2048-token prompt, with an outside stopwatch around each whole call: the vLLM call and the HTTP request to llama.cpp. Median of 5 runs after one warm-up.
• Quality: the same 250 GSM8K questions, picked with a fixed seed and committed before running. Reasoning on, deterministic decoding and up to 4096 tokens per answer. An answer that does not finish within that limit counts as wrong.
• Rule: a format «ties» with BF16 if the 95 % confidence interval of the accuracy difference fits entirely within ±3 points, and «loses» if it falls entirely below −3. Otherwise, «inconclusive».
• Extension: since 250 questions were not enough, I registered a third amendment before running it: the full GSM8K with an 8192-token cap and a token-by-token comparison with BF16 on fixed texts in English and Spanish.

What broke before measuring

Neither engine ships Dory yet, and building them on a NixOS node had stumbles worth telling.

• The llama.cpp converter does not accept the published checkpoint. The branch NVIDIA points to demands keys from an internal format (dory_format_version, ngpt, hybrid_layer_pattern) that the Hugging Face config.json does not carry. I did not make them up. The PR says it tested the bf16 format, along with q4_k_m and q5_k_m, and the converter demands keys the public config.json lacks: there is an incompatibility between that revision of the converter and the published checkpoint. So in llama.cpp I only measured the official Q5_K_M: I have no F16, Q8_0 or Q4_K_M of my own, and the GGUF is compared against vLLM’s BF16, with the engine changing at the same time.
• vLLM would not load the configuration. The image ships transformers 5.17, which, while standardizing the RoPE parameters, also walks Dory’s mlp layers and fails with KeyError: 'mlp'. With transformers 5.20.1 it loads fine.
• Three mistakes of mine in the harness, fixed before publishing. With prefix caching on, the warm-up run left the prompt cached and the «reading» came out at between 78,000 and 135,000 tokens per second: I repeated the speed runs with caching off. And when an answer was cut at 4096 tokens, vLLM returned the half-finished reasoning and my extractor took its last number as the answer. That counted up to 40 lucky hits per variant, while llama.cpp returned an empty answer. Now both branches count those cases as wrong, as the rule said from the start. The external audit found the third: in vLLM I timed the whole call and in llama.cpp I used the server’s internal rate, which leaves out the prompt reading. I repeated llama.cpp with an outside stopwatch around each request, as in vLLM: the GGUF generates at 58.8 tokens per second, not 62.2, and reads at 1,710.

Speed: the same model, five formats

Generating 256 tokens (batch 1)

1. NVFP4vLLM116.8 tok/s · 2.19 s
2. MXFP8vLLM81.4 tok/s · 3.14 s
3. FP8vLLM78.5 tok/s · 3.26 s
4. GGUF Q5_K_Mllama.cpp58.8 tok/s · 4.35 s
5. BF16vLLM51.0 tok/s · 5.02 s

Reading a 2048-token prompt

1. FP8vLLM10,118 tok/s · 0.20 s
2. BF16vLLM7,003 tok/s · 0.29 s
3. NVFP4vLLM6,978 tok/s · 0.29 s
4. MXFP8vLLM6,061 tok/s · 0.34 s
5. GGUF Q5_K_Mllama.cpp1,710 tok/s · 1.20 s

NVFP4 3.28 · MXFP8 4.76 · FP8 4.67 · GGUF Q5_K_M 3.42 · BF16 7.69 GiB of weights as loaded by vLLM (GGUF: file size)

Generating one token at a time, the order of the vLLM formats follows the size of the weights almost exactly. At batch 1, every new token forces the card to read all the weights from its memory, and the result is consistent with fewer bytes giving more speed. NVFP4 takes 2.3 times less than BF16 (3.28 against 7.69 GiB) and generated 2.3 times faster. FP8 and MXFP8 take 1.6 times less and generated 1.5 and 1.6 times faster.

Reading a long prompt, the order changes. There the card does arithmetic instead of waiting on memory, and what matters is the precision it computes in.

Ada has FP8 and INT8 tensor cores; it has no FP4 and no MX block scaling, so NVFP4 and MXFP8 are expanded on the fly. GGUF row: llama.cpp has an INT8 path (Q8_1 activations) and an FP16 one; which one ran was not recorded (n/r). bits: weights / activations.

Of the quantized formats I ran in vLLM, FP8 is the only one whose multiplications Ada does in 8 bits. vLLM runs it with weights and activations in 8 bits, and it read the prompt at 10,118 tokens per second, against 7,003 for BF16. NVFP4 and MXFP8 go through Marlin kernels, which keep the weights small and expand them to 16 bits inside the kernel: the arithmetic is the same as in BF16. That is consistent with NVFP4 reading at 6,978 and MXFP8 at 6,061, without beating BF16. It is a kernel of the same Marlin family I found in TensorRT-LLM’s code in the previous post, which never got to run there.

llama.cpp with the Q5_K_M generated at 58.8 tokens per second and read at 1,710, with weights of a size similar to NVFP4. That comparison does not let me separate the engine from the format. Its CUDA backend has a path that quantizes the activations to 8 bits and multiplies the k-quants with integer dot products, and another that expands them to 16 bits; I did not record which one each operation used. It also ran with flash attention off, as in the commands of NVIDIA’s card, with its own tokenizer and with Dory freshly added in a branch. Any of those could explain the gap.

Quality: did any of them get dumber?

I measured it three ways. The first, the one I set at the start, came out inconclusive. I registered the other two in an amendment to the sheet before running them, to get an answer with more precision.

First pass: 250 questions

The four comparisons were inconclusive. BF16 got 202 of 250 right, FP8 199, NVFP4 208, MXFP8 210 and the GGUF 212. For NVFP4, MXFP8 and the GGUF, the confidence interval rules out the format losing more than 3 points. For FP8 it does not: its interval runs from −6.0 to +3.6. With 250 questions you can detect differences of about 4 or 5 points, not 1.

What weighed most on that score was something else: between 37 and 50 answers per variant did not finish reasoning within 4096 tokens. Among the ones that did finish, every format got almost everything right: BF16 202 of 204, FP8 199 of 200, MXFP8 210 of 211, NVFP4 208 of 210 and the GGUF 212 of 213.

Full GSM8K: 1319 questions

I ran all 1319 questions of the test set with the cap raised to 8192 tokens, on the four vLLM formats. The GGUF was left out: it runs on another engine and has no reference of its own.

With the full set, the three formats tie with BF16 under the same rule. BF16 got 1249 right, MXFP8 1250, FP8 1241 and NVFP4 1238. NVFP4’s difference is −0.8 points, with an interval from −1.8 to +0.2; FP8’s, −0.6 (from −1.5 to +0.3); MXFP8’s, +0.1 (from −0.8 to +0.9). All three intervals fit entirely within ±3 points. With the new cap between 32 and 44 answers of 1319 were cut, so truncation stopped dominating the score: it is half of the errors, against almost all of them in the first pass. What separates FP8 from BF16 is nine more cut answers, not wrong ones: FP8 reasons longer (median of 2386 tokens against 1995).

Token by token

GSM8K measures whether the final answer is right. The way a quantization is compared with its original is finer: give it the same text and look, at each position, at how much the distribution of the next token changes. I did it over 32,752 positions per language. In English, 16 stretches spread along the whole wikitext-2 test text; in Spanish, the opening of ten Wikipedia articles, about 3,300 tokens of each, with their revision pinned. The 2048-token chunks the model reads do not match those stretches: several cross from one stretch into the next, the same for all four formats.

same next token as BF16

MXFP8

95.4 %95.9 %

FP8

94.1 %94.6 %

NVFP4

89.3 %90.3 %

80 % → 100 %

KL divergence from BF16 (estimated over its top 20)

MXFP8

0.0100.005

FP8

0.0170.011

NVFP4

0.0510.036

0 → 0.051

Estimate: sums BF16's 20 most likely tokens (about 90 % of its probability); a token missing from the format's top 20 gets the format's 20th probability. Comparable between formats, not a bound on the full KL.

■ English · ■ Spanish

perplexity of the text: BF16 reference 10.01 / 8.33 · MXFP8 10.08 / 8.37 · FP8 10.08 / 8.39 · NVFP4 10.00 / 8.42

Perplexity barely moves: 10.01 in English and 8.33 in Spanish for BF16, and between 10.00 and 10.08, and between 8.37 and 8.42, for the others. In English, NVFP4 even lands slightly below BF16. Looking only at that number, the formats would be the same.

Token by token they are not. NVFP4 picks the same next token as BF16 at 89.3 % of positions in English and 90.3 % in Spanish: it changes the choice roughly one time in ten. FP8 agrees at 94.1 and 94.6 %, and MXFP8 at 95.4 and 95.9 %. My estimate of NVFP4’s KL divergence is about 3 times FP8’s, and 5 times (English) and 7 times (Spanish) MXFP8’s. That was the hypothesis I wrote down before running: NVFP4 drifts more. Even so, on the full GSM8K that distance did not turn into a drop that can be measured. MXFP8 stayed closer to BF16 than FP8, which is consistent with one scale per block of 32 weights fitting better than a single scale per matrix.

For the GGUF I measured perplexity with llama-perplexity: 8.20 in English and 7.92 in Spanish. That number is not comparable with vLLM’s, because llama.cpp uses its own tokenizer and only scores the second half of each chunk.

Threats to validity

• It is a single 4B model with an uncommon architecture. The speed order may not repeat on a large model, where reading the prompt weighs more.
• Both engines run from unmerged branches. The numbers may change when they reach a published release.
• The NVFP4 card lists Hopper and Blackwell as supported hardware, not Ada: on this card it runs on a path NVIDIA does not declare.
• The GGUF is compared on another engine. I could not measure a Q4_K_M or an F16 reference in llama.cpp, and llama.cpp ran with flash attention off, 8 parallel requests for GSM8K (vLLM processed them in batches of up to 64) and its own tokenizer.
• In all four variants vLLM warned that the tokenizer loads with an incorrect regex pattern. It affects the four equally, but not the GGUF.
• FP8 is not an NVIDIA file: it is the per-tensor quantization vLLM does while loading the BF16 weights.
• I deviated from the sheet in two things: instead of peak VRAM with nvidia-smi I publish the weight memory vLLM reports on load (vLLM reserves the card up front), and the question-by-question agreement with BF16 stays in the data (215, 222, 218 and 224 of 250 for FP8, MXFP8, NVFP4 and the GGUF).
• GSM8K measures a single skill, in English. In the first pass the 4096-token cap cut between 37 and 50 of the 250 answers; with 8192 it cut between 32 and 44 of 1319.
• The KL is an estimate: it only sums BF16’s 20 most likely options, which leave out about 10 % of its probability, and when one of them is not among the variant’s 20 I use the variant’s 20th probability, because I only kept each format’s 20 most likely options. It is not a bound on the full KL; it serves to compare the formats with each other, and next-token agreement, which does not depend on that approximation, gives the same order. The token-by-token comparison does not include the GGUF.
• The first token-by-token run, through a mistake of mine, measured only the start of each file: in Spanish, only the «Chile» article, and one of the ten revisions was an empty redirect. The audit caught it; I registered a fourth amendment with the texts spread out and measured again. The figures here are from that second run; the first gave the same order in next-token agreement and in the estimated KL, but not in perplexity, which changes order between the formats from one run to the other. It stays in the repository.
• I decided on the full GSM8K and the token-by-token comparison after seeing the first pass was inconclusive. I wrote the rules down before running them, but they were not in the original plan.
• I measured at batch 1. With many requests at once, prompt reading and compute gain weight, and I expect FP8 to gain ground, but I did not measure it.
• The memory I publish is what vLLM reports when loading the weights (and the file size for the GGUF), not the total use during inference, which includes the context cache.

Verdict

On a 16 GB Ada card, with this model, NVFP4 is the format for chatting: it generates 2.3 times faster than BF16, its weights take less than half when loaded and it tied with BF16 on all 1319 GSM8K questions. Token by token it drifts more than the 8-bit formats; if what matters is reproducing BF16 as faithfully as possible, MXFP8 is the closest. FP8 is the format for reading long documents, because, of the formats I ran in vLLM, it is the only one that does the arithmetic in 8 bits on Ada’s hardware; for serving many requests at once probably too, but I did not measure that. MXFP8 generated slightly faster than FP8 and read much slower, so on this card its advantage is fidelity, not speed. And the GGUF was slower than vLLM’s three quantized formats, although that comparison does not separate the engine from the format.

Frequently asked questions

NVFP4 vs FP8: which is faster on a GPU that is not Blackwell?

It depends on what you measure. Generating one token at a time, NVFP4 was the fastest: 116.8 tokens per second against 78.5 for FP8 and 51.0 for BF16, on an RTX 4070 Ti SUPER with vLLM. Reading a 2048-token prompt, FP8 won with 10,118 tokens per second against 6,978 for NVFP4, because Ada has FP8 tensor cores and vLLM uses them with weights and activations in 8 bits. On Ada, NVFP4 only shrinks the weights.

Does NVFP4 lose quality against BF16?

On GSM8K, within the margin I set, no. On all 1319 questions BF16 got 1249 right and NVFP4 1238: −0.8 points, with a 95 % confidence interval from −1.8 to +0.2, inside the ±3 margin I set before measuring. Token by token it does drift more than the 8-bit formats: it picks the same next token as BF16 at 89-90 % of positions, against 94-96 % for FP8 and MXFP8, with practically the same perplexity.

MXFP8 vs FP8: is there a difference?

In generation speed, barely: 81.4 against 78.5 tokens per second. Reading a long prompt, FP8 was much faster (10,118 against 6,061 tokens per second), because on Ada vLLM runs FP8 on the FP8 tensor cores and MXFP8 with a kernel that only shrinks the weights. In quality, MXFP8 stayed closer to BF16: it agrees on the next token 95-96 % of the time against 94-95 % for FP8, and on all 1319 GSM8K questions both tie with BF16 (1250 and 1241 against 1249).

NVFP4 vs GGUF: which one should I use?

With this model, on my card, NVFP4 in vLLM generated at 116.8 tokens per second and NVIDIA's official GGUF Q5_K_M in llama.cpp at 58.8, with weights of similar size (3.28 against 3.42 GiB). The GGUF got 212 of 250 on GSM8K and NVFP4 208. Note: the engine changes along with the format, and I could not make my own Q4_K_M or F16 because the llama.cpp converter does not accept the published checkpoint.

How much memory do the weights take in each format?

As vLLM reports on load: 7.69 GiB in BF16, 4.67 in FP8, 4.76 in MXFP8 and 3.28 in NVFP4. The GGUF Q5_K_M takes 3.42 GiB on disk. Generation speed grew almost in the same proportion as the weights shrank.

Can I reproduce it?

Yes. The sheet with the rules set before measuring, the Dockerfiles of both engines, the benchmark and scoring scripts, the questions (250 and 1319), every answer, the fixed texts and the token-by-token probabilities are in the repository and in a gist linked in the sources.

Sources

• NVIDIA’s model cards: Dory 1.0 4B BF16, NVFP4, MXFP8 and GGUF, with the vLLM and llama.cpp commands and the two-loop recommendation.
• The Dory PR in vLLM and the one in llama.cpp.
• Introducing NVFP4, from NVIDIA, and the OCP Microscaling specification, which defines MXFP8.
• GSM8K, the grade-school math problem set.
• wikitext-2 and the Spanish Wikipedia articles (CC BY-SA 4.0) used as fixed texts; the exact revisions are in the gist.
• The sheet, the Dockerfiles, the scripts, the questions, every answer and the timings.

Measured on October 17, 2026 on a 16 GB RTX 4070 Ti SUPER (Ada, sm_89, driver 595.71.05). vLLM 0.32.0rc1.dev319+g55076950b from PR #60149 with transformers 5.20.1; llama.cpp at commit 98a74c9 built for sm_89 with CUDA 12.8. Dory 1.0 4B with two loops and per-loop cache. GSM8K: 250 questions from the test split (seed 17) with a 4096-token cap, and all 1319 with an 8192 cap. Fidelity: 16 chunks of 2048 tokens per language (16 stretches spread over the wikitext-2-raw-v1 test, and about 3,300 tokens from the opening of each of ten es.wikipedia articles).