23°
Article cover: NVIDIA's draft models for Qwen3.8-27B on 16 GB: 43 tokens per second against 88 for MTP
llama.cppSpeculative decodingQwenGPULanguage models

NVIDIA's draft models for Qwen3.8-27B on 16 GB: 43 tokens per second against 88 for MTP

NVIDIA released three draft models to speed up Qwen3.8-27B. I converted them to GGUF and measured them on an RTX 4070 Ti SUPER with the same tokens as my previous post: on llama.cpp they speed things up by 14 % at most in aggregate, with sampling they only help on code and agents, they perform like an untrained Qwen3.5-0.8B, and the 4B one does not fit next to the model. MTP reaches 88 tokens per second.

Efrain Garay 19 November 2026

Playing summary

In September I measured ExLlamaV3 against llama.cpp with Qwen3.8-27B on my 16 GB RTX 4070 Ti SUPER. Speculative decoding with MTP more than doubled the speed in both engines. On November 17 NVIDIA released three draft models trained for that same Qwen3.8-27B. According to their table, the 4B one leaves 5.58 tokens per verification step against 3.81 for the model’s own MTP head; the other two land close to MTP or below it.

I converted them to GGUF and ran them through the same battery as in September, with the same input tokens. On llama.cpp, the best NVIDIA draft that fits reached 43.5 tokens per second with greedy decoding, against 38.3 without a draft and 88.0 with MTP. With sampling, the policy they were trained for, it helped on code and agents and slowed down creative writing and translation; in aggregate it matched no draft. An untrained Qwen3.5-0.8B performed like NVIDIA’s, and the 4B one did not fit; shrinking the model so it would, it ran slower than no draft.

In 54 seconds and narrated: NVIDIA's draft models for Qwen3.8-27B against MTP on a 16 GB GPU. Half a second of verification steps with greedy decoding, slowed down 16 times: no draft 38.3 tokens per second, NVIDIA's 43.5 and MTP 88.0. Each step of NVIDIA's draft costs 59.2 ms and leaves 2.58 tokens; MTP's, 34.7 ms and 3.05. The 4B does not fit, and with the model shrunk so it does, it runs slower than no draft. Muted by default: turn the sound on in the controls.Watch it in the reel viewer →

What NVIDIA released

Three full models, not heads: Qwen3.5-0.8B, Qwen3.5-4B and Llama-3.2-1B with Qwen’s vocabulary. They were trained with Model Optimizer to imitate the output distribution of Qwen3.8-27B. The draft proposes a block of tokens and the big model checks it in one pass; in theory, what comes out is what the big model would have written on its own.

Their model card says they are meant for the Groq 3 LPU and also load in vLLM, SGLang and TRT-LLM. With 16-token blocks and sampling at temperature 1, averaged over 11 categories, the 4B one leaves 5.58 tokens per step, the 0.8B one 3.92, the Llama-1B 3.42 and the MTP head 3.81. I did not reproduce those figures: they are NVIDIA’s, on their hardware, and they say nothing about how long each step takes.

My question is a different one: what happens on llama.cpp, on a 16 GB card, against the MTP I already use.

How I measured

  • Big model: the same Q3_K_M as in September, 13.5 GB (12.57 GiB), requantized from Unsloth’s Q8_0 with the importance matrix from the custom quantization post.
  • Engine: llama.cpp from November 18 (commit 98d67c8, already in the 1.0 series), built with CUDA 12.9.
  • Drafts: NVIDIA’s three, converted with convert_hf_to_gguf.py, the 1.37 GB MTP draft that Unsloth publishes, and as a baseline an original Qwen3.5-0.8B, with no training for this at all. If NVIDIA’s draft does not beat that one, its training does not show here.
  • Prompts: the 15 prompts from the ExLlamaV3 battery that fit in 16k tokens of context (code, agents with tools, translation, creative writing), the same token ids as in September, up to 1,024 new tokens per prompt.
  • Two modes: greedy, as in September, and sampling with the policy NVIDIA states (temperature 1, top-k 20, top-p 0.95, probabilistic acceptance of the draft).

Speed is generated tokens over generation time, summed over the 15 prompts. That sum weighs by response length: the two agentic prompts contribute fewer than 150 tokens and the creative and translation ones several thousand. That is why I also give per-category figures.

The GPU is shared with ComfyUI and the desktop (525 MiB between them) and a service I stopped while measuring. A first full battery came out contaminated by another job that used the GPU midway, and I threw it away. The final one waits until the GPU has no foreign processes, watches it while measuring and repeats any configuration where one shows up. There is one run per configuration and, with sampling, a single seed. As a noise check I have the greedy half of the discarded battery: in the four configurations that ran before the interference it matched the final one within 1.2 % at most, on the same text.

That last detail matters. With greedy decoding, llama.cpp does not reproduce bit for bit when it verifies in a batch: 3 of the 15 prompts, all creative writing, produce responses of different length depending on the draft, MTP included. The speeds for those three compare slightly different texts; the other 12 are identical across configurations.

I wrote the plan with the decision rules after the main battery and before the shrunk-model experiment; it is in the repository with its date.

What went wrong

  • The cache did not fit like in September. With speculative decoding, llama.cpp keeps several copies of the model’s recurrent state, and MTP with an f16 cache and 16k tokens did not load. I switched it to q8_0, the same one September used for its long-context tests, and it fit with a 15,555 MiB peak.
  • The 4B draft does not fit. In Q4_K_M it weighs 2.74 GB and did not load. I took it down to Q3_K_S (2.07 GB) and Q2_K (1.97 GB): neither; in Q2_K it fails while reserving the 272 MiB of its own cache. That is with 525 MiB already taken, 16k tokens and a q8_0 cache: on a card without a desktop, or with less context, it might fit.
  • Neither does the Llama-1B in Q8_0. It weighs 1.59 GB because of Qwen’s 248k-token vocabulary. I measured it in Q4_K_M, 1.01 GB.

Speed: MTP doubles it, the drafts barely help

Tokens per second, Qwen3.8-27B Q3_K_M, 15 promptstokens per second, decoding, sampling at temperature 1 · higher is better
  1. MTP · window 383.9
  2. No draft38.3
  3. NVIDIA 0.8B · window 637.7
  4. Untrained Qwen3.5-0.8B · window 637.3
  5. NVIDIA Llama-1B · window 336.3
  6. NVIDIA 0.8B · window 1234.9

RTX 4070 Ti SUPER 16 GB, llama.cpp 98d67c8, 16k context with a q8_0 cache. Same token ids as the September battery. Generated tokens over generation time, summed over the 15 prompts. One run per configuration, no foreign processes on the GPU.

With greedy decoding the drafts do a little better, because they guess right more often when the big model does not sample. The window is the most the draft may propose; llama.cpp does not always fill it. With windows of 3, 6 and 12, the 0.8B one proposed 2.0, 3.5 and 6.5 tokens per step on average, and MTP 3.0 with a window of 3. I did not look into where that cut comes from.

ConfigurationGreedy (tok/s)Sampling (tok/s)Greedy acceptanceSampling acceptance
No draft38.338.3
MTP, window 388.083.968.8 %64.1 %
NVIDIA 0.8B, window 341.235.963.9 %50.8 %
NVIDIA 0.8B, window 643.537.745.1 %35.6 %
NVIDIA 0.8B, window 1242.234.928.8 %21.3 %
Untrained Qwen3.5-0.8B, window 341.536.263.5 %51.3 %
Untrained Qwen3.5-0.8B, window 646.137.348.0 %34.9 %
NVIDIA Llama-1B, window 340.636.356.5 %46.1 %
NVIDIA Llama-1B, window 641.234.637.9 %27.8 %

The best draft result came from the untrained Qwen3.5-0.8B, at 46.1 tokens per second with greedy decoding: 20 % above no draft. NVIDIA’s draft of the same size accepted practically the same: 63.9 % against 63.5 % with a window of 3. With one run per configuration I do not claim one is better than the other. I claim that, on llama.cpp, the training did not produce a difference that shows up in this test.

Before writing that I checked it was not the conversion. I had Qwen3.8-27B answer the same prompts and measured how hard it is for each small model to predict that text (perplexity, lower is better). As GGUF, NVIDIA’s draft scored 5.06 and the original 3.88. With the original weights in transformers, without GGUF, 5.81 and 4.62. The conversion does not change the order: NVIDIA’s draft predicts this 27B text worse than the original, read without its prompt and written with greedy decoding. NVIDIA trained it on temperature-1 responses with their prompt in front, so this test does not disqualify it; it only rules out that the tie comes from a broken GGUF.

By category, with sampling, NVIDIA’s draft does help in some. On code it goes from 39.2 to 48.2 tokens per second with a window of 6. On the two agentic prompts, from 38.0 to 57.2 and from 37.2 to 59.7, although each leaves only 51 and 75 tokens, so those two figures are noisy. On trivial repetition, a single prompt, it reaches 146.7 with a window of 12, more than MTP’s 118.9. On creative writing and translation, though, every external draft stayed below no draft, and since those are the longest responses, they dominate the aggregate.

Why: the cost of each step

Speculative decoding moves in steps. In each one, the draft proposes some tokens and the big model checks them in one pass; out comes what was accepted plus one token from the big model. The final speed is how many tokens each step leaves divided by how long it takes.

Two ways to draft for Qwen3.8-27BLeft: the MTP head lives inside the big model and reuses its state; each verification step costs 34.7 ms with 3 proposed tokens. Right: NVIDIA’s external 0.8B draft is a separate model with its own cache; it generates at about 2.3 ms per token on its own, yet each step costs 55.8 ms with 2 proposed tokens.MTPExternal draftQwen3.8-27BMTP head · 1.37 GBreuses the 27B stateverifies in one pass3.0 proposed34.7 ms per stepNVIDIA 0.8B · 0.81 GBown cache · 2.3 ms/tokenQwen3.8-27Bverifies in one pass2.0 proposed55.8 ms per step
Greedy, window 3, average over 15 prompts. The draft alone explains about 5 ms of the 21 ms gap; the rest stays in llama.cpp’s external-draft path.
Half a second of verification steps, slowed down twelve times for displayEach block is one step: the draft proposes and the big model checks in one pass. Qwen3.8-27B Q3_K_M, sampling at temperature 1, averaged over the 15 prompts.
MTP · window 334.8 ms per round · 2.92 tokens per round
14 rounds · 41 tokens · 83.9 tok/s
NVIDIA 0.8B · window 658.0 ms per round · 2.19 tokens per round
8 rounds · 18 tokens · 37.7 tok/s
No draft26.1 ms per round · 1.00 tokens per round
19 rounds · 19 tokens · 38.3 tok/s

MTP leaves more tokens per step, and each step also costs it 23 ms less than NVIDIA's draft. Steps are derived from the counters: generated tokens minus accepted tokens, because each step leaves what was accepted plus one.

Without a draft, each token costs 26.1 ms. MTP spends 34.7 ms per step with greedy decoding and gets 3.05 tokens: plenty of margin. NVIDIA’s 0.8B draft spends 55.8 to 67.9 ms per step and gets 2.30 to 2.87 tokens. That barely beats no draft: by 8 to 14 %.

Where does the gap come from? The cleanest comparison is NVIDIA’s draft with a window of 3 against MTP: the draft proposes 2.0 tokens per step on average and its step costs 55.8 ms; MTP proposes 3.0 and its step costs 34.7. That is 21 ms more while proposing less. My first explanation was that the draft writes its proposal one token at a time before every check. I measured it and it does not add up: the 0.8B, alone on the GPU, generates 454 tokens per second without context and 420 with 8k tokens of context, 2.2 to 2.4 ms per token. Two tokens are about 5 ms. About 16 ms remain in llama.cpp’s external-draft path: two contexts kept in sync and whatever it takes to undo rejected tokens in a model with recurrent state. I did not isolate it. The MTP head is one extra layer that reuses the big model’s state, and its step costs 8.6 ms more than a token without a draft.

With a window of 12 the draft proposes more, but with sampling it gets 21 % right: it spends more time and most of it is thrown away.

There are parts I cannot separate. NVIDIA measured on vLLM on an LPU, where running the draft may cost much less than on a consumer card with llama.cpp, and their card lists Blackwell as supported hardware; mine is Ada. Their tokens-per-step table says nothing about the time of each step, and on this card time is what decides. Also, NVIDIA trained the drafts on sequences of up to 12,288 tokens and prompts under 3,072; six of my prompts have 9k to 13.7k tokens, and three exceed 12,288.

Memory: MTP just fits

nvidia-smi · 4070 Ti SUPER · 16k · q8_0 cache · peak
4/ 4loaded next to Qwen3.8-27B Q3_K_M
No draft13.5 GBfits
+ MTP · 1.37 GB15.2 GBfits
+ NVIDIA 0.8B · Q8_014.5 GBfits
+ NVIDIA Llama-1B · Q4_K_M14.8 GBfits
  • No draft: baseline
  • + MTP · 1.37 GB: the fastest
  • + NVIDIA 0.8B · Q8_0: 0.81 GB
  • + NVIDIA Llama-1B · Q4_K_M: 1.01 GB

The peaks include the 525 MiB taken by ComfyUI and the desktop. The 4B one is missing because it never loaded.

What if I shrink the model so the 4B fits?

The 4B is the only draft to which NVIDIA’s table gives a clear edge over MTP, so I tried it the only way it fits: by shrinking the big model. With the custom quantization dial (llama.cpp PR #15550, same commit and same importance matrix as in September) I asked llama-quantize for a Qwen3.8-27B with --target-size 10.5g. It came out as a 10.74 GB file, 3.14 bits per weight, against the 13.5 GB of the Q3_K_M. Before running it I wrote down the rule: the 4B wins only if it beats MTP on the Q3_K_M by more than 5 % in both modes, and if the shrunk model stays within 2 points of agreement with the Q8_0.

With that model, the 4B in Q4_K_M loaded with a peak of 14,348 to 14,352 MiB.

On the 10.74 GB modelGreedy (tok/s)Sampling (tok/s)Tokens per step (greedy)ms per step (greedy)
No draft45.145.11.0022.2
MTP, window 385.483.32.9134.0
NVIDIA 4B, window 336.333.02.5470.1
NVIDIA 4B, window 638.131.63.1883.4
NVIDIA 4B, window 1234.2not measured3.66107.0

The 4B leaves more tokens per step than anything else I measured: 3.66 with a window of 12, more than MTP’s 2.91, which with a window of 3 cannot leave more than 4. And it is still the slowest, because each step costs it 107 ms. With every window it stays below the same model without a draft. MTP on this model reaches 85.4, almost the same as on the Q3_K_M, although its acceptance drops from 68.8 % to 63.8 %: the MTP draft is the same file, but the big model changed.

I measured quality with llama-perplexity and the KL divergence against the Q8_0, 512-token context, on the evaluation text from the dial post. These figures are not comparable with those of earlier posts, which used other tools or texts: the perplexity here is not on wikitext, which is why it is not the 7.19 I published back then, and the Q3_K_M divergence in the ExLlamaV3 post came from qbench.

  • Q3_K_M (13.5 GB): mean divergence 0.049, same most likely token as the Q8_0 at 90.7 % of positions, perplexity 5.85.
  • Custom 10.74 GB: mean divergence 0.108, same token at 85.5 %, perplexity 6.13.

The shrunk model loses 5.2 points of agreement with the reference. The rule says the 4B does not win: it is slower and forces a worse model.

One cell went unmeasured: the 4B with a window of 12 and sampling. In all four attempts another video job showed up on the GPU, and the script threw all four runs away.

My take

For Qwen3.8-27B on a 16 GB GPU with llama.cpp, MTP is the speculative decoding to use, by a wide margin: it doubles the speed in both modes, and it fits. On llama.cpp, NVIDIA’s drafts accept the same as an untrained small model from the same family. And on this engine and this card each step with an external draft costs too much, while the model already ships with a cheaper one. If you write code or use agents with sampling, the 0.8B helps a little; MTP helps twice as much.

What I cannot say is that they are useless where NVIDIA meant them to run. On a server with vLLM and hardware where the draft comes almost free, their tokens per step may turn into speed. I did not measure that.

When to use them and when not

  • Yes: if your engine does not support MTP for this model, or if you serve on the hardware NVIDIA trained them for. With sampling, the 0.8B sped up code and agents, and on very repetitive text, a single prompt, it beat MTP.
  • No: on a consumer GPU with llama.cpp and MTP available; with sampling on creative writing or translation, where every external draft fell below using none; and the 4B on my card: next to the 13.5 GB Q3_K_M it does not load, and shrinking the model to fit it makes it slower than no draft.

Sources

Measured from November 18 to 20, 2026 on a 16 GB RTX 4070 Ti SUPER with a Ryzen 7 7800X3D and NixOS, llama.cpp 98d67c8 with CUDA 12.9, Qwen3.8-27B Q3_K_M at 13.5 GB.

NEWSLETTER · MEASURED

A new measurement, when there is one.

Each post with its numbers, its failures and the scripts to repeat it. One to three emails a month. Unsubscribe with one click.

LATEST

I run Listmonk on my own server; your address stays there. You confirm by email before getting anything.

Comments

No comments yet. The first one is yours.

Reviewed before publishing. The email is not stored and never appears anywhere.