23°

Reels · 1 of 61

54 s

Reel · 54 s

NVIDIA's draft models for Qwen3.8-27B on 16 GB: 43 tokens per second against 88 for MTP

In 54 seconds and narrated: NVIDIA's draft models for Qwen3.8-27B against MTP on a 16 GB GPU. Half a second of verification steps with greedy decoding, slowed down 16 times: no draft 38.3 tokens per second, NVIDIA's 43.5 and MTP 88.0. Each step of NVIDIA's draft costs 59.2 ms and leaves 2.58 tokens; MTP's, 34.7 ms and 3.05. The 4B does not fit, and with the model shrunk so it does, it runs slower than no draft. Muted by default: turn the sound on in the controls.

Length
54 s
Published

llama.cppDecodificación especulativaQwenGPUModelos de lenguaje

This reel sums up NVIDIA's draft models for Qwen3.8-27B on 16 GB: 43 tokens per second against 88 for MTP, where the method, the tables and what did not work are.

Read the articleOpen in the viewer