Reels · 1 of 61
Reel · 54 s
NVIDIA's draft models for Qwen3.8-27B on 16 GB: 43 tokens per second against 88 for MTP
In 54 seconds and narrated: NVIDIA's draft models for Qwen3.8-27B against MTP on a 16 GB GPU. Half a second of verification steps with greedy decoding, slowed down 16 times: no draft 38.3 tokens per second, NVIDIA's 43.5 and MTP 88.0. Each step of NVIDIA's draft costs 59.2 ms and leaves 2.58 tokens; MTP's, 34.7 ms and 3.05. The 4B does not fit, and with the model shrunk so it does, it runs slower than no draft. Muted by default: turn the sound on in the controls.
- Length
- 54 s
- Published
This reel sums up NVIDIA's draft models for Qwen3.8-27B on 16 GB: 43 tokens per second against 88 for MTP, where the method, the tables and what did not work are.