More bits where it hurts: I quantized a Qwen3.8-27B to fit my 4070 Ti
Q4 overflows 16 GB and Q3 gives up quality. A llama.cpp patch spreads the bits layer by layer: I measured a Qwen3.8-27B at 94% agreement with an 8-bit reference that does fit on a 4070 Ti.
I wanted to run the best Qwen3.8-27B I could on my RTX 4070 Ti, which has 16 GB. It’s a shoe-size problem: the Q4_K_M tier is a 15.66 GB file, and once you load it with context it overflows 16 GB and errors out of memory. The tier below, Q3_K_M, fits with room to spare but gives up quality. There was no middle ground.
A llama.cpp patch supplies one: instead of a fixed tier, a continuous target. At 15 GB the model lands at 94.3% agreement with the 8-bit reference and loads where fixed Q4 won’t even start.
What quantizing is, plainly
A model is millions of numbers, its weights. Storing them in 16 bits is faithful but heavy. Quantizing stores them with fewer bits (8, 5, 4, 3) so the file fits on the GPU. It’s the same idea as taking a photo from RAW to JPG: it takes far less space and looks almost the same, but compress too hard and the blocks show up.
Until now that compression came in fixed sizes. Q4_K_M means, roughly, “four and a half bits per weight, for the whole model.” You pick a size and pray it fits your GPU.
Each step is a value it can store. The axis is the same; what drops is how many steps fit. Steps are illustrative, not to scale.
The dial: ask for a size, not a tier
The patch adds two options to llama-quantize: --target-bpw (bits per weight) and --target-size (file size). You tell it “I want this to weigh 15 GB” and an algorithm decides, tensor by tensor, which quantization type to use to meet that budget while losing the least quality possible.
The difference from a fixed tier is clearest when you look at what it actually decided. I asked the Qwen3.8-27B for 15 GB and inspected the split it made across its 503 tensors:
On a lower average budget than the comparable fixed tier (4.77 vs 4.98 bits) and better spread: the top rows get more than the bottom ones. Dark cells are blocks where that tensor does not exist (the model alternates attention and ssm).
A fixed tier would set the same height on every bar. The dial pushed ffn_down and the attention value projection to 5.5 bits — the ones that hurt most under compression — kept the queries and keys near 5.1, and sank the rest (ffn_gate, ffn_up, ssm) to 4.25. And it does this on less average budget than the comparable fixed tier: 4.77 bits against 4.98, better spread.
For the algorithm to know which tensor “hurts” more, it needs an importance matrix (imatrix): a measurement of which activations carry more weight, computed by running text through the model. I calibrated it on a diverse corpus, not just the text I later measure on, so I don’t cheat myself.
Installation, with the stumbles
The patch is EAddario’s PR #15550, still not merged into the main branch. I built it from commit 325319b. The fact that it isn’t merged matters: nobody reproduces this from llama.cpp main, and I hit at least one write bug along the way.
Three things cost me time:
- The importance matrix didn’t fit in RAM. Generating it requires running the model, and the original in 16 bits is 54 GB, more than I have across RAM and VRAM combined. I calibrated it from the
Q8_0version (8 bits, 28 GB), which for measuring importance is nearly identical to the original. The same holds for quantizing: every variant comes out of the sameQ8_0, so the comparison among them is clean. - An
iostream errorthat wasn’t the patch’s. Mid-quantization, the process died with “input/output stream error.” I chased a ghost in the MTP layers for a while. It was the disk: it had filled to 100%. The logits file the divergence comparison generates was taking hundreds of GB. Capping it fixed it. - File size is not VRAM. A 15.66 GB GGUF does not run in 16 GB. Between the weights, the context cache and the compute buffers, it overflows. You have to measure real VRAM, not read the file size.
What I measured
Perplexity on wikitext-2 and on code (outside the calibration domain), divergence against Q8_0 as the reference, real VRAM with an 8k context, and speed. All on the same 4070 Ti, all with the same imatrix.
| variant | file | PPL wiki | PPL code | top-token agreement vs Q8 | fits in 16 GB? | tok/s |
|---|---|---|---|---|---|---|
| Q3_K_M (fixed) | 12.57 GB | 7.186 | 1.949 | 90.4% | yes | 37.8 |
| dial ≈ Q3 | 12.00 GB | 7.131 | 1.945 | 90.6% | yes | 42.6 |
| dial · 15 GB | 15.00 GB | 7.037 | 1.908 | 94.3% | yes | 36.2 |
| Q4_K_M (fixed) | 15.66 GB | 6.953 | 1.906 | — | won’t load | — |
| Q5_K_M (fixed) | 18.19 GB | 6.962 | 1.897 | — | won’t load | — |
Two readings. At the same size as Q3_K_M, the dial drops perplexity (7.131 against 7.186) and runs faster on top of it (42.6 against 37.8 tokens per second). The margin is small and brushes measurement error, but the divergence confirms it: the dial stays closer to the original.
Q5 measures a hair worse than Q4 on wikitext (6.962 against 6.953): the difference falls inside the error, and neither one loads in 16 GB anyway.
The reading that matters is the other one.
The 16 GB budget
- dial · 12 GB: fits, and leaves room for MTP
- dial · 15 GB: most faithful one that fits
- Q4_K_M (fixed): out of memory: won’t load
The only fixed tier that loaded with a useful context was Q3_K_M. The Q4_K_M, better in quality, overflows. The dial at 15 GB fits with 683 MiB to spare and agrees on 94.3% of tokens with the 8-bit reference, against 90.4% for Q3; its mean error is 3.63% against 6.63%, nearly half. This is what a fixed tier can’t give you: the quality that fits exactly in your GPU, not one size more and not one less. It pays 1.6 tok/s for the extra bits (36.2 against Q3’s 37.8): cheap for almost four points of agreement.
That reference Q8_0 is how you read the table: 94.3% agreement means that, out of every 100 tokens, the 15 GB model picks the same one as the 8-bit version in 94, against 90 for Q3.
The MTP doesn’t fit, and that’s a finding
The Qwen3.8-27B ships with MTP, an internal draft that proposes several tokens at once to speed things up. In another measurement it took me from 42 to 80 tokens per second.
The draft proposes, the big model verifies. Almost twice as fast (42 → 80 tok/s) and the text that comes out is identical.
I wanted to stack it on top of the 15 GB model.
When it’s worth it and when it isn’t
The dial wins when the goal is to fill an exact budget: a GPU where the fixed tier above overflows by a little. There it adds quality the tier can’t reach and loads where the tier spills over.
I wouldn’t use it if you have memory to spare. If Q4 loads with room, you already have your model, and with headroom for MTP too, which buys more in speed than the couple tenths of perplexity the dial scratches out. Nor if you need strict reproducibility today: the patch isn’t in llama.cpp main.
For my case (27B in 16 GB, no headroom) it’s the difference between a mediocre model that fits and a good one that also fits.
Sources
- PR #15550,
--target-bpwin llama.cpp, by EAddario. Built from commit325319b. - Qwen3.8-27B, Apache-2.0 license.
- Perplexity and divergence with
llama-perplexityon wikitext-2; the GGUFs and the imatrix came from the model’sQ8_0version.
Comments
No comments yet. The first one is yours.