RTX 5080 16GB: 128k 컨텍스트에서 Qwen3.6 35B MoE 벤치마크 — 56 tok/s 및 MTP가 효과 없는 이유

The surprise: 35B is FASTER without MTP

MTP is 23% slower for the 35B MoE on 16GB. Why?

1. MTP requires --fit-target 1536 to reserve ~1.5 GB for the MTP compute buffer
2. That 1.5 GB pushes ~3 more MoE expert layers from GPU to CPU
3. CPU-bound expert layers are the bottleneck for MoE inference
4. MTP's multi-token speculation (~79% acceptance) doesn't compensate for the slower per-step speed

For the 27B, MTP helps because the model fits entirely on GPU (12.45 GB) — --fit-target 0 works with and without MTP, so there's no VRAM penalty. The 27B goes from ~56 tok/s (no MTP, older builds) to 73 tok/s with MTP.

Rule of thumb: MTP helps when your model fits on GPU. It hurts when the MTP compute buffer forces more layers to CPU.

Speed at Coding-Agent Context Lengths (the real test)

Everyone runs coding agents at 128k. Here's what actually happens as you fill the context window. Tested with synthetic prompts (Python classes, architecture docs, error stack traces — varied enough to prevent tokenizer compression), prompt cache disabled, 35B Q4_K_XL with --fit-target 1536:

8k/32k TG measured in a separate run from 64k/128k — expect ~5-10% variance between rows from measurement noise.

At 128k context, MTP and no-MTP converge to the same TG speed (~56 tok/s). The KV cache fills VRAM at long context regardless of MTP, so the offload split ends up identical. MTP's multi-token speculation is offset by its compute overhead.

PP degrades gracefully: 1,855 → 1,584 tok/s from 8k to 128k (~15% decline). A 128k prompt processes in ~81 seconds.

The "97 tok/s" only exists at short context with --fit-target 0. At 64k+, --fit-target 0 OOMs because there's no headroom for KV cache growth. You must use --fit-target 1536 for long-context work, which brings speed down to ~73 tok/s at short context and ~56 tok/s at 128k.

Bottom line for coding agents: expect ~56 tok/s TG and ~1,500 tok/s PP at 128k context on 16GB. MTP is a wash — doesn't help or hurt at full context.

VRAM Usage

Context Limits (push to OOM)

This is the biggest differentiator. The 35B MoE handles 131k context easily because its hybrid architecture (Gated DeltaNet + Attention) only has ~10 full-attention layers that need KV cache. The remaining SSM layers use a tiny recurrent state. The 27B dense model has KV on every layer, so it maxes out at 56k with q8_0 KV.

Tip for 27B users: switching from -ctk q8_0 -ctv q8_0 to -ctk q4_0 -ctv q4_0 extends your max context from 56k → 110k. Quality cost is minimal: q4_0 KV at 56k scores 218/220 CodeNeedle vs 220/220 with q8_0 KV (q4_0 at regular context: 219/220 — so most of the 2-line drop is from q4_0 itself, not the longer context).

The OOM at higher contexts is the MTP compute buffer (529 MiB fixed allocation), not the KV cache itself. This is a llama.cpp implementation detail that may improve in future versions.

Quality — CodeNeedle (positional recall)

11 functions from Python's http.server, ~50k char corpus, testing exact line-level recall:

The 27B IQ3 has a perfect score — every line exact, zero hallucinations. The 35B models are close but not quite there. Interesting that Q8_0 doesn't beat Q4_K_XL here.

Quality — GSM8K (grade school math, 100 cases)

All three overlap in confidence intervals — the quality difference is negligible. But the 35B Q4_K_XL is 37% faster to evaluate (67 vs 106 min) with fewer truncations.

Note: AIME2025 was also tested on the 27B — 50% overall but 100% on non-truncated cases*. Every failure was context exhaustion at 32k, not wrong reasoning. The 35B MoE with 131k context would likely score higher.*

Ubatch PP Trick (coder543, May 18)

u/coder543 discovered that increasing -ub from 512→8192 gives 5.5x prompt processing speedup for --n-cpu-moe partially offloaded models. I tested this on the 35B:

Result: doesn't apply with --fit on**.** The -ub 2048+ OOMs because --fit on already maximizes VRAM for model layers — no headroom for larger batch buffers. If you use --n-cpu-moe manual offload instead, the trick works. But --fit on is simpler and handles the split automatically.

Concurrency (-np sweep)

Tested -np 1/2/4 on 10 GSM8K cases:

-np 2 doubles batch throughput at 30% slower per-request speed. -np 4 pushes layers to CPU — 27B drops to 10 tok/s, 35B partially fails. Use -np 1 for interactive chat, -np 2 for batch evaluation.

MTP Reference (for 27B / fully-on-GPU setups)

MTP is worth it when the model fits entirely on GPU (no offload penalty). For the 27B IQ3 on 12GB: 73 tok/s with MTP vs ~56 without. For the 35B on 16GB: skip it (see speed table above).

If you do use MTP:

1. --spec-type draft-mtp — not mtp. Mainline renamed it.
2. -np 1 — b9204 defaults to 4 slots which pushes layers to CPU.
3. --spec-draft-n-max 2 beats 3 (lower acceptance at 3 = slower overall).
4. --fit-target 1536 for partial-offload models. --fit-target 0 for fully-on-GPU.
5. At 128k context, MTP gives no speedup — KV cache dominates VRAM regardless.

Other notes:

• Hadamard KV rotation (-khad) is enabled by default since b8607 — no flag needed.
• -np 2 doubles batch throughput at 30% slower per-request. Good for eval, bad for interactive.

Recommendation

The Config (just copy this)

./llama-server \
  -m Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf \
  -c 131072 -np 1 --fit on --fit-target 1536 \
  -fa on -t 20 --no-mmap --jinja \
  -ctk q8_0 -ctv q8_0

No MTP. No special flags. --fit-target 1536 is the key — it reserves VRAM headroom so the KV cache doesn't OOM at 128k. Load it, leave it running, point your coding agent at localhost:8080/v1/chat/completions.

What you get: 56 tok/s generation at 128k context. 1,584 tok/s prompt processing (81s to ingest 128k tokens). 131k max context. GSM8K 91%. Stable.

Why no MTP? At 128k context both MTP and no-MTP give the same 56 tok/s — the KV cache dominates VRAM either way. MTP adds 5 gotchas for zero benefit. Skip the complexity.

GGUF: havenoammo/Qwen3.6-35B-A3B-MTP-GGUF (the MTP GGUF works fine without --spec-type draft-mtp — it just ignores the extra tensors).

27B GGUF: GazTrab/Qwen3.6-27B-MTP-UD-IQ3_XXS-GGUF

Other VRAM budgets (community data, not tested by us)

Everything above was tested on our RTX 5080 16GB. These estimates for other GPUs are from community reports:

The 27B IQ3+MTP needs the MTP head grafted — graft-mtp.py in the repo.

Why not the others?

27B IQ3 — We tested it on our 16GB card where it fits fully on GPU (12.45 GB model). Perfect CodeNeedle (220/220), 73 tok/s with MTP (GGUF). But it caps at 56k context (110k with q4_0 KV). If your coding agent needs 128k, it's out. Better suited for 12 GB cards where the 35B won't fit.

35B Q8_0 — 38% slower (46 tok/s with MTP), negligible quality gain (GSM8K 90% vs 91%, overlapping CIs). Not worth the VRAM on 16 GB.

Credits

This post exists because of the community:

• am17an — original MTP implementation (PR #22673), merged mainline b9190
• havenoammo — MTP GGUF variants + graft script
• u/janvitos — 80 tok/s MTP config on 12GB (635 upvotes), documented the flags
• u/coder543 — ubatch PP trick for --n-cpu-moe (May 18)
• u/OsmanthusBloom — earlier ubatch discovery
• u/Still-Notice8155 — GTX 1070 8GB MTP benchmarks proving it works everywhere
• u/raketenkater — run-time-repack, defrag-thold, -khad flags documentation
• u/moflinCASIO — 4060 Ti 16GB reference benchmarks
• u/WarthogConfident4039 — requested this benchmarking round
• ggerganov — llama-eval, MTP mainline merge
• u/simracerman — pushed for PP speed benchmarks ("your typical coding agent dumps 10k tokens")
• u/danielhanchen (Unsloth) — Dynamic quantization formula behind UD-Q4_K_XL
• u/alexziskind1 — CodeNeedle positional recall benchmark

What's Next

vLLM vs llama.cpp head-to-head. vLLM >= 0.19.0 supports MTP natively with PagedAttention (dynamic KV allocation — no fixed compute buffer eating VRAM). Could make MTP actually faster for partial-offload models. Stay tuned.

EDIT: u/Look_0ver_There — corrected 24 GB VRAM table (Q8_0 is 36 GB, doesn't fit)

EDIT 2: u/FusionX correctly points out that --fit-target 1536 is too conservative for headless setups. My machine runs a desktop compositor + terminal that eats ~1 GB VRAM before the model loads. If you're running headless, --fit-target 128 keeps more expert layers on GPU. FusionX reports 70-80 tok/s at 131k context on the same GPU with this setting. I'll re-benchmark with a lower fit-target and update. The recommended config is adjust --fit-target down if you're headless.

EDIT 3: Hey thanks everyone for commenting, and for the ones who really skeptical of the results because the post was AI generated. u/the__storm u/Special_Animal2049 kevin_1994 I really appreciate your criticisms, and I should have been more upfront about this. So to remedy this I have posted the scripts that produced these results and the raw data themselves, you can find them here: https://github.com/gaztrabisme/llm-server/tree/main/docs/dev

EDIT 4: u/OsmanthusBloom caught that the community VRAM table incorrectly listed the 27B dense model for the 8 GB and 12 GB rows. Both sources actually ran the 35B MoE with CPU offload.

솔직히 말해서, 글쓴이 의도는 진심이고 선의로 하는 거라는 느낌은 들어. 근데 사람들이 AI를 너무 맹신하는 거 아니냐? 특히 빠르게 변하는 최첨단 분야(이번엔 llama.cpp, 더 넓게는 AI 전반)에서는 너무 신뢰할 수 없고 근거 없는 자신감만 넘쳐나.