당신의 추론 엔진이 모델을 바보로 만들고 있나요? 어느 정도는 그렇습니다, 제가 측정해 봤거든요
Is your engine lobotomizing your model? Kinda, and I measured it
핵심 요약
추론 엔진의 성능 최적화 '핵'이 모델의 지능과 정확도를 얼마나 심각하게 저하시키는지 실험을 통해 증명했습니다.
- 성능 최적화의 대가 — 라우팅 편향과 테일 스킵은 속도를 높이지만 모델의 추론 능력을 크게 떨어뜨림
- 측정된 성능 저하 — 최적화 설정 시 Q8 모델이 Q2 수준의 성능으로 하락하는 현상 확인
- 실제 작업 영향 — 복잡한 다단계 추론과 사실 관계 기억에서 눈에 띄는 정확도 손실 발생
- 옵트인 권장 — 이러한 손실을 유발하는 설정은 기본값이 아닌 사용자 선택 사항이어야 함
So I've been reading a lot about "engines lobotomizing the models they run", either because they quantize already quantized weights even further, or because they automatically turn on lossy flags to gain decode/prefill performance. I wanted to test this out by myself, not only with KLD but with real tasks, so I set up an experiment using Qwen 3.8 Flash Next with Basalt (disclaimer: I'm the developer) and two of the most common lossy flags:
- Routing bias: the engine pushes the model toward experts that are already on the GPU, so it sometimes gets a convenient expert instead of the one it actually wanted.
- Tail skip: when one of the model's lower expert picks isn't on the GPU, the engine just skips it instead of fetching it.
Both are faster because fewer experts have to come over from RAM, but both change the model's output. I added them to the engine as opt in flags so I could measure exactly what they cost with everything else fixed, only changing the flags.
DISCLAIMER: This post and the experiment were run by me, but it has been polished by AI.
The rig: RTX 5090 (400 W) + RTX 5060 Ti (180 W) and 128 GB of DDR5. Settings tested:
- Shipped: bias 1.5, skip rank ≥ 7. Values other engines ship (or have shipped) as defaults on smaller cards.
- Worst: bias 16, skip rank ≥ 1, plus relaxed draft acceptance. Worst case scenario, not something anyone ships or would make sense to use.
1. KLD
100 chunks of wikitext, decode path, against a llama.cpp Q8_0 reference. Exact IQ3_XXS and Q2_0 are in there for scale.
| Pack + setting | File | PPL | KLD mean | Same top-1 |
|---|---|---|---|---|
| Q8_0 exact | 178 GiB | 3.276 |
