8GB VRAM과 32GB RAM에서 190k 컨텍스트로 Qwen3.6 35B A3B 구동하기
Running Qwen3.6 35b a3b on 8gb vram and 32gb ram ~190k context
핵심 요약
8GB VRAM 환경에서 Qwen3.6 35B 모델을 190k 컨텍스트로 원활하게 구동하는 최적화 설정 공유.
- 최적화 설정 — 8GB VRAM 환경에서 190k 컨텍스트를 안정적으로 구동하는 튜닝값 공유함.
- TurboQuant 활용 — llama.cpp 포크를 사용하여 고속 추론 및 KV 캐시 효율을 극대화함.
- 하드웨어 제약 — DDR5 RAM 대역폭이 성능에 결정적인 영향을 미치며 리눅스 환경을 권장함.
- 성능 비교 — Q5 양자화 모델이 Q4보다 긴 문맥 추론에서 더 나은 성능을 보여줌.
If anyone is looking for a good high-speed setup with ~190k context, this config has been working insanely well for me.
I’m using my laptop as a server over Tailscale. Installed Linux on it and running:
- Qwen3.6 35B A3B
- RTX 4060 8GB VRAM
- 32GB DDR5 5600MHz RAM
- Q5 quant models
Current models tested:
- `mudler/Qwen3.6-35B-A3B-APEX-GGUF`
- ~40 tok/sec → 37 tok/sec
- `hesamation/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-GGUF`
- ~43 tok/sec → 37 tok/sec
I can push it up to ~51 tok/sec by tweaking:
- `--ctx-size 192640`
- `--n-gpu-layers 430`
- `--n-cpu-moe 35`
and adjusting those values slightly higher/lower depending on stability and memory usage.
Here’s my current config:
#!/bin/bash
# --- LLAMA SERVER LAUNCHER SCRIPT ---
#SELECTED_MODEL="/home/atulloq/.lmstudio/models/hesamation/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled.Q5_K_M.gguf"
SELECTED_MODEL="/home/atulloq/.lmstudio/models/mudler/Qwen3.6-35B-A3B-APEX-GGUF/Qwen3.6-35B-A3B-APEX-I-Balanced.gguf"
echo "Starting Llama Server..."
echo "Model: $SELECTED_MODEL"
/home/atulloq/llama-cpp-turboquant/build/bin/llama-server \
--model "$SELECTED_MODEL" \
--host 0.0.0.0 \
--port 8085 \
--ctx-size 192640 \
--n-gpu-layers 430 \
--n-cpu-moe 35 \
--cache-type-k "turbo4" \
--cache-type-v "turbo4" \
--flash-attn on \
--batch-size 2048 \
--parallel 1 \
--no-mmap \
--mlock \
--ubatch-size 512 \
--threads 6 \
--cont-batching \
--timeout 300 \
--temp 0.2 \
--top-p 0.95 \
