[Deepseek-V4-Flash-0731] RTX 5090 + DDR5 데스크탑에서 VLLM CPU/RAM 오프로딩을 활용한 1M 컨텍스트 구현 (~800 tps pp & 15+ tps decode)

This post of mine is based on the link above.

My Hardware:

• RTX 5090 32GB
• Ryzen 9 9950X3D
• 256GB DDR5-5600
• Single NUMA node
• Linux Mint
• NVIDIA driver 595.71.05
• CUDA 13.2

Software

• guqiong96/Lvllmds4-x
• vLLM 2.3.9
• lk_moe 2.3.2
• PyTorch 2.11.0+cu130
• native DeepSeek-V4-Flash-0731 safetensors checkpoint
• 48 safetensors shards
• ~155.4 GiB checkpoint size

One fix I needed

During startup, FlashInfer's CUDA IPC helper could accidentally find TileLang's:

libcudart_stub.so

instead of the real loaded CUDA runtime.

That eventually caused:

undefined symbol: cudaDeviceReset

The problem was FlashInfer's find_loaded_library("libcudart") doing a substring search over /proc/self/maps.

I patched:

flashinfer/comm/cuda_ipc.py

so it checks the actual filename instead:

def find_loaded_library(lib_name):
    with open("/proc/self/maps") as f:
        for line in f:
            if "/" not in line:
                continue

            start = line.index("/")
            path = line[start:].strip()
            filename = path.split("/")[-1]

            if (
                filename.startswith(lib_name + ".so")
                or filename.startswith(lib_name + "-")
            ):
                return path

    return None

After that, FlashInfer correctly resolves the real libcudart instead of the TileLang stub.

This is a local patch and obviously needs to be reapplied if the package gets replaced.

Current launch configuration

This is the configuration I ended up using:

source ~/ds4x-venv/bin/activate

MODEL="/home/blackbeard/models/DeepSeek-V4-Flash-0731"

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0

export LVLLM_MOE_NUMA_ENABLED=1
export LK_THREADS=12
export OMP_NUM_THREADS=12
export LK_THREAD_BINDING=CPU_CORE

# Keep two complete routed MoE layers GPU-resident on the GPU.
export LVLLM_GPU_RESIDENT_MOE_LAYERS=0,1

# CPU/hybrid prefill path for now.
export LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=0

export FLASHINFER_DISABLE_VERSION_CHECK=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

vllm serve "$MODEL" \
  --host 0.0.0.0 \
  --port 8070 \
  --tensor-parallel-size 1 \
  --max-model-len 1048576 \
  --gpu-memory-utilization 0.92 \
  --trust-remote-code \
  --served-model-name DeepSeek-V4-Flash-0731 \
  --compilation_config.cudagraph_mode FULL_DECODE_ONLY \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --dtype bfloat16 \
  --max-num-seqs 2 \
  --enable-auto-tool-choice \
  --tool-call-parser deepseek_v4 \
  --kv-cache-dtype fp8_ds_mla \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "max"}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":2,"draft_sample_method":"greedy"}' \
  --disable-custom-all-reduce

Full native 1M context fits on the 32GB GPU even with two complete routed MoE layers resident on the GPU.

The rest of the experts remain in system RAM.

DSpark behaves very differently during reasoning

During long reasoning sections, draft acceptance can collapse.

I observed extended periods around:

Draft acceptance: ~30-50%
Generation:       ~11-13 tok/s

There was one ~6 minute section averaging roughly:

Draft acceptance: ~40%
Generation:       ~11.9 tok/s

Then the model transitioned into a much more predictable generation phase and the numbers jumped to roughly:

Draft acceptance: ~87-88%
Generation:       ~17.4-17.6 tok/s

The relationship is extremely strong: throughput basically tracks DSpark acceptance.

Some high-acceptance windows look like:

Avg Draft acceptance rate: 89.8%
Avg generation throughput: 17.9 tokens/s

while low-acceptance reasoning windows look like:

Avg Draft acceptance rate: 38%
Avg generation throughput: ~12 tokens/s

This suggests an obvious optimization.

Dynamic DSpark depth

For this workload I suspect the ideal behavior would be approximately:

• reasoning/thinking: 1 speculative token
• normal/final decoding: 2 speculative tokens

The second draft token often isn't worth computing while the model is doing difficult reasoning, but becomes very valuable when it transitions into more predictable code/text generation.

vLLM does not currently give me a simple runtime switch for this, so I may patch the speculative decoding path later and experiment with changing the draft depth based on whether the model is currently emitting reasoning or final output.

That looks like one of the biggest remaining decode optimizations.

---non AI comment section begins---

Stay tuned, I am working on a if/else block to fix that stupid behavior slowing down during reasoning and squeeze even more tps out of this stack.

---non AI comment section ends---

그 "vllmds4"는 어디서 구하나요? 원래 vLLM도 아직 DSpark를 지원하지 않는데요.