RTX Pro 4500(Oculink 연결)에서 PrismaQuant, INT4 Autoround 및 NVFP4 W4A4 양자화 모델 테스트

PrismaSCOUT docker command with 0.24 vLLM:

docker run -d --name vllm-prismascout --restart unless-stopped --gpus all --ipc=host -p 8000:8000 --env-file .env -e HF_HUB_OFFLINE=1 -e VLLM_USE_FLASHINFER_SAMPLER=1 -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/rw/vllm/models:/models:ro --entrypoint /bin/bash "vllm/vllm-openai:v0.24.0-cu129-ubuntu2404" -lc 'exec vllm serve /models/rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm --served-model-name Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm --host 0.0.0.0 --port 8000  --max-model-len 200000 --gpu-memory-utilization 0.96 --kv-cache-dtype fp8 --quantization compressed-tensors --trust-remote-code --enable-chunked-prefill --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice --max-num-seqs 1 --max-num-batched-tokens 8192 --speculative-config '"'"'{"method":"mtp","num_speculative_tokens":3}'"'"' --performance-mode interactivity --attention-backend flashinfer --enable-prefix-caching --no-disable-hybrid-kv-cache-manager --limit-mm-per-prompt '"'"'{"image":4}'"'"' --chat-template /models/rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm/chat_template.jinja'

PrismaAURA docker command with 0.24 vLLM:

docker run -d --name vllm-prismaaura --restart unless-stopped --gpus all --ipc=host -p 8000:8000 --env-file .env -e HF\_HUB\_OFFLINE=1 -e VLLM\_USE\_FLASHINFER\_SAMPLER=1 -e VLLM\_NVFP4\_GEMM\_BACKEND=flashinfer-cutlass -e PYTORCH\_CUDA\_ALLOC\_CONF=expandable\_segments:True -v /home/rw/vllm/models:/models:ro --entrypoint /bin/bash "vllm/vllm-openai:v0.24.0-cu129-ubuntu2404" -lc 'exec vllm serve /models/rdtand/Qwen3.6-27B-PrismaAURA-5.5bit-vllm --served-model-name Qwen3.6-27B-PrismaAURA-5.5bit-vllm --host 0.0.0.0 --port 8000 --max-model-len 150000 --gpu-memory-utilization 0.96 --kv-cache-dtype fp8 --quantization compressed-tensors --trust-remote-code --enable-chunked-prefill --reasoning-parser qwen3 --tool-call-parser qwen3\_coder --enable-auto-tool-choice --max-num-seqs 1 --max-num-batched-tokens 8192 --speculative-config '"'"'{"method":"mtp","num\_speculative\_tokens":3}'"'"' --performance-mode interactivity --attention-backend flashinfer --enable-prefix-caching --no-disable-hybrid-kv-cache-manager --limit-mm-per-prompt '"'"'{"image":4}'"'"' --chat-template /models/rdtand/Qwen3.6-27B-PrismaAURA-5.5bit-vllm/chat\_template.jinja'

Note that because PrismaAURA has higher precision, its weights take up 2GB more space than PrismaSCOUT, so the context length needs to be dropped to 150K.

----

KLD Result from PrismaQuant Repo for PrismaSCOUT, as well as model card for PrismaAURA

Other KLD result on NVFP4 and INT4 Autoround could take reference here.

Sakamakismile, PrismaSCOUT, and Lorbus are tested until 210K context length, the tokenizer being the one that comes with the model.

Testing parameters on those 3 with llama-benchy:

--pp 2048 --tg 480 --depth 0 1000 5000 10000 20000 50000 100000 150000 200000 210000 --latency-mode generation --skip-coherence --concurrency 1--pp 2048 --tg 480 --depth 0 1000 5000 10000 20000 50000 100000 150000 200000 210000 --latency-mode generation --skip-coherence --concurrency 1

PrismaAURA till 130K context length with llama-benchmarks:

--pp 2048 --tg 480 --depth 0 1000 5000 10000 20000 50000 100000 130000 --latency-mode generation --skip-coherence --concurrency 1--pp 2048 --tg 480 --depth 0 1000 5000 10000 20000 50000 100000 130000 --latency-mode generation --skip-coherence --concurrency 1

I think it may be useful for others to take reference related to the KLD and generation speed; hence, this post was born.

The raw llama-bency test result and progress log has been dumped onto this repo as well

I plan to test it with the tool-eval-bench later on as well, maybe another post.

PS: Edit for spelling and provide model card