r/LocalLLaMA•1일 전•83•4

[논문] SpecOPD: 추론 가속을 위한 브랜치 병렬 온폴리시 증류 - Qwen 팀

[Paper] SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding - By Qwen

핵심 요약

Qwen 팀이 추론 속도를 획기적으로 높이는 새로운 추론 가속 프레임워크인 SpecOPD를 공개했습니다.

SpecOPD 프레임워크 — 추론 가속을 위해 브랜치 병렬 온폴리시 증류 방식을 도입함
성능 향상 — 기존 대비 수용 길이 41.45% 증가 및 최대 3.56배의 생성 속도 달성
병렬 처리 — 독립적인 브랜치에서 샘플링과 타겟 감독을 병렬화하여 효율성 극대화
다양한 타겟 지원 — 8B부터 125B 모델까지 다양한 아키텍처와 도메인에서 검증됨

Speculative decoding (SD) accelerates large language model inference by reducing the sequential cost of token-by-token generation. Advanced SD methods reduce proposal cost with parallel drafters that predict entire blocks in one forward pass, but overall speedup still depends on acceptance length. On-policy distillation (OPD) can push well-trained drafters beyond acceptance-length plateaus, yet conventional pipelines collect supervision through sequential draft-and-verify rollouts. Our key insight is that onpolicy supervision need not inherit these dependencies: a draft block can be supervised using only its reference prefix and draft-sampled history. We propose SpecOPD, a branch-parallel OPD framework that parallelizes draft sampling and target supervision across independent branches drawn from the evolving drafter policy at reference anchors. We pack these branches into proposal trees to share target-prefix computation, using tree masks for full attention, branch-consistent indexer groups for sparse attention, and indexed state replay for linear attention to preserve each branch’s history. We train the drafter with an E2E-TV surrogate that directly targets acceptance length. Our study spans 8B full-attention (GQA), 27B full–linear (GQA–GDN), and 125B sparse–linear (QSA–GDN) targets, three parallel draft architectures (DFlash, DSpark, and DFlash V2), and four task domains (dialogue, coding, mathematics, and agentic). .

원문

커뮤니티 목록

r/LocalLLaMA•1일 전•83•4

[논문] SpecOPD: 추론 가속을 위한 브랜치 병렬 온폴리시 증류 - Qwen 팀

[Paper] SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding - By Qwen

핵심 요약

Qwen 팀이 추론 속도를 획기적으로 높이는 새로운 추론 가속 프레임워크인 SpecOPD를 공개했습니다.

SpecOPD 프레임워크 — 추론 가속을 위해 브랜치 병렬 온폴리시 증류 방식을 도입함
성능 향상 — 기존 대비 수용 길이 41.45% 증가 및 최대 3.56배의 생성 속도 달성
병렬 처리 — 독립적인 브랜치에서 샘플링과 타겟 감독을 병렬화하여 효율성 극대화
다양한 타겟 지원 — 8B부터 125B 모델까지 다양한 아키텍처와 도메인에서 검증됨

Speculative decoding (SD) accelerates large language model inference by reducing the sequential cost of token-by-token generation. Advanced SD methods reduce proposal cost with parallel drafters that predict entire blocks in one forward pass, but overall speedup still depends on acceptance length. On-policy distillation (OPD) can push well-trained drafters beyond acceptance-length plateaus, yet conventional pipelines collect supervision through sequential draft-and-verify rollouts. Our key insight is that onpolicy supervision need not inherit these dependencies: a draft block can be supervised using only its reference prefix and draft-sampled history. We propose SpecOPD, a branch-parallel OPD framework that parallelizes draft sampling and target supervision across independent branches drawn from the evolving drafter policy at reference anchors. We pack these branches into proposal trees to share target-prefix computation, using tree masks for full attention, branch-consistent indexer groups for sparse attention, and indexed state replay for linear attention to preserve each branch’s history. We train the drafter with an E2E-TV surrogate that directly targets acceptance length. Our study spans 8B full-attention (GQA), 27B full–linear (GQA–GDN), and 125B sparse–linear (QSA–GDN) targets, three parallel draft architectures (DFlash, DSpark, and DFlash V2), and four task domains (dialogue, coding, mathematics, and agentic). .