r/LocalLLaMA•1일 전•36•3

[논문] Stepped MoE: 설정 가능한 추론 복잡도를 갖춘 세그먼트 단위 라우팅

[Paper] Stepped MoE: Segment-Level Routing with Configurable Inference Complexity

핵심 요약

배포 환경과 작업 요구사항에 맞춰 모델 용량을 유연하게 조절하는 Stepped MoE 프레임워크를 소개합니다.

모델 효율성 — 추론 시 정확도와 효율성 사이의 세밀한 균형 조절 가능
동적 파라미터 — 단일 모델 내에서 1B~4B 파라미터까지 유연하게 활용
성능 우위 — 기존 밀집 모델 대비 지식 집약적 벤치마크에서 2~5% 높은 정확도 기록
자원 최적화 — 파라미터 공유를 통해 기기 저장 공간 절약 및 DRAM 환경에 맞춘 서빙 지원

Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures enable flexible model deployment and sparsely activated models allow input-adaptive computation, existing approaches treat these dimensions independently. Moreover, models catered towards on-device edge inference need to conform to the memory and compute limitations of the serving devices. In this paper, we introduce a unified framework that combines elastic structures with sparsely gated architectures to create models that adapt simultaneously to both deployment constraints and task requirements. Our approach employs a model backbone that conditions on both the context and target efficiency specifications, enabling fine-grained control over the accuracy-efficiency trade-off at inference time. The model learns to activate task-relevant parameters within elastically-nested sub-networks, allowing a single model to span multiple capacity points while maintaining input-adaptive routing. Through experiments we demonstrate that we can create a model that allows the flexibility to use 1,2,3,4 billion parameters while being more accurate than their dense counter-parts (2-5\% on knowledge-intensive benchmarks) and at par with their static versions while delivering similar latency metrics as dense models. Overall, we save on device disk space by sharing the model parameters, allow flexibility of serving based on DRAM and compute available while delivering more accurate results.

원문

커뮤니티 목록

r/LocalLLaMA•1일 전•36•3

[논문] Stepped MoE: 설정 가능한 추론 복잡도를 갖춘 세그먼트 단위 라우팅

[Paper] Stepped MoE: Segment-Level Routing with Configurable Inference Complexity

핵심 요약

배포 환경과 작업 요구사항에 맞춰 모델 용량을 유연하게 조절하는 Stepped MoE 프레임워크를 소개합니다.

모델 효율성 — 추론 시 정확도와 효율성 사이의 세밀한 균형 조절 가능
동적 파라미터 — 단일 모델 내에서 1B~4B 파라미터까지 유연하게 활용
성능 우위 — 기존 밀집 모델 대비 지식 집약적 벤치마크에서 2~5% 높은 정확도 기록
자원 최적화 — 파라미터 공유를 통해 기기 저장 공간 절약 및 DRAM 환경에 맞춘 서빙 지원

Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures enable flexible model deployment and sparsely activated models allow input-adaptive computation, existing approaches treat these dimensions independently. Moreover, models catered towards on-device edge inference need to conform to the memory and compute limitations of the serving devices. In this paper, we introduce a unified framework that combines elastic structures with sparsely gated architectures to create models that adapt simultaneously to both deployment constraints and task requirements. Our approach employs a model backbone that conditions on both the context and target efficiency specifications, enabling fine-grained control over the accuracy-efficiency trade-off at inference time. The model learns to activate task-relevant parameters within elastically-nested sub-networks, allowing a single model to span multiple capacity points while maintaining input-adaptive routing. Through experiments we demonstrate that we can create a model that allows the flexibility to use 1,2,3,4 billion parameters while being more accurate than their dense counter-parts (2-5\% on knowledge-intensive benchmarks) and at par with their static versions while delivering similar latency metrics as dense models. Overall, we save on device disk space by sharing the model parameters, allow flexibility of serving based on DRAM and compute available while delivering more accurate results.