Rotary GPU: Exploring Local Execution for Large MoE Models Under Limited VRAM

Researchers present Rotary GPU, a method for running large Mixture-of-Experts language models on consumer hardware with limited VRAM, successfully executing a 35B-parameter model on a laptop with an 8GB GPU while maintaining reasonable performance (21 tokens/second). The approach aims to improve accessibility of advanced AI models to organizations and environments lacking access to large accelerator clusters, addressing practical deployment constraints beyond raw model capability.