Anatomy of a high-performance EP kernel

Large language models require multiple GPUs working together for inference, and for mixture-of-experts (MoE) models, "wide Expert Parallelism" (wideEP) is the most efficient approach at scale. Unlike other parallelism methods with fixed communication patterns, expert parallelism requires dynamic, runtime-determined routing of tokens to different GPUs based on router decisions, making the communication kernel design more complex. This article explains the anatomy of high-performance expert parallelism kernels, building up from how tokens with their routing assignments must be dispatched across a distributed GPU cluster to meet their assigned experts.