Accelerated Out of Core Shuffling

RapidsMPF is an out-of-core shuffling system that enables efficient data shuffling at speeds of 1.8 TiB/s by managing memory constraints through controlled spillover to disk rather than causing out-of-memory errors. Shuffling is a critical bottleneck in distributed data analytics for operations like joins, groupby, and sort, as it requires moving data between processes and managing synchronization across all producers. The system addresses the memory-intensive, transport-heavy, and synchronization-dependent nature of shuffles by providing a reusable framework that allows users to budget for spillover costs instead of facing unpredictable memory failures.