Faster Retrieval for Better, More Efficient RAG

Researchers from Apple present a new approach to maximum inner product search (MIPS)—a core technique used in RAG systems—that trains small neural networks to learn query routing patterns specific to a database, enabling faster and more accurate retrieval. The key innovation recognizes that MIPS queries (short user questions) differ substantially from keys (long documents), and that traditional query-agnostic methods fail to leverage this known query distribution. By framing MIPS as a support function problem from convex analysis, the researchers trade an upfront offline training cost for significantly improved inference-time performance.