Benchmarking retrieval for agents on messy real-world company knowledge

Kapa, a company knowledge indexing platform, created the Company Knowledge Bench—a dataset of 1,000 evaluation cases from real production data—to measure retrieval performance on messy real-world company knowledge like documentation, tickets, and code. Their benchmarking results show that an optimized agentic retriever (Kapa Deep) achieved the best score of 0.65 in about five seconds, outperforming both traditional retrieval methods and frontier models using simple grep searches. Public benchmarks were insufficient for their needs because they don't cover the diverse use cases and query types that different teams (developers, sales, support) require when searching company knowledge.