| |
14× faster embeddings: how we rebuilt the ONNX path in Manticore
Manticore Search rebuilt its ONNX model inference path and achieved approximately 14× faster embedding performance compared to the previous SentenceTransformers/Candle implementation, with throughput increasing from 5-11 docs/sec to 70-230 docs/sec on standard hardware. The new backend, released in Manticore Search 27.1.5, leverages ONNX Runtime and eliminates previous bottlenecks related to lock contention and thread management, automatically benefiting users with no API changes required.
Read Full Article →
← More Tech news