DuckDB – Data power tools for your laptop, now in Clojure (2023)

DuckDB, a powerful in-memory column-major data processing platform, has been integrated into Clojure through tmducken to handle large datasets that don't fit in memory while maintaining functional programming advantages. Recent improvements to DuckDB's C interface now enable batched inserts and queries, allowing efficient processing of massive datasets—demonstrated by loading a 50GB CSV file with 400 million rows into an 18GB indexed database in under two minutes. This integration allows Clojure developers to leverage DuckDB's vectorized SQL execution engine without sacrificing the benefits of functional programming or requiring complex distributed systems like Spark.