It’s not clear to me why this would be faster than polars, duckdb, vaex or clickhouse. They seem to be taking the same approach of multithreading, optimizing the plan, using arrow, optimizing the core functions like group by.
I’m asking more about what techniques did they use to get the performance improvements in the slides.
They are showing a 20-30% improvement over Polars, Clickhouse and Duckdb. But those 3 tools are SOTA in this area and generally rank near eachother in every benchmark.
So 20-30% improvement over that cluster makes me interested to know what techniques they are using to achieve that over their peers.
