Abstract
In the past two years I have developed a new query execution engine for Polars, which not only tries to execute as much of your query in parallel as possible, but in a streaming fashion as well, such that you can process data sets which do not fit in memory. This new query execution engine is heavily based on the async/await paradigm provided by Rust, and uses it to great effect.
In this talk I’d like to show that async/await is not just a tool to perform I/O requests asynchronously, but can in fact be used to build powerful cooperatively scheduled systems. Featuring custom async channels, primitives and multiple async runtimes, I show how it's used in Polars to achieve fine-grained, cache-friendly and NUMA-aware parallelism. My hope is to inspire you to look at async/await in a different way.
Speaker
Orson Peters
Senior Engineer of Query Execution @Polars, (Co-)Author of Stdlib Sort in Rust & Go
Orson works at Polars, where he focuses on query execution. He is passionate about optimizing code and squeezing the most performance possible out of a CPU, which is a perfect fit. He is the (co-)author of various sorting algorithms (e.g. pdqsort, driftsort, ipnsort), some of which are used in the Rust and Go standard libraries. He has also published fast hashing algorithms (foldhash, polymur-hash), and several other open-source Rust crates (e.g. slotmap).
Find Orson Peters at:
Date
Wednesday Mar 18 / 01:35PM GMT ( 50 minutes )
Location
Mountbatten (6th Fl.)
