Speaker
Abstract
Analytical data management systems have long been monolithic monsters far removed from the action by ancient protocols. Redesigning them to move into the application process greatly streamlines data transfer, deployment, and management. This new class of systems a whole new class of use cases, for example in-browser or edge OLAP, running SQL queries in lambdas, and Big Data on laptops.
DuckDB is a new analytical data management system that is built for an in-process use case. DuckDB speaks SQL, is trivially integrated as a library, and uses state-of-the art query processing techniques with vectorized execution and lightweight compression. DuckDB is Free and Open Source software that is distributed under the permissive MIT license. In my talk, I will explain the rationale and design decisions behind DuckDB and give a tour of the internals.
Interview
I spend most of my time working on DuckDB. It's the open source project that I co-founded, and we have spun out from the research institute that I worked on into a separate company, which is called DuckDB Labs, which I'm also leading.
The motivation for the talk came from interacting with data practitioners some years ago; we found out that they really hated using data systems. As somebody who builds data systems, I was a bit concerned that the world hated us, so we were starting to rethink how data systems could work. We came up with this idea that they should be running in process, and that's what I want to talk about. I want to show people how powerful this new way of thinking about data systems is.
I think there are two groups. The first group consists of data analysts and data scientists that are interested in processing large data sets with SQL. The other group is data engineers that are trying to build data pipelines, as DuckDB does it. I'm going to talk about this and it's going to be very useful for those with a more embedded role. These are the two groups that I think would most interested.
My first motivation is for them to have heard of us. We are a fast-growing project, but I'm told there are still some people out there that haven't heard of us. I think the way DuckDB works can really open up new possibilities and dimensions for people to think about how to build data pipelines and how to analyze data. So I think for them to walk away with that insight would be great.
Topics
QCon London 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
From the same track
Monday 27 March
10:35 Mountbatten (6th Fl.) Session A New Era for Database Design with TigerBeetle Joran Greef Founder and CEO @TigerBeetle The pre-recorded video of this presentation will become available within the next few hours. 11:50 Windsor (5th Fl.) Session Apache Pinot Speed of Apache Pinot at the Cost of Cloud Object Storage with Tiered Storage Neha Pawar Founding Engineer @StarTree For real-time analytics, you need systems that can provide ultra low latency (milliseconds) and extremely high throughput (hundreds of thousands of queries per second). 13:40 Churchill (Ground Fl.) Session Microservices Change Data Capture for Microservices Gunnar Morling Senior Staff Software Engineer @Decodableco Microservices represent complex business domains in the form of loosely coupled systems, but these don't exist in isolation: services need to propagate data changes amongst each other, in a reliable and scalable way. 14:55 Mountbatten (6th Fl.) Session transactions Amazon DynamoDB Distributed Transactions at Scale Akshat Vig Senior Principal Engineer NoSQL databases @awscloud NoSQL databases are popular for their high availability, high scalability, and predictable performance. 16:10 Mountbatten (6th Fl.) Session raft Multi-Region Data Streaming with Redpanda Michał Maślanka Software Engineer @Redpanda Real time data streaming platforms such as Redpanda have become a mission critical component in enterprise infrastructure. Multi-region deployments of streaming applications can provide important benefits, such as improved resiliency, better performance and cost reduction. 17:25 Mountbatten (6th Fl.) Session processing techniques In-Process Analytical Data Management with DuckDB Hannes Mühleisen Co-founder and CEO @duckdblabs Analytical data management systems have long been monolithic monsters far removed from the action by ancient protocols. Redesigning them to move into the application process greatly streamlines data transfer, deployment, and management.