Looking Under the Hood: Data Processing Systems Performance Tricks (and How to Apply Them to Your Code)

QCon London 2026

Session Data Management

Looking Under the Hood: Data Processing Systems Performance Tricks (and How to Apply Them to Your Code)

Wednesday Mar 18 / 02:45PM GMT, Mountbatten (6th Fl.) at The QEII Centre, London

Abstract

Modern data processing systems—databases, analytics engines, vector stores, and stream processors—hide an extraordinary amount of performance engineering beneath their abstractions. In this talk, we “open the lid” on data systems and reveal the low‑level performance tricks they rely on to achieve staggering throughput: vectorized execution, branch‑aware operators, cache‑optimized data layouts, late materialization, compressed‑buffer processing, adaptive execution paths, and microarchitectural tuning. We’ll break down what really happens between a high‑level query and the CPU pipeline—and why systems spend so much energy manipulating data before doing any useful computation.

Drawing on data processing systems research, we explore the surprising bottlenecks hidden inside operators, memory hierarchies, and parallelization strategies. We abstract these insights into concrete techniques you can apply directly in your own codebases: when to favor columnar layouts, how to reduce branch mispredictions, when compression accelerates computation, how to leverage vectorization safely, and how modern profiling tools can expose CPU‑level inefficiencies that traditional metrics miss.

Whether you build databases, ML pipelines, backend services, or high‑performance libraries, this talk will give you a new mental model of how data really flows through modern hardware—and how to “liberate" the best ideas from the fastest systems in the world to make your code faster.

Topics

Data Management systems performance
76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon London 2026 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Share

From the same track

Wednesday 18 March

10:35 Windsor (5th Fl.) Session AI/ML Machine Learning at the Edge of Scale and Speed: Nanosecond Inference at the CERN Large Hadron Collider Thea Klaeboe Aarrestad Particle Physics and Real-Time ML @CERN @ETH Zürich 11:45 Windsor (5th Fl.) Session Data Systems Vector Search on Columnar Storage Peter Boncz Professor @CWI, Co-Creator of MonetDB, VectorWise and MotherDuck, Database Systems Researcher, and Entrepreneur 13:35 Mountbatten (6th Fl.) Session architecture Not Just I/O: Using Async/Await for Computational Scheduling Orson Peters Senior Engineer of Query Execution @Polars, (Co-)Author of Stdlib Sort in Rust & Go 14:45 Mountbatten (6th Fl.) Session Data Management Looking Under the Hood: Data Processing Systems Performance Tricks (and How to Apply Them to Your Code) Holger Pirk Associate Professor for Data Management Systems at Imperial College London and Avid Runner — Minimizing Cache Misses, Thread Divergence and Aerobic Decoupling 15:55 Windsor (5th Fl.) Session compilers Automatically Retrofitting JIT Compilers Laurence Tratt Shopify / Royal Academy of Engineering Research Chair in Language Engineering @King's College London