Practical Benchmarking: How To Detect Performance Changes in Noisy Results

Finding statistically significant changes in performance results has always been challenging but now that most of our code runs on hardware and infrastructure we don't own, we need methods and tools for detecting performance changes in noisy data. This talk will provide tips for understanding noise, where it comes, and techniques for fighting it. Plus some anecdotes and real-life examples from the world of open-source.

Interview:

What's the focus of your work these days?

I'm leading the development of Nyrkiö's performance analysis product.

What's the motivation for your talk at QCon London 2024?

I struggled with getting reliable benchmark results for a long time until I learned the tools and tricks discussed in my talk and I want to share these with everyone.

How would you describe your main persona and target audience for this session?

Senior engineers or architects.

Is there anything specific that you'd like people to walk away with after watching your session?

The knowledge of how to handle noise in benchmark results and the latest techniques for doing so.


Speaker

Matt Fleming

CTO @Nyrkiö, Former Linux Kernel Maintainer @Intel and @SUSE

Matt is the co-founder and CTO of Nyrkiö, a company that wants to arm every developer with the tools to understand the performance of their code. He has spent over 15 years working on low-level, high-performance systems and was previously the maintainer for the Linux kernel EFI subsystem while at Intel and SUSE. Most recently, Matt has focused on the performance of distributed systems and databases including leading a globally distributed team of engineers that contributed significant performance changes to the Apache Cassandra 5.0 release. He has co-authored papers on performance change detection and distributed systems testing and served on the ACM/SPEC ICPE program committee. Matt can often be found on Twitter, discussing topics such as software performance, benchmarking and statistics.

Read more
Find Matt Fleming at:

From the same track

Session performance

A Walk Along the Complexity-Performance Curve

Monday Apr 8 / 10:35AM BST

Software performance and complexity are related. It’s common for refactoring to introduce unanticipated regressions, and for performance optimisations to attract scrutiny in code review; how much performance improvement is worth a perceived loss of readability?

Speaker image - Richard Startin

Richard Startin

Senior Software Engineer @Datadog

Session

Panel: What Does the Future of Computing Look Like

Monday Apr 8 / 05:05PM BST

The future of computing promises to be revolutionary. This panel dives into cutting-edge advancements that will redefine how we interact with technology. We'll explore groundbreaking concepts and discuss their potential to transform our world.

Speaker image - Julia Lawall

Julia Lawall

Senior Scientist @INRIA

Speaker image - Matt Fleming

Matt Fleming

CTO @Nyrkiö, Former Linux Kernel Maintainer @Intel and @SUSE

Speaker image - Joe Rowell

Joe Rowell

Founding Engineer @poolside.ai, Low-Level Performance Engineer, Cryptographer and PhD Candidate @RHUL

Session Linux kernel

Opening the Box: Diagnosing Operating-System Task-Scheduler Behavior on Highly Multicore Machines

Monday Apr 8 / 11:45AM BST

An operating system task scheduler is responsible for placing tasks on cores and for selecting which task is allowed to run, at what time. As such, the scheduler is a critical component of any operating system and has a major impact on application performance.

Speaker image - Julia Lawall

Julia Lawall

Senior Scientist @INRIA

Session

Pitfalls of Unified Memory Models in GPUs

Monday Apr 8 / 01:35PM BST

Modern GPUs offer support for so-called unified memory, providing a universal address space for both CPUs and GPUs.

Speaker image - Joe Rowell

Joe Rowell

Founding Engineer @poolside.ai, Low-Level Performance Engineer, Cryptographer and PhD Candidate @RHUL

Session

Unconference: Performance Engineering Unleashed

Monday Apr 8 / 02:45PM BST

An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.