Speaker
Abstract
Abstractions are what allow us to build the complex applications that we all use day-to-day. For example, it's rare for us to care about the precise details of on-disk storage when building an application — that's why databases exist!
Debugging is different though. It forces us to break through those abstractions in order to understand what the computer is really doing.
In this talk, we'll explore the aftermath of a complex outage in a Postgres cluster. We'll retrace the steps we took to reliably reproduce the failure in a local environment and pull out lessons about debugging complex systems along the way. At one point, we'll dive into the depths of how Postgres represents data on disk, and realise that even unfamiliar layers of a system don't need to be scary.
Interview
I work as an infrastructure engineer at a company called PlanetScale where we build a MySQL managed database platform. The focus of my work specifically is building all of the infrastructure underneath the database that helps it to run super reliably and automated and without too many hands.
My talk's motivation is about trying to help people get better at debugging really complex problems. I think it's really easy to become very skilled in other parts of writing software and lag behind in terms of debugging skills. It's something I feel very passionate about as an infrastructure engineer because a lot of my job is figuring out why things are going wrong. I actually don't think it's that hard, I think people often find it tricky because it's outside of their comfort zone. In my talk I go through a really complex example of a failure, but show that it's really just about applying the same step-by-step approach.
Anyone who builds and runs software in production. In terms of level, I'd say probably mid to senior plus. I try to avoid assuming any domain knowledge in the talk. I do assume a base level of programming knowledge to get there.
I'd like people to believe that they can do things that they're not so familiar with. The specific example I use in the talk is of a database clustering outage. It's about following that methodical debugging process, in which we had to dive all the way down into the binary on disk format of the database - which is a scary place that you don't normally go to.
Topics
QCon London 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Part of the track
Debugging Production Hosted by Abby Bangser Principal Engineer @SyntassoFrom the same track
Tuesday 28 March
10:35 Churchill (Ground Fl.) Session debugging Deconstructing an Abstraction to Reconstruct an Outage Chris Sinjakli Infra Engineer @planetscaledata Abstractions are what allow us to build the complex applications that we all use day-to-day. For example, it's rare for us to care about the precise details of on-disk storage when building an application — that's why databases exist! 11:50 Fleming (3rd Fl.) Session application security Celebrity Vulnerabilities: Effective Response to Critical Production Threats Alyssa Miller Chief Information Security Officer @EpiqGlobal Log4Shell, Spring4Shell, are you tired of being told to drop everything and respond to the next critical vulnerability in an open-source package? Chances are, if you work in the engineering team of any software development organization, the answer is yes. 13:40 Churchill (Ground Fl.) Session web development Observable Frontends Jessica Kerr Principal Developer Evangelist @honeycombio As an industry, we’ve made big strides in working within complexity in microservices: we build in observability with OpenTelemetry standards. But what about client-side? This is the most inscrutable part of our system, because it runs on anyone’s computer. 14:55 Rutherford (4th Fl.) Unconference Unconference: Debugging in Production Shane Hastie Global Delivery Lead @SoftEd, Lead Editor for Culture & Methods @InfoQ What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions. 16:10 Churchill (Ground Fl.) Session Java Your Java Application Is Slow? Check Out These Open-Source Profilers Johannes Bechberger Software Developer @SAP Profilers help to analyze performance bottlenecks of your application - if you know which to use and how to work with them. There are many open-source profilers, like async-profiler or JMC. This talk will give you insights into these tools, focusing on: 17:25 Mountbatten (6th Fl.) Session No Instrumentation Observability With eBPF - Are We There Yet? Anna Kapuścińska Software Engineer @Isovalent Gaining interest for the past few years, eBPF promises zero-instrumentation observability with low performance overhead. Sounds like a dream, but are we there already?