Deconstructing an Abstraction to Reconstruct an Outage

QCon London 2023

Session debugging

Deconstructing an Abstraction to Reconstruct an Outage

Tuesday Mar 28 / 10:35AM BST, Churchill (Ground Fl.)

Abstract

Abstractions are what allow us to build the complex applications that we all use day-to-day. For example, it's rare for us to care about the precise details of on-disk storage when building an application — that's why databases exist!

Debugging is different though. It forces us to break through those abstractions in order to understand what the computer is really doing.

In this talk, we'll explore the aftermath of a complex outage in a Postgres cluster. We'll retrace the steps we took to reliably reproduce the failure in a local environment and pull out lessons about debugging complex systems along the way. At one point, we'll dive into the depths of how Postgres represents data on disk, and realise that even unfamiliar layers of a system don't need to be scary.

Interview

I work as an infrastructure engineer at a company called PlanetScale where we build a MySQL managed database platform. The focus of my work specifically is building all of the infrastructure underneath the database that helps it to run super reliably and automated and without too many hands.

My talk's motivation is about trying to help people get better at debugging really complex problems. I think it's really easy to become very skilled in other parts of writing software and lag behind in terms of debugging skills. It's something I feel very passionate about as an infrastructure engineer because a lot of my job is figuring out why things are going wrong. I actually don't think it's that hard, I think people often find it tricky because it's outside of their comfort zone. In my talk I go through a really complex example of a failure, but show that it's really just about applying the same step-by-step approach.

Anyone who builds and runs software in production. In terms of level, I'd say probably mid to senior plus. I try to avoid assuming any domain knowledge in the talk. I do assume a base level of programming knowledge to get there.

I'd like people to believe that they can do things that they're not so familiar with. The specific example I use in the talk is of a database clustering outage. It's about following that methodical debugging process, in which we had to dive all the way down into the binary on disk format of the database - which is a scary place that you don't normally go to. 

Topics

debugging database best practices
76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon London 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Share

From the same track

Tuesday 28 March

10:35 Churchill (Ground Fl.) Session debugging Deconstructing an Abstraction to Reconstruct an Outage Chris Sinjakli Infra Engineer @planetscaledata 11:50 Fleming (3rd Fl.) Session application security Celebrity Vulnerabilities: Effective Response to Critical Production Threats Alyssa Miller Chief Information Security Officer @EpiqGlobal 13:40 Churchill (Ground Fl.) Session web development Observable Frontends Jessica Kerr Principal Developer Evangelist @honeycombio 14:55 Rutherford (4th Fl.) Unconference Unconference: Debugging in Production Shane Hastie Global Delivery Lead @SoftEd, Lead Editor for Culture & Methods @InfoQ 16:10 Churchill (Ground Fl.) Session Java Your Java Application Is Slow? Check Out These Open-Source Profilers Johannes Bechberger Software Developer @SAP 17:25 Mountbatten (6th Fl.) Session No Instrumentation Observability With eBPF - Are We There Yet? Anna Kapuścińska Software Engineer @Isovalent