Cloud-native engineering talks frequently focus on implementing and operating the stateless parts of applications. Designing and deploying stateful services such as databases and message brokers, involves additional concerns: we must ensure data durability as well as service availability, and consider the cost/performance tradeoffs of placing data on local disk, block storage, or object storage.
This talk delves into the often-overlooked complexities of stateful cloud service design, using Neon Serverless Postgres as a case study. We will discuss the key decision points: where do you put your data, and how many copies? How do we ensure availability when starting a new node with an empty cache drive? How can we efficiently scale a service that requires local disk storage? Additionally, we'll examine Kubernetes' role in this landscape and the cost implications of achieving data durability across multiple availability zones or regions.
Storage Engineering Lead @neon.tech, Formerly Redpanda, Inktank (Ceph), Whamcloud (Lustre)
John leads the storage team at Neon, building the persistent storage services behind Neon's serverless Postgres platform. His background is in high scale data-intensive systems, spanning filesystems (Ceph, Lustre), streaming (Redpanda) and databases (Neon). He enjoys writing performant Rust and C++, with a particular focus on the tradeoffs in turning high performance designs into systems that are robust at scale.