Stateful Cloud Services at Neon: Navigating Design Decisions and Trade-Offs

Cloud-native engineering talks frequently focus on implementing and operating the stateless parts of applications. Designing and deploying stateful services such as databases and message brokers, involves additional concerns: we must ensure data durability as well as service availability, and consider the cost/performance tradeoffs of placing data on local disk, block storage, or object storage.

This talk delves into the often-overlooked complexities of stateful cloud service design, using Neon Serverless Postgres as a case study. We will discuss the key decision points: where do you put your data, and how many copies? How do we ensure availability when starting a new node with an empty cache drive? How can we efficiently scale a service that requires local disk storage? Additionally, we'll examine Kubernetes' role in this landscape and the cost implications of achieving data durability across multiple availability zones or regions.


Speaker

John Spray

Storage Engineering Lead @neon.tech, Formerly Redpanda, Inktank (Ceph), Whamcloud (Lustre)

John leads the storage team at Neon, building the persistent storage services behind Neon's serverless Postgres platform. His background is in high scale data-intensive systems, spanning filesystems (Ceph, Lustre), streaming (Redpanda) and databases (Neon). He enjoys writing performant Rust and C++, with a particular focus on the tradeoffs in turning high performance designs into systems that are robust at scale.

Read more
Find John Spray at:

Date

Monday Apr 8 / 05:05PM BST ( 50 minutes )

Location

Churchill (Ground Fl.)

Topics

storage Platform Engineering SaaS

Share

From the same track

Session architecture

Building SaaS Using Cloud-Native Patterns: A Deep Dive Into Diagrid Cloud

Monday Apr 8 / 02:45PM BST

A robust and extensible Cloud platform is the foundation on which to build and distribute powerful Software as a Service (SaaS). It provides a common layer upon which you can quickly iterate to deliver specialized services to meet the needs of your end users.

Speaker image - Joni Collinge
Joni Collinge

Founding Software Engineer @Diagrid

Session Infrastructure as Code

Borderless Cloud: Designing, Implementing, and Securing Apps Across Multiple Clouds

Monday Apr 8 / 11:45AM BST

There are different reasons why your team might want to leverage the multi-cloud architecture. It could be a result of the desire to enhance redundancy and optimize costs, to ensure regulatory compliance, or to capitalize on the diverse services offered by different cloud providers.

Speaker image - Adora Nwodo
Adora Nwodo

Senior Software Engineer, Founder of NexaScale, Multi-Published Author

Session AWS

Efficient Serverless Development: Latest Patterns and Practices on AWS

Monday Apr 8 / 10:35AM BST

Ready to dive into the world of serverless with AWS? In this talk, we're going to cut through the jargon and get straight to what serverless really means and why it's a game-changer for developers like us.

Speaker image - Yan Cui
Yan Cui

AWS Serverless Hero & Independent Consultant

Session DevSecOps

Efficient DevSecOps Workflows With a Little Help From AI

Monday Apr 8 / 03:55PM BST

From idea to the first line of code to production deployments - DevSecOps workflows help develop software faster. Everyone has reached a different level of adoption, and some processes may feel inefficient, or blocking progress and innovation.

Speaker image - Michael Friedrich
Michael Friedrich

Senior Developer Evangelist @GitLab

Session

Unconference: Cloud-Native Engineering

Monday Apr 8 / 01:35PM BST