Track: Modern Distributed Architectures

Location:

Day of week:

Building robust distributed systems is hard -- it requires a solid theoretical base grounded in hard-earned practical experience. What architectural patterns should you be aware of when designing for your distributed needs? What are some anti-patterns that have been revealed through war stories from the field? What do recently-released cloud services (e.g. AWS Lambda, GCP dataflow & dataproc, AWS Kinesis) now make possible in cloud-based architectures?

Track Host:
Danny Yuan
Real-time Streaming Lead @Uber
Danny Yuan is a software engineer in Uber. He’s currently working on streaming systems for Uber’s marketplace platform. Prior to joining Uber, he worked on building Netflix’s cloud platform. His work includes predictive autoscaling, distributed tracing service, real-time data pipeline that scaled to process hundreds of billions of events every day, and Netflix’s low-latency crypto services.
10:35am - 11:25am

by Gary Lam
Tech Lead Notifications, Staff Software Engineer @ Twitter

by Saurabh Pathak
Leads Notifications Team @Twitter

Twitter Notifications Infrastructure enables hundreds of millions of users to stay informed about what’s going on in their Twitter world. Our systems process large volumes of data (aka the Twitter firehose) and deliver realtime and personalized notifications to all kinds of users, ranging from Katy Perry with ~95M followers to brand new users trying out our product for the very first time. We will give an overview of unique challenges in building our notifications infrastructure, present how...

11:50am - 12:40pm

by Alvaro Videla
Distributed Systems Engineer

Distributed Systems are a complex topic. There's abundant research about it but sometimes it is hard for a beginner to know where to start. I would like to outline the main concepts of distributed systems, so the interested person can have a clear path on how to start their own research as well. In this talk I will review the different models: asynchronous vs. synchronous distributed systems; message passing vs shared memory communication; failure detectors and...

1:40pm - 2:30pm

by Igor Maravic
Software Engineer @Spotify

Spotify’s event delivery system is one of the foundational pieces of Spotify’s data infrastructure. It has a key requirement to reliably deliver complete data with a predictable latency and make it available to Spotify developers via well-defined interface. Delivered data is than used to produce Discover Weekly, Fresh Finds, Spotify Party and many other Spotify features. Currently 1M events is delivered via Spotify's event delivery system every second. To...

2:55pm - 3:45pm

Open Space
4:10pm - 5:00pm

by Eugene Kirpichov
Cloud Dataflow Sr SE @Google

One of the main causes of performance problems in distributed data processing systems (from the original MapReduce to modern Spark and Flink) is "stragglers." Stragglers are parts of the input that take an unexpectedly long time to process, delaying the completion of the whole job, and wasting resources that stay idle. Stragglers can happen due to imbalance of data distribution or processing complexity, hardware/networking anomalies, and a variety of other...

5:25pm - 6:15pm

by Jim Webber
Chief Scientist @Neo4j

In this talk we'll explore the new Causal clustering architecture for Neo4j. We'll see how Neo4j uses the Raft protocol for a robust underlay for intensive write operations, and how the asynchronous new scale-out mechanism provides enormous capacity for very demanding graph workloads.

We'll discuss the cluster architecture's new causal consistency model. Causal consistency is a big leap forward compared to the commonplace...

Tracks