Streaming Data & Architectures AMA w/ F. Paganelli & L. Mezzalira

Extreme Programming Meets Realtime Data

At Unruly, we're fast and furious both in terms of the volumes we're handling and in our development process. Scaling incrementally has raised some interesting challenges in how to deal with growing data volumes, and by trying to deliver value in small steps, we've ended up with some...

Microservices At The Heart of BBC iPlayer

The BBC iPlayer API used to be a monolithic Java application. Two years ago we switched to Node.js microservices and we have been iterating on it ever since. We have learnt many things along the way including how to keep distributed systems decoupled, how to monitor the health of our services in...

Power of the Log:LSM & Append Only Data Structures

This talk is about the beauty of sequential access and append only data structures. We'll do this in the context of a little known paper entitled “Log Structured Merge Trees”. LSM describes a surprisingly counterintuitive approach to storing and accessing data in a sequential fashion. It came...

Streaming a Million likes/second: Real-time Interactions on Live Video

When a broadcaster like BBC streams a live video on LinkedIn, tens of thousands of viewers will watch it concurrently. Typically, hundreds of likes on the video will be streamed in real-time to all of these viewers. That amounts to a million likes/second streamed to viewers per live video. How do...

From Batch to Streaming to Both

In this talk I walk through how the streaming data platform at Skyscanner evolved over time. This platform now processes hundreds of billions of events per day, including all our application logs, metrics and business events. But streaming platforms are hard, and we did not get it right on day...