Streaming
Past Presentations
Extreme Programming Meets Realtime Data
At Unruly, we're fast and furious both in terms of the volumes we're handling and in our development process. Scaling incrementally has raised some interesting challenges in how to deal with growing data volumes, and by trying to deliver value in small steps, we've ended up with some...
Microservices At The Heart of BBC iPlayer
The BBC iPlayer API used to be a monolithic Java application. Two years ago we switched to Node.js microservices and we have been iterating on it ever since. We have learnt many things along the way including how to keep distributed systems decoupled, how to monitor the health of our services in...
Power of the Log:LSM & Append Only Data Structures
This talk is about the beauty of sequential access and append only data structures. We'll do this in the context of a little known paper entitled “Log Structured Merge Trees”. LSM describes a surprisingly counterintuitive approach to storing and accessing data in a sequential fashion. It came...
Streaming a Million likes/second: Real-time Interactions on Live Video
When a broadcaster like BBC streams a live video on LinkedIn, tens of thousands of viewers will watch it concurrently. Typically, hundreds of likes on the video will be streamed in real-time to all of these viewers. That amounts to a million likes/second streamed to viewers per live video. How do...
From Batch to Streaming to Both
In this talk I walk through how the streaming data platform at Skyscanner evolved over time. This platform now processes hundreds of billions of events per day, including all our application logs, metrics and business events. But streaming platforms are hard, and we did not get it right on day...
Streaming Data Architectures Open Space
Details to follow.
Interviews
Streaming a Million likes/second: Real-time Interactions on Live Video
What is the work you're doing today?
I'm the Tech Lead for LinkedIn Messaging and LinkedIn’s Real-time Distribution Platform. This is a platform that we use to deploy server-to-client streaming technology to power many dynamic experiences on LinkedIn. This includes instant distribution of likes, comments and concurrent viewer counts on live videos, instant...
Read Full InterviewFrom Batch to Streaming to Both
What is the work you're doing today?
I am a Principal Software Engineer at Skyscanner working on the data platform. This is the central data platform that powers all the Skyscanners' events, metrics and logs. My primary role there is making sure that the 2 million or so events we receive every second arrives safely and securely in long term storage, which is in S3,...
Read Full Interview