You are viewing content from a past/completed QCon -

Track Overview

Streaming Data Architectures

Today’s modern systems generate, as well as process data faster, and in larger volumes than ever before. Whether it’s IOT sensor data, business or social media feeds, or even internal system logs, data often arrives as a continuous stream of events needing attention. Whilst there is no one size fits all, it's crucial to ensure you have a solid streaming architecture in place capable of handling everything thrown at it. Architectures which can scale and are resilient to failures and data surges, and likewise allow for intelligent, timely decision making processes (including AI/ML models) to be embedded and executed at the most appropriate time.  

Join us on this track to learn from innovators and engineers in the trenches on how they are designing systems and leveraging modern data stream processing platforms to deal with such challenges. Hear war stories and gain insights into the concrete approaches and solutions applied, as well as pitfalls to avoid. Finally, peer into the future world of the event streaming database - an emerging new category of database that looks to focus on 'data that moves’.


From this track

SESSION + Live Q&A Interview Available

Streaming a Million likes/second: Real-time Interactions on Live Video

When a broadcaster like BBC streams a live video on LinkedIn, tens of thousands of viewers will watch it concurrently. Typically, hundreds of likes on the video will be streamed in real-time to all of these viewers. That amounts to a million likes/second streamed to viewers per live video. How do...

Akhilesh Gupta

Sr. Staff Software Engineer @LinkedIn

SESSION + Live Q&A Interview Available

Internet of Tomatoes: Building a Scalable Cloud Architecture

Five years ago we started on a journey of building a website monitoring tool. Little did I know that this would land up morphing into a full IoT based agriculture platform. Discussing if tomatoes need dark hours to sleep was not the type of question I had anticipated having to answer. But...

Flavia Paganelli

CTO and Founder @30Mhz

SESSION + Live Q&A London

Databases and Stream Processing: A Future of Consolidation

Are databases and stream processors wholly different things, or are they really two sides of the same coin? Certainly, stream processors feel very different from traditional databases when you use them. In this talk, we’ll explore why this is true, but maybe more importantly why it's...

Benjamin Stopford

Author of “Designing Event Driven Systems” & Senior Director @confluentinc

SESSION + Live Q&A Interview Available

From Batch to Streaming to Both

In this talk I walk through how the streaming data platform at Skyscanner evolved over time. This platform now processes hundreds of billions of events per day, including all our application logs, metrics and business events. But streaming platforms are hard, and we did not get it right on day...

Herman Schaaf

Senior Software Engineer @Skyscanner

SESSION + Live Q&A Silicon Valley

Machine Learning Through Streaming at Lyft

Uses of Machine Learning are pervasive in today’s world. From recommendations systems to ads serving. In the world of ride sharing we use Machine Learning to make a lot of decisions in realtime, for example: supply/demand curves are used to get an accurate ETA(estimated time of arrival) and...

Sherin Thomas

Senior Software Engineer @Lyft

UNCONFERENCE + Live Q&A Silicon Valley

Streaming Data Architectures Open Space

Details to follow.


Speakers from this track

Akhilesh Gupta

Sr. Staff Software Engineer @LinkedIn

Akhilesh is the technical lead for LinkedIn's Real-time delivery infrastructure and LinkedIn Messaging. He has been working on the revamp of LinkedIn’s offerings to instant, real-time experiences. Before this, he was the head of engineering for the Ride Experience program at Uber...

Read more
Find Akhilesh Gupta at:

Flavia Paganelli

CTO and Founder @30Mhz

Flavia is co-founder and CTO of 30MHz, a data platform for agriculture with a mission to grow food more sustainably and efficiently. At 30MHz, the challenges range from managing a fleet of thousands of devices to making beautiful visualizations for data, and let's not forget the terabytes of...

Read more

Benjamin Stopford

Author of “Designing Event Driven Systems” & Senior Director @confluentinc

Ben is a Senior Director at Confluent (a company that backs Apache Kafka) where he runs the Office of the CTO. He's worked on a wide range of projects from implementing the latest version of Kafka’s replication protocol through to assessing and shaping Confluent's strategy. His...

Read more
Find Benjamin Stopford at:

Herman Schaaf

Senior Software Engineer @Skyscanner

Herman Schaaf is a senior software engineer at Skyscanner, where he works primarily on building the central data platform. Before this he worked on applications in machine learning and machine translation, including an offline mobile application that can recognize and translate Chinese to...

Read more
Find Herman Schaaf at:

Sherin Thomas

Senior Software Engineer @Lyft

Sherin is a Senior Software Engineer at Lyft. In her career spanning 8 years, she has worked on most parts of the tech stack, but enjoys the challenges in Data and Machine Learning the most. Most recently she has been focussed on building products that would facilitate advances in Artificial...

Read more

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.