Streaming

From Batch to Streaming to Both

What is the work you're doing today?

I am a Principal Software Engineer at Skyscanner working on the data platform. This is the central data platform that powers all the Skyscanners' events, metrics and logs. My primary role there is making sure that the 2 million or so events we receive every second arrives safely and securely in long term storage, which is in S3, and that they are auditable and reliable. We also need to capture metadata about these events and be able to trace the lineage. That's what I'm working on, and that's what my talk is about as well.

What are the goals you have for the talk?

My main goal is sharing our story of how we used the Agile method to build a data platform and how there is some fundamental tensions between using Agile and delivering in Agile fashion, and the long term planning that you need for a data platform to succeed. My goal here is to share that story, share how that happened, how we got to where we are and what we're doing about it now, and hopefully share a number of lessons that we've learned along the way to help my audience avoid those same mistakes. Hopefully skipping some steps and skipping right to--I wouldn't say the final solution--but a solution that was learned after a couple of hard years of iterating on the problem.