Speaker
Abstract
Cloudflare uses Kafka to decouple microservices and communicate the creation, change or deletion of various resources via a common data format in a fault-tolerant manner. This decoupling is one of many factors that enables Cloudflare engineering teams to work on multiple features and products concurrently.
We learnt a lot about Kafka on the way to one trillion messages, and built some interesting internal tools to ease adoption as well as improve resiliency that will be explored in this talk.
Interview
Andrea: I work as a Systems Engineer at Cloudflare. I research and help prioritize what's needed to make other developers' lives easier. I design, implement and maintain platforms used by other teams to do their job effectively. For example, we maintain SDKs for Kafka and platforms that allow emitting alerts, emails audit logs and so on.
We wanted to share more about what we do for productivity, organizational culture, and also I've never spoken to such a big conference before, so it's an amazing chance for me to become more confident.
I think there will be a lot of people that have a lot of experience, senior engineering, CTOs, tech leads.
A better understanding of how to set up for success when deciding to integrate with Kafka, especially in a big company where there's a lot of traffic. Many of the points in our presentation also apply to different worlds, different platforms and systems. I think everyone can get something from it.
Topics
QCon London 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Part of the track
Architectures You've Always Wondered About Hosted by Matt Turner Software Engineer @Tetrateio and Co-Founder for Istio LondonFrom the same track
Monday 27 March
10:35 Churchill (Ground Fl.) Session scalability Scaling Google's Global Cloud L7 Load Balancer James Spooner Principal Engineer, Load Balancing @Google We'll take a look at Google's Global Cloud L7 Balancer, how it's put together and how we've scaled it to meet the reliability and performance demands of our Cloud customers. 11:50 Rutherford (4th Fl.) Unconference Unconference: Architectures You've Always Wondered About Shane Hastie Global Delivery Lead @SoftEd, Lead Editor for Culture & Methods @InfoQ What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions. 13:40 Whittle (3rd Fl.) Session Building High-Fidelity Data Streams Sid Anand Fellow, Cloud & Data Platform @Walmart, Apache Airflow Committer/PMC, Ex-Netflix, LinkedIn, eBay, Etsy, & PayPal Low latency data streaming technology and practices remain a hot and trending topic among data engineers today. At its core, it promises to deliver data in near real time in order to provide snappy data-driven user experiences. 14:55 Fleming (3rd Fl.) Session Microservices Tales of Kafka @Cloudflare: Lessons Learnt on the Way to 1 Trillion Messages Andrea Medda, Matt Boyle Cloudflare uses Kafka to decouple microservices and communicate the creation, change or deletion of various resources via a common data format in a fault-tolerant manner. 16:10 Churchill (Ground Fl.) Session scalability Zoom: Why Does It Work? Ian Sleebe Senior Solutions Architect @Zoom During the pandemic Zoom had to scale massively to support the big move from working in the office every day to meeting online for both business and private use. How did Zoom manage this scaling dilemma? And when you join a Zoom call how does that actually work? 17:25 Fleming (3rd Fl.) Session Microservices Banking on Thousands of Microservices Suhail Patel Senior Staff Engineer @Monzo Leading the Platform and Data Functions, Previously @Citymapper Monzo has built an entire banking platform from scratch composed of many microservices; it serves over 7 million customers daily with an organisationally lean engineering team. All aspects of the bank are deployed hundreds of times a day (even on Fridays!).