You are viewing content from a past/completed QCon -


Cloud Native Data Pipelines With Apache Kafka

As microservices, data services and serverless APIs proliferate in a cloud-native world, analysts still need to report on the business as a whole. Data engineers need to collect and standardize data in an increasingly complex and diverse system. Luckily, the problem is also the solution – the way to manage data in a cloud native environment is to build cloud native data pipelines.


In this presentation, we’ll discuss how data engineering requirements changed in a cloud-native world, and how the solutions change with them. We’ll share architectural patterns that are commonly used to build cloud native data infrastructure, and how they help you build flexible, scalable and reliable pipelines to give your business visibility on all your data.


Gwen Shapira

Principal Data Architect @Confluent, PMC Member @Kafka, & Committer Apache Sqoop

Gwen is a principal data architect at Confluent helping customers to achieve success with their Apache Kafka implementation. She has 15 years of experience working with code and customers to build scalable data architectures, integrating microservices, relational and big data technologies. She...

Read more
Find Gwen Shapira at:

From the same track

SESSION + Live Q&A Architecture

Crossing the River by Feeling the Stones

Deng Xiaoping once described managing the economy as crossing the river by feeling the stones—in other words have a direction but be adaptive. But in a world of constant change, how do you determine the right thing to do? Which pebble to tread on? How do you understand where you’re going and...

Simon Wardley

Top UK Tech Influencer & Researcher & @lefep (Leading Edge Forum)

SESSION + Live Q&A Apache Kafka

Streaming Log Analytics with Kafka

Humio is a log analytics system that is optimised to run in on-prem instances. From a software provider’s point of view, fixing issues in an on-prem solution is inherently problematic, and we have strived to make the solution simple.To achieve this goal, a Humio installation consists only...

Kresten Krab Thorup

CTO @MeetHumio (Humio)

SESSION + Live Q&A Architecture

RSocket: Solving Real-World Architectural Challenges

It may seem surprising that a networking protocol can have such wide-ranging implications for applications are architected – but RSocket, with its unique flexibility and set of guarantees, is being used by some of the largest cloud-based companies to achieve order-of-magnitude reductions in...

Robert Roeser

Co-Founder & CEO @netifi_inc

Andy Shi

Developer Advocate @AlibabaGroup

Ondrej Lehecka

Software Engineer @Facebook

SESSION + Live Q&A Cloud Computing

A Guided Journey of Cloud Native Featuring Monzo

The Cloud Native Trail Map provides a path for enterprises to deploy cloud native applications, and the steps they’ll encounter along their journey. However, the needs of each enterprise are rarely the same and these journeys are rarely linear.   In this talk we’ll look at how the...

Cheryl Hung

Director of Ecosystem @CloudNativeFdn

Matt Heath

Distributed Systems Engineer @monzo

View full Schedule