Innovations in Data Engineering

Data engineering has become an indispensable function in most software engineering organizations today. Data engineering as a discipline has broadened to encompass all practices, systems, and architectures involved in storing and serving data for a myriad of needs. From OLTP systems that power user experiences to the analytics systems that power business & user insights to all of the connective tissue that keeps data consistent between these systems, data engineers have their hands full managing complex systems and architectures. The promise of the modern data stack was to simplify these architectures to reduce the operational burden many of us still wrestle with today. But, what really works? Which technologies and practices live up to their promises? What patterns and technologies have stood the test of time? What are some pitfalls that you need to be aware of? Come to this track to learn from data engineers facing & solving these problems today.


From this track

Session

Building High-Fidelity Data Streams

Low latency data streaming technology and practices remain a hot and trending topic among data engineers today. At its core, it promises to deliver data in near real time in order to provide snappy data-driven user experiences.

Sid Anand

Chief Architect and Head of Engineering @Datazoom

Session

Change Data Capture for Microservices

Microservices represent complex business domains in the form of loosely coupled systems, but these don't exist in isolation: services need to propagate data changes amongst each other, in a reliable and scalable way.

Gunnar Morling

Senior Staff Software Engineer @Decodableco

Session

DynamoDB Transactions

NoSQL cloud database services are popular for their simple key-value operations, high availability, high scalability, and predictable performance.

Akshat Vig

Principal Engineer NoSQL databases @awscloud

Session

Speed of Apache Pinot at the Cost of Cloud Object Storage with Tiered Storage

For real-time analytics, you need systems that can provide ultra low latency (milliseconds) and extremely high throughput (hundreds of thousands of queries per second).

Neha Pawar

Founding Engineer @StarTree

Date

Monday Mar 27 / 10:00AM PDT

Share

Register

QCon London 2023
March 27 - 29, 2023

Register

Unable to make QCon London?

Join us at QCon New York on June 13-15, 2023 (in-person & online)

Registration is now open!

Track Host

Sid Anand

Chief Architect and Head of Engineering @Datazoom

Sid Anand currently serves as the Chief Architect and Head of Engineering for Datazoom, where he and his team build autonomous streaming data systems for Datazoom's high-fidelity, low latency streaming analytics needs. Prior to joining Datazoom, Sid served as PayPal's Chief Data Engineer, focusing on ways to realize the value of PayPal's hundreds of petabytes of data. Prior to joining PayPal, he held several positions including Agari's Data Architect, a Technical Lead in Search @ LinkedIn, Netflix’s Cloud Data Architect, Etsy’s VP of Engineering, and several technical roles at eBay. Sid earned his BS and MS degrees in CS from Cornell University, where he focused on Distributed Systems. Outside of work, Sid is a maintainer/committer on Apache Airflow and advises early-stage companies and several conferences (QCon, Data Council, and conferences under Skills Matter).

Read more
Find Sid Anand at: