Data engineering has become an indispensable function in most software engineering organizations today. Data engineering as a discipline has broadened to encompass all practices, systems, and architectures involved in storing and serving data for a myriad of needs. From OLTP systems that power user experiences to the analytics systems that power business & user insights to all of the connective tissue that keeps data consistent between these systems, data engineers have their hands full managing complex systems and architectures. The promise of the modern data stack was to simplify these architectures to reduce the operational burden many of us still wrestle with today. But, what really works? Which technologies and practices live up to their promises? What patterns and technologies have stood the test of time? What are some pitfalls that you need to be aware of? Come to this track to learn from data engineers facing & solving these problems today.
From this track
How Xata Improved the Way Developers Work With Data and Solved Some Tough Problems Along to Way
Validating your code against actual production data can be challenging. We have all been at least once on the receiving end of a "test1" email subject because somebody somewhere did a test with the production database.
Senior Software Engineer @Xata
Staff Software Engineer @Xata
High Performance Time-Series Database Design With QuestDB
In this talk we will explore the world of time series and unique set of problems time series present to the developers. We will discuss the engineering principles behind QuestDB's design, focusing on high performance.
Co-Founder & CTO @QuestDB
Powering User Experiences with Streaming Dataflow
Streaming dataflow provides a unique solution to scaling OLTP applications by allowing for an efficient cache implementation that does not diverge from the relational model of the underlying data store.
Founder & CEO @ReadySet
Open Formats: The Happy Accident Disrupting the Data Industry
Analytic databases are quietly going through an unprecedented transformation. Open table formats, like Apache Iceberg, enable multiple query engines to share one central copy of a table.
Co-Founder and CEO @Tabular, Co-creator of Apache Iceberg
Rockset - Building a Modern Analytics Database on Top of RocksDB
RocksDB, a key-value store built on the foundation of Log-Structured Merge-Tree data structures and originally open-sourced by Facebook, has played a significant role in shaping data systems over the past decades.
Founding Engineer and Architect @Rockset, Previously at RocksDB and Facebook
The Harsh Reality of Building a Realtime ML Feature Store
In a world where AI and ML are rapidly evolving, the need for efficient Realtime Feature Stores has never been greater. But the journey to create one is far from straightforward.
Principal Software Engineer @ShareChat
Book your ticket for QCon London
on April 8-10, 2024.