Data engineering has become an indispensable function in most software engineering organizations today. Data engineering as a discipline has broadened to encompass all practices, systems, and architectures involved in storing and serving data for a myriad of needs. From OLTP systems that power user experiences to the analytics systems that power business & user insights to all of the connective tissue that keeps data consistent between these systems, data engineers have their hands full managing complex systems and architectures. The promise of the modern data stack was to simplify these architectures to reduce the operational burden many of us still wrestle with today. But, what really works? Which technologies and practices live up to their promises? What patterns and technologies have stood the test of time? What are some pitfalls that you need to be aware of? Come to this track to learn from data engineers facing & solving these problems today.
From this track
A New Era for Database Design with TigerBeetle
Monday Mar 27 / 10:35AM BST
The pre-recorded video of this presentation will become available within the next few hours.
Founder and CEO @TigerBeetle
Speed of Apache Pinot at the Cost of Cloud Object Storage with Tiered Storage
Monday Mar 27 / 11:50AM BST
For real-time analytics, you need systems that can provide ultra low latency (milliseconds) and extremely high throughput (hundreds of thousands of queries per second).
Founding Engineer @StarTree
Change Data Capture for Microservices
Monday Mar 27 / 01:40PM BST
Microservices represent complex business domains in the form of loosely coupled systems, but these don't exist in isolation: services need to propagate data changes amongst each other, in a reliable and scalable way.
Senior Staff Software Engineer @Decodableco
Amazon DynamoDB Distributed Transactions at Scale
Monday Mar 27 / 02:55PM BST
NoSQL databases are popular for their high availability, high scalability, and predictable performance.
Senior Principal Engineer NoSQL databases @awscloud
Multi-Region Data Streaming with Redpanda
Monday Mar 27 / 04:10PM BST
Real time data streaming platforms such as Redpanda have become a mission critical component in enterprise infrastructure. Multi-region deployments of streaming applications can provide important benefits, such as improved resiliency, better performance and cost reduction.
Software Engineer @Redpanda
In-Process Analytical Data Management with DuckDB
Monday Mar 27 / 05:25PM BST
Analytical data management systems have long been monolithic monsters far removed from the action by ancient protocols. Redesigning them to move into the application process greatly streamlines data transfer, deployment, and management.
Co-founder and CEO @duckdblabs