Speaker
Abstract
There is a growing trend of databases specializing in derived data ingestion and serving. They complement more traditional “primary data” (or “source of truth”) systems. This talk proposes a new vocabulary for navigating the evolving landscape of data systems and exploring their tradeoffs.
Besides explaining what derived data is, we will also dive into four major use cases which fit in the derived data bucket, including: graphs, search, OLAP and ML feature storage. We will explore the systems which support these use cases at LinkedIn, including open source solutions such as Pinot, which powers Who Viewed My Profile, and Venice, which powers People You May Know.
Topics
QCon London 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Part of the track
Building Modern Backends Hosted by Gunnar Morling Senior Staff Software Engineer @DecodablecoFrom the same track
Monday 27 March
10:35 Fleming (3rd Fl.) Session consistency Eventual Consistency – Don’t Be Afraid! Susanne Braun Principal Tech Lead @SAPSignavio Distributed data-intensive systems are increasingly designed to be only eventually consistent. 11:50 Fleming (3rd Fl.) Session cloud native The Commoditization of the Software Stack: How Application-first Cloud Services are Changing the Game Bilgin Ibryam Principal Product Manager @Diagrid, Co-author of “Kubernetes Patterns“, Previously Architect @RedHat The runtime boundaries between applications and the cloud are shifting from virtual machines to containers and functions. The integration boundaries are moving away from pure data access to one where the mechanical parts of the application are running within the cloud. 13:40 Rutherford (4th Fl.) Unconference Unconference: Building Modern Backends Shane Hastie Global Delivery Lead @SoftEd, Lead Editor for Culture & Methods @InfoQ What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions. 14:55 Churchill (Ground Fl.) Session OpenTelemetry Effective and Efficient Observability with OpenTelemetry Daniel Gomez Blanco Principal Engineer @Skyscanner Modern architectures require effective observability solutions to be able to monitor their health and understand how system changes affect operations distributed across multiple services. 16:10 Fleming (3rd Fl.) Session api Connecting the Dots: API Design in a Distributed World Ben Gamble Adviser, Architect & Speaker About Interactive Technology, Startups & Event Driven Systems As we’ve gone from building monoliths to building microservices, the number of APIs we’ve got to manage has gone from just the database and front end, to at least one per service. 17:25 Churchill (Ground Fl.) Session Data What is Derived Data? (And do You Already Have Any?) Felix GV Principal Staff Engineer @LinkedIn There is a growing trend of databases specializing in derived data ingestion and serving. They complement more traditional “primary data” (or “source of truth”) systems.