Speaker
Abstract
Over the last decade, streaming architectures have largely been built around topic-centric primitives—logs, streams, and event pipelines—then stitched together with databases, caches, OLAP engines, and (increasingly) new serving systems. This approach scales, but it also accumulates architectural debt: duplicated data, fractured "truth," inconsistent guarantees, and rising operational overhead as batch, streaming, and product analytics diverge.
In this talk we introduce an emerging shift we call the Streamhouse: a table-centric streaming architecture that treats tables as the primary primitive, and models "real-time" as freshness tiers rather than separate systems. Conceptually, it extends the lakehouse by making continuous ingestion + continuous maintenance the default, so one copy of data can serve both low-latency and historical workloads with straightforward SQL access.
Then we switch from idea to practice: we'll walk through how platforms evolve from batch refreshes -> "near-real-time" -> hot/warm/cold SLAs. We'll show a pattern we built first for analytics: an OLAP serving layer with federated SQL over tiered data. That solves the initial problem—until you notice the same shape repeating across other workloads that also want "fresh + queryable" canonical data, not just dashboards. The talk follows how this pushes teams to generalize from "OLAP over the lake" to "tiered access as a platform primitive," and what trade-offs you must get right: where state lives, how tier boundaries are defined, how continuous maintenance is paid for, and what consistency guarantees you can realistically promise.
The session is practical but forward-looking: a working mental model for tiered real-time lakehouse systems, plus reference architecture patterns and a decision framework for applying the Streamhouse idea as repeatable architecture rather than a custom stack.
Interview
It's about replacing the patchwork of topics, databases, warehouses, and feature stores with a table-centric streaming architecture, the Streamhouse. A unified, continuously updated dataset serves real-time pipelines, analytics, and AI.
Senior developers should care because it cuts system complexity while improving correctness and maintainability.
Data now needs to be fresh, consistent, and immediately usable for both analytics and AI. Traditional streaming architectures scale but at the cost of duplicated data, heavy state management, and fragile pipelines.
As real-time intelligence goes mainstream, those costs become blockers. It's time to fix the foundation, not keep layering tools on top.
Keeping streaming and batch in sync. Managing large state inside stream processors. Maintaining multiple systems that all claim to be the source of truth. AI often amplifies the problem: separate feature stores, vector systems, and analytics pipelines all operating on slightly different data. The result is complexity that slows teams down and makes systems hard to reason about.
Pick one pipeline and ask: can this be modeled as a continuously updated table instead of a stream plus a database? Even a small shift toward a shared, table-backed model often simplifies joins, speeds up recovery and clarifies data ownership.
QCon attracts engineers who care about architectural fundamentals - primitives and trade-offs, not just tools. It's the right environment to have an honest conversation about what's working, what's not, and how to build simpler, more open data foundations for real-time intelligent systems.
Topics
QCon London 2026 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
From the same track
Tuesday 17 March
10:35 Fleming (3rd Fl.) Session Generative AI Ontology‐Driven Observability: Building the E2E Knowledge Graph at Netflix Scale Prasanna Vijayanathan, Renzo Sanchez-Silva As Netflix scales hundreds of client platforms, microservices, and infrastructure components, correlating user experience with system performance has become a hard data problem, not just an observability one. 11:45 Windsor (5th Fl.) Session Machine Learning Infrastructure From S3 to GPU in One Copy: Rethinking Data Loading for ML Training Onur Satici Staff Engineer @SpiralDB & a Core Maintainer of Vortex (LF AI & Data), Previously Building Distributed Systems @Palantir ML training pipelines treat data as static. Teams spend weeks preprocessing datasets into WebDataset or TFRecords, and when they want to experiment with curriculum learning or data mixing, they reprocess everything from scratch. 13:35 Whittle (3rd Fl.) Session Kafka Introducing Tansu.io -- Rethinking Kafka for Lean Operations Peter Morgan Founder @tansu.io What if Kafka brokers were ephemeral, stateless and leaderless with durability delegated to a pluggable storage layer? 14:45 Mountbatten (6th Fl.) Session AI/ML Chronon - Mixed-Workload Data Processing Framework Nikhil Simha Co-Founder & CTO @zipline.ai, Author of "Chronon Feature Platform", Previously @Airbnb, @Meta, and @Walmartlabs Chronon is a data processing framework open-sourced by Airbnb. It is adopted across organizations like Stripe, Netflix, OpenAI, and Uber. Chronon was originally built for ML applications. 15:55 Whittle (3rd Fl.) Session streaming The Rise of the Streamhouse: Idea, Trade-Offs, and Evolution Giannis Polyzos, Anton Borisov Over the last decade, streaming architectures have largely been built around topic-centric primitives—logs, streams, and event pipelines—then stitched together with databases, caches, OLAP engines, and (increasingly) new serving systems. 17:05 Rutherford (4th Fl.) Event Connecting the Dots: Modern Data Engineering & Architectures (Limited Space - Registration Required)