The Rise of the Streamhouse: Idea, Trade-Offs, and Evolution

QCon London 2026

Session streaming

The Rise of the Streamhouse: Idea, Trade-Offs, and Evolution

Tuesday Mar 17 / 03:55PM GMT, Whittle (3rd Fl.) at The QEII Centre, London

Abstract

Over the last decade, streaming architectures have largely been built around topic-centric primitives—logs, streams, and event pipelines—then stitched together with databases, caches, OLAP engines, and (increasingly) new serving systems. This approach scales, but it also accumulates architectural debt: duplicated data, fractured "truth," inconsistent guarantees, and rising operational overhead as batch, streaming, and product analytics diverge.

In this talk we introduce an emerging shift we call the Streamhouse: a table-centric streaming architecture that treats tables as the primary primitive, and models "real-time" as freshness tiers rather than separate systems. Conceptually, it extends the lakehouse by making continuous ingestion + continuous maintenance the default, so one copy of data can serve both low-latency and historical workloads with straightforward SQL access.

Then we switch from idea to practice: we'll walk through how platforms evolve from batch refreshes -> "near-real-time" -> hot/warm/cold SLAs. We'll show a pattern we built first for analytics: an OLAP serving layer with federated SQL over tiered data. That solves the initial problem—until you notice the same shape repeating across other workloads that also want "fresh + queryable" canonical data, not just dashboards. The talk follows how this pushes teams to generalize from "OLAP over the lake" to "tiered access as a platform primitive," and what trade-offs you must get right: where state lives, how tier boundaries are defined, how continuous maintenance is paid for, and what consistency guarantees you can realistically promise.

The session is practical but forward-looking: a working mental model for tiered real-time lakehouse systems, plus reference architecture patterns and a decision framework for applying the Streamhouse idea as repeatable architecture rather than a custom stack.

Interview

It's about replacing the patchwork of topics, databases, warehouses, and feature stores with a table-centric streaming architecture, the Streamhouse. A unified, continuously updated dataset serves real-time pipelines, analytics, and AI.

Senior developers should care because it cuts system complexity while improving correctness and maintainability.

Data now needs to be fresh, consistent, and immediately usable for both analytics and AI. Traditional streaming architectures scale but at the cost of duplicated data, heavy state management, and fragile pipelines.

As real-time intelligence goes mainstream, those costs become blockers. It's time to fix the foundation, not keep layering tools on top.

Keeping streaming and batch in sync. Managing large state inside stream processors. Maintaining multiple systems that all claim to be the source of truth. AI often amplifies the problem: separate feature stores, vector systems, and analytics pipelines all operating on slightly different data. The result is complexity that slows teams down and makes systems hard to reason about.

Pick one pipeline and ask: can this be modeled as a continuously updated table instead of a stream plus a database? Even a small shift toward a shared, table-backed model often simplifies joins, speeds up recovery and clarifies data ownership.

QCon attracts engineers who care about architectural fundamentals - primitives and trade-offs, not just tools. It's the right environment to have an honest conversation about what's working, what's not, and how to build simpler, more open data foundations for real-time intelligent systems.

Topics

streaming lakehouse architecture AI/ML
76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon London 2026 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Share

From the same track

Tuesday 17 March

10:35 Fleming (3rd Fl.) Session Generative AI Ontology‐Driven Observability: Building the E2E Knowledge Graph at Netflix Scale Prasanna Vijayanathan, Renzo Sanchez-Silva 11:45 Windsor (5th Fl.) Session Machine Learning Infrastructure From S3 to GPU in One Copy: Rethinking Data Loading for ML Training Onur Satici Staff Engineer @SpiralDB & a Core Maintainer of Vortex (LF AI & Data), Previously Building Distributed Systems @Palantir 13:35 Whittle (3rd Fl.) Session Kafka Introducing Tansu.io -- Rethinking Kafka for Lean Operations Peter Morgan Founder @tansu.io 14:45 Mountbatten (6th Fl.) Session AI/ML Chronon - Mixed-Workload Data Processing Framework Nikhil Simha Co-Founder & CTO @zipline.ai, Author of "Chronon Feature Platform", Previously @Airbnb, @Meta, and @Walmartlabs 15:55 Whittle (3rd Fl.) Session streaming The Rise of the Streamhouse: Idea, Trade-Offs, and Evolution Giannis Polyzos, Anton Borisov 17:05 Rutherford (4th Fl.) Event Connecting the Dots: Modern Data Engineering & Architectures (Limited Space - Registration Required)