Multi-Region Data Streaming with Redpanda

Real time data streaming platforms such as Redpanda have become a mission critical component in enterprise infrastructure. Multi-region deployments of streaming applications can provide important benefits, such as improved resiliency, better performance and cost reduction.

Developers and architects must consider various trade-offs in multi-region streaming: the level of data consistency required, the desired behaviour on failures, and how to configure data placement for best performance. This talk will introduce the design of Redpanda’s Multi-Region feature, and describe how we leveraged Raft protocol properties, a hierarchical constraint solver, automatic data balancing, and tiered storage to provide an easy to use and flexible model of building reliable multi-region data streaming applications


Speaker

Michał Maślanka

Software Engineer @Redpanda

Michal has 10+ years of experience in software engineering across different industries, with focus on distributed systems. He joined Redpanda in 2019 where he is one of the primary contributors to Redpanda core. Michal is currently responsible for Redpanda's Raft implementation and cluster orchestration bits.

Read more

Date

Monday Mar 27 / 04:10PM BST ( 50 minutes )

Location

Mountbatten (6th Fl.)

Topics

raft streaming enterprise infrastructure resiliency performance data balancing tiered storage

Share

From the same track

Session Microservices

Change Data Capture for Microservices

Monday Mar 27 / 01:40PM BST

Microservices represent complex business domains in the form of loosely coupled systems, but these don't exist in isolation: services need to propagate data changes amongst each other, in a reliable and scalable way.

Speaker image - Gunnar Morling

Gunnar Morling

Senior Staff Software Engineer @Decodableco

Session transactions

Amazon DynamoDB Distributed Transactions at Scale

Monday Mar 27 / 02:55PM BST

NoSQL databases are popular for their high availability, high scalability, and predictable performance.

Speaker image - Akshat Vig

Akshat Vig

Senior Principal Engineer NoSQL databases @awscloud

Session Apache Pinot

Speed of Apache Pinot at the Cost of Cloud Object Storage with Tiered Storage

Monday Mar 27 / 11:50AM BST

For real-time analytics, you need systems that can provide ultra low latency (milliseconds) and extremely high throughput (hundreds of thousands of queries per second).

Speaker image - Neha Pawar

Neha Pawar

Founding Engineer @StarTree

Session processing techniques

In-Process Analytical Data Management with DuckDB

Monday Mar 27 / 05:25PM BST

Analytical data management systems have long been monolithic monsters far removed from the action by ancient protocols. Redesigning them to move into the application process greatly streamlines data transfer, deployment, and management.

Speaker image - Hannes Mühleisen

Hannes Mühleisen

Co-founder and CEO @duckdblabs

Session

A New Era for Database Design with TigerBeetle

Monday Mar 27 / 10:35AM BST

The pre-recorded video of this presentation will become available within the next few hours.  

Speaker image - Joran Greef

Joran Greef

Founder and CEO @TigerBeetle