Simplifying Streaming Data Transforms with WebAssembly

The majority of stream processing work is spent on mundane transformation tasks like scrubbing, normalisation, and filtering. Yet to perform these relatively simple tasks involves standing up multiple distributed systems that add complexity and take time to learn. Worse yet, once you’re done, you end up ping-ponging data back and forth between storage and compute just to remove a field from a JSON object. 

To the data engineer, it can feel like an endless game of system whack-a-mole just to start the interesting work of actually understanding the data. 

Fortunately, help has arrived in the form of WebAssembly (WASM), which enables users to create transformation modules - in the language of their choice— to perform fast data transformations on topics. By shipping these computations to the storage engine, developers can codify business practices like GDPR compliance or schema normalisation, with near native-level performance at runtime. 

In this talk, James Kinley and Paul Wilkinson will provide an overview of Redpanda's WASM-based data transformation architecture, and show how it can simplify as well as boost the performance of real-time applications and data pipelines.


Speaker

James Kinley

Principal Solutions Architect @Redpanda

James loves to write code and using software to solve practical problems. His experience encompasses a mix of software engineering, machine learning, and professional services, across a variety of industries including cyber security and financial services. James is focused on large-scale data processing and streaming technologies such as Apache Spark, Kafka, and now Redpanda. As a Principal Solutions Architect, James works closely with Redpanda’s customers in EMEA to accelerate the development of production data streaming applications.

Read more

Speaker

Paul Wilkinson

Principal Solutions Architect @Redpanda

Paul Wilkinson is an innovative technologist with a multi-faceted background in research, development, training and consulting. As a hands-on practitioner with two decades of experience in the technology industry, Paul has developed a passion for solving the most difficult challenges. His current focus at Redpanda is on reducing the time-to-value for Redpanda customers through automation and tooling. As a Principal Solutions Architect, Paul works closely with Redpanda's customers in EMEA to accelerate the development of production data streaming applications.
 

Read more

Session Sponsored By

Redpanda is a Kafka®-compatible streaming data platform that is proven to be 10x faster and 6x lower total costs.

Date

Wednesday Apr 10 / 10:35AM BST ( 50 minutes )

Location

Westminster (4th Fl.)

Video

Video is not available

Share

From the same track

Session

GraphRAG - Connecting the Dots for Information Discovery

Wednesday Apr 10 / 11:45AM BST

The classic RAG application extends an LLM with private information, able to fetch answers to questions that are contained in a single chunk of text. What if the answer requires connecting the dots across multiple chunks that may not be directly similar to the question?

Speaker image - Andreas Kollegger

Andreas Kollegger

Software Engineer @Neo4J

Session

Elevating Developer Experiences with AI-Powered API Testing & Documentation

Wednesday Apr 10 / 01:35PM BST

In the modern era, APIs have exploded in their popularity, powering absolutely everything we interact with on a daily basis (whether we know it or not). Maintaining their coherence and functionality has become, to say the least, a difficult task for developers and engineering leaders.

Speaker image - Jamie White

Jamie White

Strategic Solutions Engineer @Postman

Session

Building InfluxDB 3.0 using Modular Open Source Components

Wednesday Apr 10 / 03:55PM BST

In this talk, we will describe the design of InfluxDB 3.0's new high performance time series database engine using Apache Arrow, DataFusion, Parquet and Rust.

Speaker image -  Raphael  Taylor-Davies

Raphael Taylor-Davies

Senior software engineer @InfluxData

Session

An Introduction to Event APIs

Wednesday Apr 10 / 02:45PM BST

Learn how asynchronous communication can revolutionize business, positively impact scalability and real-time decision-making, and grasp synchronous-asynchronous federation technologies for seamless integration.

Speaker image - Linus Hakansson

Linus Hakansson

Field CTO @Gravitee