The majority of stream processing work is spent on mundane transformation tasks like scrubbing, normalisation, and filtering. Yet to perform these relatively simple tasks involves standing up multiple distributed systems that add complexity and take time to learn. Worse yet, once you’re done, you end up ping-ponging data back and forth between storage and compute just to remove a field from a JSON object.
To the data engineer, it can feel like an endless game of system whack-a-mole just to start the interesting work of actually understanding the data.
Fortunately, help has arrived in the form of WebAssembly (WASM), which enables users to create transformation modules - in the language of their choice— to perform fast data transformations on topics. By shipping these computations to the storage engine, developers can codify business practices like GDPR compliance or schema normalisation, with near native-level performance at runtime.
In this talk, James Kinley and Paul Wilkinson will provide an overview of Redpanda's WASM-based data transformation architecture, and show how it can simplify as well as boost the performance of real-time applications and data pipelines.
Speaker
James Kinley
Principal Solutions Architect @Redpanda
James loves to write code and using software to solve practical problems. His experience encompasses a mix of software engineering, machine learning, and professional services, across a variety of industries including cyber security and financial services. James is focused on large-scale data processing and streaming technologies such as Apache Spark, Kafka, and now Redpanda. As a Principal Solutions Architect, James works closely with Redpanda’s customers in EMEA to accelerate the development of production data streaming applications.
Speaker
Paul Wilkinson
Principal Solutions Architect @Redpanda
Paul Wilkinson is an innovative technologist with a multi-faceted background in research, development, training and consulting. As a hands-on practitioner with two decades of experience in the technology industry, Paul has developed a passion for solving the most difficult challenges. His current focus at Redpanda is on reducing the time-to-value for Redpanda customers through automation and tooling. As a Principal Solutions Architect, Paul works closely with Redpanda's customers in EMEA to accelerate the development of production data streaming applications.
Session Sponsored By
Redpanda is a Kafka®-compatible streaming data platform that is proven to be 10x faster and 6x lower total costs.
Date
Wednesday Apr 10 / 10:35AM BST ( 50 minutes )
Location
Westminster (4th Fl.)
Video
Video is not available
