Simplifying Real-Time ML Pipelines with Quix Streams: An Open Source Python Library for ML Engineers

QCon London 2023

Session python

Simplifying Real-Time ML Pipelines with Quix Streams: An Open Source Python Library for ML Engineers

Wednesday Mar 29 / 04:10PM BST, Mountbatten (6th Fl.)

Abstract

As data volume and velocity continue to increase, the need for real-time machine learning (ML) is becoming more pressing. However, building real-time ML pipelines can be complex and time-consuming, requiring expertise in both ML and streaming application development.

This talk will address this problem by introducing Quix Streams, an open-source Python library that makes it easy for data scientists and ML engineers to build real-time ML pipelines without having to learn the intricacies of building a streaming application from scratch.

In this talk, we’ll cover:

  • The growing importance of real-time ML in today's application stack, and the use cases for real-time ML processing.
  • A comparison of different ML architectures (batch, request-response, stream, and hybrid) and their pros and cons
  • The current state of streaming architecture, which is typically Java-based, and the challenges this poses for data scientists and ML engineers who primarily work in Python
  • An overview of Quix Streams and its features, including a demo of how to use it to build real-time ML pipelines

This talk is relevant for data scientists, ML engineers, and software engineers who are looking to adopt new technologies and practices in order to build real-time ML pipelines and stay current in their field.

Interview

I work as a technical authority for the engineering team here at Quix and I’m responsible for the direction of the company across the full technical stack. On top of that, I’m often on the road speaking at conferences and meetups about enabling data teams to do stream processing.

It’s traditionally been very difficult for data teams to do stream processing. It’s something I saw first-hand working at McLaren F1. Enabling Data Scientists to work directly with the product using ML/AI has the potential to revolutionalize next-gen products. The current streaming stack is not built to help with this transition to empower data teams and that’s the driving motivation behind open-sourcing Quix Streams and this talk.

Senior data scientists, software engineers and ML engineers.

I'd like them to walk away with an idea of how to build next-gen real-time products powered by ML.

Topics

python library Machine Learning real time open source
76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon London 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Share

From the same track

Wednesday 29 March

10:35 Mountbatten (6th Fl.) Session Machine Learning Strategy & Principles to Scale and Evolve MLOps @DoorDash Hien Luu Sr. Engineering Manager @Zoox & Author of MLOps with Ray, Speaker and Conference Committee Chair 11:50 Mountbatten (6th Fl.) Session Digital Twins Cognitive Digital Twins: A New Era of Intelligent Automation Yannis Georgas Intelligent Industry Lead @Capgemini 13:40 Mountbatten (6th Fl.) Session AI Responsible AI: From Principle to Practice! Mehrnoosh Sameki Principal PM Manager @Microsoft 14:55 Mountbatten (6th Fl.) Session Graphs Graph Learning at the Scale of Modern Data Warehouses Subramanya Dulloor Founding Engineer @Kumo.ai 16:10 Mountbatten (6th Fl.) Session python Simplifying Real-Time ML Pipelines with Quix Streams: An Open Source Python Library for ML Engineers Tomáš Neubauer CTO & Co-Founder @Quix