Speaker
Abstract
For autonomous vehicle companies, finding valuable insights within millions of hours of video data is essential yet challenging. This talk explores how we at Wayve are leveraging foundation models and embeddings to build scalable search tools that make data discovery faster and labeling more efficient.
Attendees will learn how we leverage vision-language models (VLMs) to retrieve relevant scenarios at scale, which is invaluable for pinpointing scenes needed to meet safety standards or evaluate specific driving behaviors. By using embeddings, we can train classifiers to detect specific driving competencies. Through an active learning loop, we refine these classifiers, enabling them to label similar scenarios across the entire dataset with high efficiency. This embedding-based approach is both fast and scalable, and it also helps us spot “bad data” clusters, like images with droplets on the lens or scenes from test tracks.
The presentation will delve into the technical infrastructure behind these tools, from vector databases that enable rapid similarity search to Flyte workflows that orchestrate scalable processing across distributed systems. We’ll also explore how query generation helps bridge the gap in positional awareness within text embeddings, allowing for more precise search across video datasets. Finally, the talk will close with a look toward future possibilities, such as on-device edge filtering, which would use embeddings to reduce storage costs by capturing only the most interesting scenarios in real time.
Designed for engineers and data scientists, this session provides a deep dive into the power of embeddings and VLMs for labeling and retrieving data at scale, making it possible to unlock insights and drive advancements in autonomous vehicle technology.
Interview
My work focuses on building scalable pipelines for running perception models at scale (e.g., segmentation, cuboids, CLIP) and enhancing semantic search capabilities to enable users to search and retrieve relevant video data using natural language. I’m also experimenting with the latest vision-language models (VLMs) for video understanding to address these challenges and perform data mining at scale.
The motivation stems from the growing need for efficient data mining and discovery in the autonomous vehicle industry. With petabytes of data collected from our fleet, we need to surface valuable insights across diverse teams—from safety validation to offline evaluation—while addressing specific challenges like dataset coverage, behavioural evaluation, and bad data removal. This talk aims to highlight how we leverage foundation models and embeddings to solve these challenges, showcasing how scalable search and retrieval tools can transform data understanding and accelerate innovation.
This talk is designed for data scientists, machine learning engineers, and anyone working on video understanding, large-scale ML pipelines, search and retrieval, or autonomous driving technology. It’s particularly relevant for teams dealing with vast datasets and looking to leverage open foundation models for smarter data processing and retrieval.
I want attendees to leave with a clear understanding of how foundation models and embeddings can revolutionise data retrieval and labelling at scale. They’ll learn how to design robust pipelines for large-scale data processing, use VLMs for scene retrieval, and apply learning loops to scale classifier training. Additionally, I hope to inspire ideas for exploring future possibilities, like on-device edge filtering, to optimise storage and processing costs.
The next big disruption will likely come from the intersection of edge computing and foundation models. Real-time, on-device filtering and decision-making will enable systems to process and prioritise data at the source, dramatically reducing storage and compute costs while enhancing the speed and accuracy of downstream tasks. This will be particularly transformative in fields like autonomous vehicles, IoT, and video understanding, where real-time insights are critical.
Topics
QCon London 2025 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
From the same track
Tuesday 8 April
10:35 Whittle (3rd Fl.) Session AI/ML Deploy MultiModal RAG Systems with vLLM Stephen Batifol Developer Advocate @Zilliz, Founding Member of the MLOps Community Berlin, Previously Machine Learning Engineer @Wolt, and Data Scientist @Brevo While text-based RAG systems have been everywhere in the last year and a half, there is so much more than text data. Images, audio, and documents often need to be processed together to provide meaningful insights, yet most RAG implementations focus solely on text. 11:45 Rutherford (4th Fl.) Unconference Unconference: AI and ML for Software Engineers 13:35 Whittle (3rd Fl.) Session AI/ML AI for Food Image Generation in Production: How & Why Iaroslav Amerkhanov Senior Data Scientist @Delivery Hero, Founder of T4lky, Creator & Host of EPAM Podcast, Speaker In this talk, we will conduct a technical overview of a client-facing Food Image Generation solution developed at Delivery Hero. 14:45 Whittle (3rd Fl.) Session AI/ML Foundation Models for Ranking: Challenges, Successes, and Lessons Learned Moumita Bhattacharya Senior Research Scientist @Netflix, Previously @Etsy Recommender systems are an integral part of most products nowadays and are often a key driver of discovery for users of the product. 15:55 Churchill (Ground Fl.) Session AI/ML Building Embedding Models for Large-Scale Real-World Applications Sahil Dua Senior Software Engineer, Machine Learning @Google, Stanford AI, Co-Author of “The Kubernetes Workshop”, Open-Source Enthusiast Embedding models are at the core of search, recommendation, and retrieval-augmented generation (RAG) systems, transforming data into meaningful representations. 17:05 Whittle (3rd Fl.) Session AI/ML How to Unlock Insights and Enable Discovery Within Petabytes of Autonomous Driving Data Kyra Mozley ML Engineer @Wayve, Previously Security & AI PhD Candidate @Royal Holloway University For autonomous vehicle companies, finding valuable insights within millions of hours of video data is essential yet challenging.