Speaker
Abstract
Modern data analytics workflows rely on scaling out to huge numbers of users and compute nodes. Managing database installations to handle this scale can be unsustainably complex and expensive. Is it instead possible to get rid of all this complexity and build a database with just a client-side library and object storage?
At Man Group we have evolved over time from managing one of the largest MongoDB installations in Europe to a serverless model where users interact directly with object storage using ArcticDB, our own database engine. What we've learnt from this should be interesting to anyone interested in distributed computing, not just database development.
We will focus on topics such as:
- Our choices around ACID and our core data structures
- How to manage global state with lock-free techniques such as CRDTs
- How we manage to work with relatively high latency commodity object storage
- How object storage has evolved over time, and how advanced it is becoming
Interview
I work on ArcticDB, a client-side database engine optimised for timeseries data that's been developed from scratch in Man Group and now its own business. A lot of my work on the project has been on an optional set of server side processes to manage data replication and streaming data ingestion.
ArcticDB has an unusual serverless architecture where users use our library to interact directly with object storage, with no co-ordinating server. We've learnt a lot about distributed computing and working with object storage by building this, and I want to share some interesting techniques and design choices that we've used.
The concepts in my talk should be interesting to anyone working on distributed computing problems, whether for database development or not. It should be particularly interesting to people who work with object storage like S3. We will discuss specific design choices we've made and why, especially around our data structures and data format, so a good audience would be senior developers and technical architects who make similar decisions in their own projects.
New ideas about how to make useful software in a serverless architecture and an appreciation of how powerful modern object storage technologies are.
It might not be as big a disruption as AI but it will be interesting to see how columnar file formats evolve and whether a successor to Parquet as a de facto standard will emerge.
Topics
QCon London 2025 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
From the same track
Wednesday 9 April
10:35 Mountbatten (6th Fl.) Session database How to Build a Database Without a Server Alex Seaton Staff Engineer @ArcticDB, Previously Working on Quant Trading Systems @Man Group Modern data analytics workflows rely on scaling out to huge numbers of users and compute nodes. Managing database installations to handle this scale can be unsustainably complex and expensive. 11:45 Mountbatten (6th Fl.) Session architecture Latency: The Race to Zero...Are We There Yet? Amir Langer Principal Software Engineer @Adaptive Financial Consulting Low and predictable latency have been an edge in financial trading. Aeron has been pushing the limit on what is possible for IPC, on-premise, and in the cloud messaging. Can we do better? 13:35 Churchill (Ground Fl.) Session The Way We Manage Compliance Is Wrong… And Is Changing! Bringing DevOps Principles to Controls and Audit Ian Miell Author of "Docker in Practice" & "Learn Git/Bash the Hard Way", Consultant Partner @Container Solutions In 2025, many organizations still manage critical compliance controls through manual checks, spreadsheets, Word documents, and Confluence pages—approaches that are error-prone, inefficient, and increasingly outdated. 14:45 Churchill (Ground Fl.) Session Thinking Out of the Sandbox - Using Product Management to De-Risk Technical Innovation Miriam Fahim R&D Product Owner, Digital Assets @LSEG, Computing Rising Stars 30 R&D is the driving force behind innovation, yet many enterprises struggle to translate ambitious ideas into real-world impact. 15:55 Churchill (Ground Fl.) Session Platform Engineering Extreme DevOps Automation Sérgio Amorim Systems Engineering @Revolut, Working in Developer Experience DevOps Platform Team Revolut's rapid growth relies on a powerful microservices architecture, but this presents a significant scaling challenge.