Fixing the AI Infra Scale Problem by Stuffing 1M Sandboxes in a Single Server

The past year has seen an absolute explosion in the use of AI and agents in particular, a trend that is guaranteed to accelerate going forward. The sheer usage and amount of agents puts an immense amount of pressure on the underlying cloud infra they run on; it is not uncommon to hear of services creating 10s of millions such sandboxes in a matter of weeks. In the face of such disruptive scale, what can be done to ensure that we don’t keep throwing money at the problem by building more and increasingly larger data centers to cope with the load?

To answer the question, in this talk we’ll cover our years-long journey aimed at severely optimizing and increasing the efficiency of how workloads are deployed on the cloud, beginning with research and OSS work into unikernels (specialized, ultra-efficient virtual machines) and covering the basics of virtualization and isolation primitives. From that basis, we will describe how we leveraged that work to build a cloud platform that can start any workload in a few milliseconds, and cram up to 1M+ such lightweight VMs into a single, off-the-shelf server, allowing for millions of strongly-isolated agents to be hosted in a rack, rather than an entire data center. Finally, we will show a live demo of this in action, including details of a k8s integration that retains these millisecond semantics.