Resource allocation and tuning of large data-parallel pipelines has traditionally been a manual process based on human oversight and as such is costly, wasteful, and high latency. Pipelines might see spikes in input rates, organic traffic growth, or fall behind due to outages or throttling of other services. Typically, such variation forces operators to either overprovision their resources for the worst-case, or to manually monitor and adjust resources when necessary. Both of these approaches are costly.

In this talk, we describe how we tackled one particular resource allocation aspect of Google Cloud Dataflow pipelines, namely, horizontal scaling of worker pools as a function of pipeline input rate. Managing the re-distribution of key ranges across new pool sizes and the associated persistent data storage was particularly challenging. We will go into details on the signals we use and the up- and down-scaling logic.