Presentation: Streaming auto-scaling in Google Cloud Dataflow
Location:
- Mountbatten, 6th flr.
Duration
Day of week:
- Monday
Abstract
Resource allocation and tuning of large data-parallel pipelines has traditionally been a manual process based on human oversight and as such is costly, wasteful, and high latency. Pipelines might see spikes in input rates, organic traffic growth, or fall behind due to outages or throttling of other services. Typically, such variation forces operators to either overprovision their resources for the worst-case, or to manually monitor and adjust resources when necessary. Both of these approaches are costly.
In this talk, we describe how we tackled one particular resource allocation aspect of Google Cloud Dataflow pipelines, namely, horizontal scaling of worker pools as a function of pipeline input rate. Managing the re-distribution of key ranges across new pool sizes and the associated persistent data storage was particularly challenging. We will go into details on the signals we use and the up- and down-scaling logic.
Tracks
Covering innovative topics
Monday, 7 March
-
Back to Java
What to expect in Java 9 and Spring 5
-
Stream Processing @ Scale
Big data, fast-moving data. Practical implementation lessons on Real-time Data
-
DevOps & CI/CD
Lessons/stories on optimizing the deployment pipeline
-
Head-to-Tail Functional Languages
Free-range Monads, Tackling immutability, tales from production, and more...
-
Architecting for Failure
Your system will fail. Take control before it takes you with it
-
21st Century Culture from Geeks on the Ground
New ways to organise technology companies and workplace culture
Tuesday, 8 March
-
Architectures You've Always Wondered about
In-depth technical case studies from giants like: Microsoft, Netflix, Google, Twitter, and more...
-
Close to the Metal
Get efficiency back into your code, concepts like: cache efficient algorithm and lock free data structures
-
Containers (in production)
Real-world lessons on scalability and reliability in production container deployments
-
Modern CS in the real world
Real-world Industry adoption of modern CS ideas
-
Security, Incident Response & Fraud Detection
Master-level classes on building security into your system and responding to incidents when things go wrong.
-
Optimizing You
Keeping life in balance is always a challenge. Learning lifehacks
Wednesday, 9 March
-
Disrupting Finance
Technology advances in finance (blockchain, P2P, Machine Learning, API's)
-
Modern Native Languages
Modern native languages: Safe efficiency with Go, Rust, Swift
-
Full Stack Javascript
Level up Javascript with topics like Angular, React/ReactNative, Node, Mongo/Couch/Other, Falcor, GraphQL, etc
-
Data Science & Machine Learning Methods
A developer's data science and machine learning toolkit
-
Microservices for Mega-Architectures
Practical lessons on Microservices success.
-
Modern Agile Development
Revisiting Agile today and tackling challenges we are seeing in the wild