You are viewing content from a past/completed QCon -

Presentation: What Lies Between: The Challenge of Operationalising Microservices

Track: Operationalizing Microservices: Design, Deliver, Operate

Location: Fleming, 3rd flr.

Duration: 1:40pm - 2:30pm

Day of week: Tuesday

This presentation is now available to view on

Watch video

What You’ll Learn

  1. Hear about the operational challenges of running microservices in production that are rooted in the space between services.
  2. Learn about integrating microservices at scale, including asset management, security, observability, failure, and representing uncertainty in data.
  3. Find out how actors and reactive streams are a good fit for this task.


The biggest challenge in operationalising microservices is managing the space between them. This is the land of distributed systems: uncertainty and non-determinism. I will present practical approaches that you can use to take microservices into production or increase the value provided by existing systems. I will explore how to integrate microservices at scale, including asset management, security considerations, and representing uncertainty in data. I will examine approaches that can be used to debug, monitor, adapt, and control microservices, and I will expand on why I do not like calling this observability. Finally, I will detail tools and models for handling failure, including how to avoid making reliability worse, rather than better. I will conclude with some thoughts on how the space between microservices, and the challenges it introduces, depends on scale and perspective.


What is the work you’re doing today?


I build and operate software platforms for monitoring, aggregating, optimizing, and controlling renewable-energy assets, including energy production, battery storage, grid services, and electric-vehicle charging. These platforms leverage microservices and reactive-systems principles. I am most focused on mentoring and growing teams, and supporting these teams in their technical decisions.


Tell us about the projects you've been working on lately.


One of the most interesting things that I work on is the aggregation and control of batteries; taking thousands of batteries, in people's homes, and making them look like one power plant. It involves low-latency, closed-loop control, at scale, over the Internet, and all the problems that come with it. Using Scala, Akka, and Akka Streams for building the microservices in these systems has been a really great fit for the domain and has resulted in extremely robust services.


What are your goals for the talk?


Designing and building microservices is not easy, but I think the hardest part of operationalising microservices is in effectively managing the space between them. I will present some of the challenges that I have encountered, along with practical techniques that can be used to manage the relationships among microservices, at scale.


What do you want people to leave the talk with?


An appreciation for the operational challenges that are inherent in the relationships among microservices. I want to inspire people to use a systems-engineering approach to tackling these problems.


What do you think is the next big disruption in software?


I am not very good at predicting the future, so I probably shouldn't try. Perhaps the next big disruption is already here, but it remains underutilized, and that's cloud services. A lot of organisations run their services on cloud providers, but they just use the provider for easily provisioning compute and storage. I still see a lot of organisations building infrastructure and managing infrastructure, and not necessarily doing it very well. The opportunity cost can be huge, but is often ignored. Many cloud services are just fundamentally superior technologies for operating in the cloud. They are engineered for the cloud and they are not just service adaptations of something we could otherwise run on VMs in a data centre. I think a lot of organisations could operate with smaller engineering teams that are more focused on the core business, rather than providing infrastructure or tooling, and these teams could be responsible for a larger number of services, yet still operate them very reliably. But to do so, organisations will need to fully embrace service providers and stop worrying so much about vendor lock-in, or the illusion of “lift-and-shift”.

Speaker: Colin Breck

Sr. Staff Software Engineer @Tesla

Colin Breck has experience developing software infrastructures for the near real-time monitoring and control of industrial applications. At Tesla, he works on distributed systems for the monitoring, aggregation, optimization, and control of distributed-energy assets, including solar generation, battery storage, and the Supercharging network. Previously, he worked on the PI System at OSIsoft, a time-series platform for industrial monitoring and automation. He is interested in the intersection of developing people, teams, and software systems. He writes a monthly essay at

Find Colin Breck at

2020 Tracks

  • Monday, 8 March

  • Tuesday, 9 March

  • Wednesday, 10 March