You are viewing content from a past/completed QCon

Presentation: What Lies Between: The Challenge of Operationalising Microservices

Track: Operationalizing Microservices: Design, Deliver, Operate

Location: Fleming, 3rd flr.

Duration: 1:40pm - 2:30pm

Day of week: Tuesday

Share this on:

This presentation is now available to view on

Watch video

What You’ll Learn

  1. Hear about the operational challenges of running microservices in production that are rooted in the space between services.
  2. Learn about integrating microservices at scale, including asset management, security, observability, failure, and representing uncertainty in data.
  3. Find out how actors and reactive streams are a good fit for this task.


The biggest challenge in operationalising microservices is managing the space between them. This is the land of distributed systems: uncertainty and non-determinism. I will present practical approaches that you can use to take microservices into production or increase the value provided by existing systems. I will explore how to integrate microservices at scale, including asset management, security considerations, and representing uncertainty in data. I will examine approaches that can be used to debug, monitor, adapt, and control microservices, and I will expand on why I do not like calling this observability. Finally, I will detail tools and models for handling failure, including how to avoid making reliability worse, rather than better. I will conclude with some thoughts on how the space between microservices, and the challenges it introduces, depends on scale and perspective.


What is the work you’re doing today?


I build and operate software platforms for monitoring, aggregating, optimizing, and controlling renewable-energy assets, including energy production, battery storage, grid services, and electric-vehicle charging. These platforms leverage microservices and reactive-systems principles. I am most focused on mentoring and growing teams, and supporting these teams in their technical decisions.


Tell us about the projects you've been working on lately.


One of the most interesting things that I work on is the aggregation and control of batteries; taking thousands of batteries, in people's homes, and making them look like one power plant. It involves low-latency, closed-loop control, at scale, over the Internet, and all the problems that come with it. Using Scala, Akka, and Akka Streams for building the microservices in these systems has been a really great fit for the domain and has resulted in extremely robust services.


What are your goals for the talk?


Designing and building microservices is not easy, but I think the hardest part of operationalising microservices is in effectively managing the space between them. I will present some of the challenges that I have encountered, along with practical techniques that can be used to manage the relationships among microservices, at scale.


What do you want people to leave the talk with?


An appreciation for the operational challenges that are inherent in the relationships among microservices. I want to inspire people to use a systems-engineering approach to tackling these problems.


What do you think is the next big disruption in software?


I am not very good at predicting the future, so I probably shouldn't try. Perhaps the next big disruption is already here, but it remains underutilized, and that's cloud services. A lot of organisations run their services on cloud providers, but they just use the provider for easily provisioning compute and storage. I still see a lot of organisations building infrastructure and managing infrastructure, and not necessarily doing it very well. The opportunity cost can be huge, but is often ignored. Many cloud services are just fundamentally superior technologies for operating in the cloud. They are engineered for the cloud and they are not just service adaptations of something we could otherwise run on VMs in a data centre. I think a lot of organisations could operate with smaller engineering teams that are more focused on the core business, rather than providing infrastructure or tooling, and these teams could be responsible for a larger number of services, yet still operate them very reliably. But to do so, organisations will need to fully embrace service providers and stop worrying so much about vendor lock-in, or the illusion of “lift-and-shift”.

Speaker: Colin Breck

Engineer @Tesla

Colin Breck has two decades of experience in developing software infrastructures for the monitoring and control of industrial applications. At Tesla, he works on distributed systems for the monitoring, aggregation, and control of distributed, renewable-energy assets. Previously, he worked on the PI System at OSIsoft, including the time-series database and publish-subscribe infrastructures. He writes a monthly essay about software, people, or teams at

Find Colin Breck at


  • Architectures You've Always Wondered About

    Hard-earned lessons from the names you know on scalability, reliability, security, and performance.

  • Machine Learning: The Latest Innovations

    AI and machine learning is more approachable than ever. Discover how ML, deep learning, and other modern approaches are being used in practice.

  • Kubernetes and Cloud Architectures

    Practical approaches and lessons learned for deploying systems into Kubernetes, cloud, and FaaS platforms.

  • Evolving Java

    JVM futures, JIT directions and improvements to the runtimes stack is the theme of this year’s JVM track.

  • Next Generation Microservices: Building Distributed Systems the Right Way

    Microservice-based applications are everywhere, but well-built distributed systems are not so common. Early adopters of microservices share their insights on how to design systems the right way.

  • Chaos and Resilience: Architecting for Success

    Making systems resilient involves people and tech. Learn about strategies being used, from cognitive systems engineering to chaos engineering.

  • The Future of the API: REST, gRPC, GraphQL and More

    The humble web-based API is evolving. This track provides the what, how, and why of future APIs.

  • Streaming Data Architectures

    Today's systems move huge volumes of data. Hear how the innovators in this space are designing systems and leveraging modern data stream processing platforms.

  • Modern Compilation Targets

    Learn about the innovation happening in the compilation target space. WebAssembly is only the tip of the iceberg.

  • Leaving the Ivory Tower: Modern CS Research in the Real World

    Thoughts pushing software forward, including consensus, CRDT's, formal methods & probabilistic programming.

  • Bare Knuckle Performance

    Crushing latency and getting the most out of your hardware.

  • Leading Distributed Teams

    Remote and distributed working are increasing in popularity, but many organisations underestimate the leadership challenges. Learn from those who are doing this effectively.

  • Full Cycle Developers: Lead the People, Manage the Process & Systems

    "Full cycle developers" is not just another catch phrase; it's about engineers taking ownership and delivering value, and doing so with the support of their entire organisation. Learn more from the pioneers.

  • JavaScript: Pushing the Client Beyond the Browser

    JavaScript is not just the language of the web. Join this track to learn how the innovators are pushing the boundaries of this classic language and ecosystem.

  • When Things Go Wrong: GDPR, Ethics, & Politics

    Privacy, confidentiality, safety and security: learning from the frontlines, from both good and bad experiences

  • Growing Unicorns in the EU: Building, Leading and Scaling Financial Tech Start Ups

    Learn how EU FinTech innovators have designed, built, and led both their technologies and organisations.

  • Building High Performing Teams

    To have a high-performing team, everybody on it has to feel and act like an owner. Learn about cultivating culture, creating psychological safety, sharing the vision effectively, and more

  • Scaling Security, from Device to Cloud

    Implementing effective security is vitally important, regardless of where you are deploying software applications.