You are viewing content from a past/completed QCon

Presentation: What Lies Between: The Challenge of Operationalising Microservices

Track: Operationalizing Microservices: Design, Deliver, Operate

Location: Fleming, 3rd flr.

Duration: 1:40pm - 2:30pm

Day of week: Tuesday

Share this on:

This presentation is now available to view on

Watch video

What You’ll Learn

  1. Hear about the operational challenges of running microservices in production that are rooted in the space between services.
  2. Learn about integrating microservices at scale, including asset management, security, observability, failure, and representing uncertainty in data.
  3. Find out how actors and reactive streams are a good fit for this task.


The biggest challenge in operationalising microservices is managing the space between them. This is the land of distributed systems: uncertainty and non-determinism. I will present practical approaches that you can use to take microservices into production or increase the value provided by existing systems. I will explore how to integrate microservices at scale, including asset management, security considerations, and representing uncertainty in data. I will examine approaches that can be used to debug, monitor, adapt, and control microservices, and I will expand on why I do not like calling this observability. Finally, I will detail tools and models for handling failure, including how to avoid making reliability worse, rather than better. I will conclude with some thoughts on how the space between microservices, and the challenges it introduces, depends on scale and perspective.


What is the work you’re doing today?


I build and operate software platforms for monitoring, aggregating, optimizing, and controlling renewable-energy assets, including energy production, battery storage, grid services, and electric-vehicle charging. These platforms leverage microservices and reactive-systems principles. I am most focused on mentoring and growing teams, and supporting these teams in their technical decisions.


Tell us about the projects you've been working on lately.


One of the most interesting things that I work on is the aggregation and control of batteries; taking thousands of batteries, in people's homes, and making them look like one power plant. It involves low-latency, closed-loop control, at scale, over the Internet, and all the problems that come with it. Using Scala, Akka, and Akka Streams for building the microservices in these systems has been a really great fit for the domain and has resulted in extremely robust services.


What are your goals for the talk?


Designing and building microservices is not easy, but I think the hardest part of operationalising microservices is in effectively managing the space between them. I will present some of the challenges that I have encountered, along with practical techniques that can be used to manage the relationships among microservices, at scale.


What do you want people to leave the talk with?


An appreciation for the operational challenges that are inherent in the relationships among microservices. I want to inspire people to use a systems-engineering approach to tackling these problems.


What do you think is the next big disruption in software?


I am not very good at predicting the future, so I probably shouldn't try. Perhaps the next big disruption is already here, but it remains underutilized, and that's cloud services. A lot of organisations run their services on cloud providers, but they just use the provider for easily provisioning compute and storage. I still see a lot of organisations building infrastructure and managing infrastructure, and not necessarily doing it very well. The opportunity cost can be huge, but is often ignored. Many cloud services are just fundamentally superior technologies for operating in the cloud. They are engineered for the cloud and they are not just service adaptations of something we could otherwise run on VMs in a data centre. I think a lot of organisations could operate with smaller engineering teams that are more focused on the core business, rather than providing infrastructure or tooling, and these teams could be responsible for a larger number of services, yet still operate them very reliably. But to do so, organisations will need to fully embrace service providers and stop worrying so much about vendor lock-in, or the illusion of “lift-and-shift”.

Speaker: Colin Breck

Sr. Staff Software Engineer @Tesla

Similar Talks

Monitoring All the Things: Keeping Track of a Mixed Estate


Principal Engineer Operations and Reliability Programme @FT

Luke Blaney

3 Disciplines for Leading a Distributed Agile Organization


Distributed Coach/Mentor & Community Cultivator

Mark Kilby

Why Distributed Systems Are Hard


Software Engineer @Pivotal

Denise Yu

A Brief History of the Future of the API


Co-Author of gRPC for WCF Developers and Creator @VisualRecode

Mark Rendle

Preparing for the Unexpected


Principal Engineer @FinancialTimes

Samuel Parkinson

Security Vulnerabilities Decomposition


Principal Application Security Consultant @Veracode

Katy Anton


  • Architectures You've Always Wondered About

    Hard-earned lessons from the names you know on scalability, reliability, security, and performance.

  • Machine Learning: The Latest Innovations

    AI and machine learning is more approachable than ever. Discover how ML, deep learning, and other modern approaches are being used in practice.

  • Kubernetes and Cloud Architectures

    Learn about cloud native architectural approaches from the leading industry experts who have operated Kubernetes and FaaS at scale, and explore the associated modern DevOps practices.

  • Evolving Java

    JVM futures, JIT directions and improvements to the runtimes stack is the theme of this year’s JVM track.

  • Next Generation Microservices: Building Distributed Systems the Right Way

    Microservice-based applications are everywhere, but well-built distributed systems are not so common. Early adopters of microservices share their insights on how to design systems the right way.

  • Chaos and Resilience: Architecting for Success

    Making systems resilient involves people and tech. Learn about strategies being used, from cognitive systems engineering to chaos engineering.

  • The Future of the API: REST, gRPC, GraphQL and More

    The humble web-based API is evolving. This track provides the what, how, and why of future APIs.

  • Streaming Data Architectures

    Today's systems move huge volumes of data. Hear how the innovators in this space are designing systems and leveraging modern data stream processing platforms.

  • Modern Compilation Targets

    Learn about the innovation happening in the compilation target space. WebAssembly is only the tip of the iceberg.

  • Modern CS in the Real World

    Head back to academia to solve today's problems in software engineering.

  • Bare Knuckle Performance

    Crushing latency and getting the most out of your hardware.

  • Leading Distributed Teams

    Remote and distributed working are increasing in popularity, but many organisations underestimate the leadership challenges. Learn from those who are doing this effectively.

  • Driving Full Cycle Engineering Teams at Every Level

    "Full cycle developers" is not just another catch phrase; it's about engineers taking ownership and delivering value, and doing so with the support of their entire organisation. Learn more from the pioneers.

  • JavaScript: Pushing the Client Beyond the Browser

    JavaScript is not just the language of the web. Join this track to learn how the innovators are pushing the boundaries of this classic language and ecosystem

  • When Things Go Wrong: GDPR, Ethics, & Politics

    Privacy, confidentiality, safety and security: learning from the frontlines, from both good and bad experiences

  • Growing Unicorns in the EU: Building, Leading and Scaling Financial Tech Start Ups

    Learn how EU FinTech innovators have designed, built, and led both their technologies and organisations.

  • Building High Performing Teams

    There are many discussions outlining the secret sauce of high-performing teams. Learn how to balance the essential ingredients of high performing teams such as trust and delegation, as well as recognising the pitfalls and problems that will ruin any recipe.

  • Scaling Security, from Device to Cloud

    Implementing effective security is vitally important, regardless of where you are deploying software applications