Connecting the Dots: API Design in a Distributed World

As we’ve gone from building monoliths to building microservices, the number of APIs we’ve got to manage has gone from just the database and front end, to at least one per service. 

The problem is that these are not just REST APIs. Adoption of Pub/Sub services like Apache Kafka, and RPC mechanisms like Thrift and gRPC have made streaming data possible but also increased the surface area for problems. The question is one of governance as well as one of discovery:

  • How can we let users find the data they need? 

  • How can we control and manage access?

  • What can we do to make this programmatic?

Starting with specs like OpenAPI & AyncAPI, with schemas like Avro, ProtocolBuffers and JSON, we’ll go on a walk through building a modern data stack and the tradeoffs I’ve seen along the way.  We’ll explore options for data governance and residency and what we can do about making this efficient enough for scale and fast enough for even the most demanding customers. 

We’ll be exploring this through the lens of a developer and a consumer of internal and external APIs. Follow along from familiar beginnings to modern setups and see the tradeoffs and tools along the way.

What's the focus of your work these days?

These days I mostly work around making streaming data accessible to people with open source technologies, and then the API design that goes with that, everything from async API to open API and to Kafka.

What's the motivation for your talk at QCon London 2023?

This mostly comes out of a series of continual experiences I've had around API design and the principles behind, as we advanced in the world of technology. We've gone from a single machine running a single application to many machines building into one massive distributed model. You end up with a simple problem of everything is now not on one machine and it's no longer a function call, it's an API call. What happens is a lot of the principles behind this are still in flux and we have not reached a consensus on the right way to do a lot of these things. 

I'm going to talk through some of the learnings I've had and the war stories I've seen, because it's really hard to get this right. But there are some very core principles which have still yet to be adopted when it comes to making it both a product which is built out of APIs, but also an API-first product.

How would you describe your main persona and target audience for this session?

The target audience here is anyone building either an API-first product or building a service for an internal product where it's like a microservice called by other things. Anyone who has a clear API plane that's one of their deliverables or something they have to interact with. The reasoning being that as soon as you have one of those things, you are an API vendor, even if you didn't expect it to be. 

From a level standpoint, I'm going from intermediate up. We're going to go quite deep into specs and wire protocols because these things matter quite a lot. It's still some good stuff for people who are just getting started on this journey, but I'm going to make some assumptions on understanding what some of these things are. I'm always happy to follow up.


Is there anything specific that you'd like people to walk away with after watching your session?

The big two things I'd like people to take away from this, the first one is that it really, really matters to understand how you return an event or a reply from your API, whether it's a stream, whether it's a single shot, and understand what both its success and failure conditions are. Is it idempotent? Is it typed? Is it an error code? Is it a success code? Things like this matter so much that you don't really understand the consequences of what you can do with one of these APIs until you have at least encapsulated what these are.

The other one really is about performance. You end up with this idea that the network is free and it isn't. So you get to design APIs in mind that don't tweak 20 other API calls to succeed. You want to make it so it does a thing, it returns and you're done, or you send a stream out. Stream is processed in one or two steps and it's done where you build out your microservices with a steady, sprawling manner or you come up with a design, you still need to think in terms of the APIs when it comes to how fast we actually expect this thing to return and what does it depend on?


Ben Gamble

Adviser, Architect & Speaker About Interactive Technology, Startups & Event Driven Systems

Ben has spent over 10 years leading engineering in startups and high-growth companies. As a founder, a CTO, a producer and a product leader he's bridged the gap between research and product development. Having worked with the cutting edge of augmented reality, scaling 3D gaming, and same day logistics, he’s no stranger to taking on technical challenges, and the commercial realities that entails. Having found a home in DevRel in recent times, Ben now works to make real-time data a reality for anyone who needs it with open source tools and shared ideas.

Read more


Monday Mar 27 / 04:10PM BST ( 50 minutes )


Fleming (3rd Fl.)


api scalability


From the same track

Session consistency

Eventual Consistency – Don’t Be Afraid!

Monday Mar 27 / 10:35AM BST

Distributed data-intensive systems are increasingly designed to be only eventually consistent.

Speaker image - Susanne Braun
Susanne Braun

Principal Tech Lead @SAPSignavio

Session cloud native

The Commoditization of the Software Stack: How Application-first Cloud Services are Changing the Game

Monday Mar 27 / 11:50AM BST

The runtime boundaries between applications and the cloud are shifting from virtual machines to containers and functions. The integration boundaries are moving away from pure data access to one where the mechanical parts of the application are running within the cloud.

Speaker image - Bilgin Ibryam
Bilgin Ibryam

Principal Product Manager @Diagrid, Co-author of “Kubernetes Patterns“, Previously Architect @RedHat

Session OpenTelemetry

Effective and Efficient Observability with OpenTelemetry

Monday Mar 27 / 02:55PM BST

Modern architectures require effective observability solutions to be able to monitor their health and understand how system changes affect operations distributed across multiple services.

Speaker image - Daniel Gomez Blanco
Daniel Gomez Blanco

Principal Engineer @Skyscanner

Session Data

What is Derived Data? (And do You Already Have Any?)

Monday Mar 27 / 05:25PM BST

There is a growing trend of databases specializing in derived data ingestion and serving. They complement more traditional “primary data” (or “source of truth”) systems.

Speaker image - Felix GV
Felix GV

Principal Staff Engineer @LinkedIn


Unconference: Building Modern Backends

Monday Mar 27 / 01:40PM BST

What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.

Speaker image - Shane Hastie
Shane Hastie

Global Delivery Lead @SoftEd, Lead Editor for Culture & Methods @InfoQ