As we’ve gone from building monoliths to building microservices, the number of APIs we’ve got to manage has gone from just the database and front end, to at least one per service.

The problem is that these are not just REST APIs. Adoption of Pub/Sub services like Apache Kafka, and RPC mechanisms like Thrift and gRPC have made streaming data possible but also increased the surface area for problems. The question is one of governance as well as one of discovery:

How can we let users find the data they need?
How can we control and manage access?
What can we do to make this programmatic?

Starting with specs like OpenAPI & AyncAPI, with schemas like Avro, ProtocolBuffers and JSON, we’ll go on a walk through building a modern data stack and the tradeoffs I’ve seen along the way. We’ll explore options for data governance and residency and what we can do about making this efficient enough for scale and fast enough for even the most demanding customers.

We’ll be exploring this through the lens of a developer and a consumer of internal and external APIs. Follow along from familiar beginnings to modern setups and see the tradeoffs and tools along the way.

Interview:

What's the focus of your work these days?

These days I mostly work around making streaming data accessible to people with open source technologies, and then the API design that goes with that, everything from async API to open API and to Kafka.

What's the motivation for your talk at QCon London 2023?

This mostly comes out of a series of continual experiences I've had around API design and the principles behind, as we advanced in the world of technology. We've gone from a single machine running a single application to many machines building into one massive distributed model. You end up with a simple problem of everything is now not on one machine and it's no longer a function call, it's an API call. What happens is a lot of the principles behind this are still in flux and we have not reached a consensus on the right way to do a lot of these things.

I'm going to talk through some of the learnings I've had and the war stories I've seen, because it's really hard to get this right. But there are some very core principles which have still yet to be adopted when it comes to making it both a product which is built out of APIs, but also an API-first product.

How would you describe your main persona and target audience for this session?

The target audience here is anyone building either an API-first product or building a service for an internal product where it's like a microservice called by other things. Anyone who has a clear API plane that's one of their deliverables or something they have to interact with. The reasoning being that as soon as you have one of those things, you are an API vendor, even if you didn't expect it to be.

From a level standpoint, I'm going from intermediate up. We're going to go quite deep into specs and wire protocols because these things matter quite a lot. It's still some good stuff for people who are just getting started on this journey, but I'm going to make some assumptions on understanding what some of these things are. I'm always happy to follow up.

Is there anything specific that you'd like people to walk away with after watching your session?

The big two things I'd like people to take away from this, the first one is that it really, really matters to understand how you return an event or a reply from your API, whether it's a stream, whether it's a single shot, and understand what both its success and failure conditions are. Is it idempotent? Is it typed? Is it an error code? Is it a success code? Things like this matter so much that you don't really understand the consequences of what you can do with one of these APIs until you have at least encapsulated what these are.

The other one really is about performance. You end up with this idea that the network is free and it isn't. So you get to design APIs in mind that don't tweak 20 other API calls to succeed. You want to make it so it does a thing, it returns and you're done, or you send a stream out. Stream is processed in one or two steps and it's done where you build out your microservices with a steady, sprawling manner or you come up with a design, you still need to think in terms of the APIs when it comes to how fast we actually expect this thing to return and what does it depend on?

Speaker

Ben Gamble

Adviser, Architect & Speaker About Interactive Technology, Startups & Event Driven Systems

Ben has spent over 10 years leading engineering in startups and high-growth companies. As a founder, a CTO, a producer and a product leader he's bridged the gap between research and product development. Having worked with the cutting edge of augmented reality, scaling 3D gaming, and same day logistics, he’s no stranger to taking on technical challenges, and the commercial realities that entails. Having found a home in DevRel in recent times, Ben now works to make real-time data a reality for anyone who needs it with open source tools and shared ideas.