Connecting the Dots: API Design in a Distributed World

QCon London 2023

Session api

Connecting the Dots: API Design in a Distributed World

Monday Mar 27 / 04:10PM BST, Fleming (3rd Fl.)

Abstract

As we’ve gone from building monoliths to building microservices, the number of APIs we’ve got to manage has gone from just the database and front end, to at least one per service. 

The problem is that these are not just REST APIs. Adoption of Pub/Sub services like Apache Kafka, and RPC mechanisms like Thrift and gRPC have made streaming data possible but also increased the surface area for problems. The question is one of governance as well as one of discovery:

  • How can we let users find the data they need? 

  • How can we control and manage access?

  • What can we do to make this programmatic?

Starting with specs like OpenAPI & AyncAPI, with schemas like Avro, ProtocolBuffers and JSON, we’ll go on a walk through building a modern data stack and the tradeoffs I’ve seen along the way.  We’ll explore options for data governance and residency and what we can do about making this efficient enough for scale and fast enough for even the most demanding customers. 

We’ll be exploring this through the lens of a developer and a consumer of internal and external APIs. Follow along from familiar beginnings to modern setups and see the tradeoffs and tools along the way.

Interview

These days I mostly work around making streaming data accessible to people with open source technologies, and then the API design that goes with that, everything from async API to open API and to Kafka.

This mostly comes out of a series of continual experiences I've had around API design and the principles behind, as we advanced in the world of technology. We've gone from a single machine running a single application to many machines building into one massive distributed model. You end up with a simple problem of everything is now not on one machine and it's no longer a function call, it's an API call. What happens is a lot of the principles behind this are still in flux and we have not reached a consensus on the right way to do a lot of these things. 

I'm going to talk through some of the learnings I've had and the war stories I've seen, because it's really hard to get this right. But there are some very core principles which have still yet to be adopted when it comes to making it both a product which is built out of APIs, but also an API-first product.

The target audience here is anyone building either an API-first product or building a service for an internal product where it's like a microservice called by other things. Anyone who has a clear API plane that's one of their deliverables or something they have to interact with. The reasoning being that as soon as you have one of those things, you are an API vendor, even if you didn't expect it to be. 

From a level standpoint, I'm going from intermediate up. We're going to go quite deep into specs and wire protocols because these things matter quite a lot. It's still some good stuff for people who are just getting started on this journey, but I'm going to make some assumptions on understanding what some of these things are. I'm always happy to follow up.

 

The big two things I'd like people to take away from this, the first one is that it really, really matters to understand how you return an event or a reply from your API, whether it's a stream, whether it's a single shot, and understand what both its success and failure conditions are. Is it idempotent? Is it typed? Is it an error code? Is it a success code? Things like this matter so much that you don't really understand the consequences of what you can do with one of these APIs until you have at least encapsulated what these are.

The other one really is about performance. You end up with this idea that the network is free and it isn't. So you get to design APIs in mind that don't tweak 20 other API calls to succeed. You want to make it so it does a thing, it returns and you're done, or you send a stream out. Stream is processed in one or two steps and it's done where you build out your microservices with a steady, sprawling manner or you come up with a design, you still need to think in terms of the APIs when it comes to how fast we actually expect this thing to return and what does it depend on?

Topics

api scalability
76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon London 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Share

From the same track

Monday 27 March

10:35 Fleming (3rd Fl.) Session consistency Eventual Consistency – Don’t Be Afraid! Susanne Braun Principal Tech Lead @SAPSignavio 11:50 Fleming (3rd Fl.) Session cloud native The Commoditization of the Software Stack: How Application-first Cloud Services are Changing the Game Bilgin Ibryam Principal Product Manager @Diagrid, Co-author of “Kubernetes Patterns“, Previously Architect @RedHat 13:40 Rutherford (4th Fl.) Unconference Unconference: Building Modern Backends Shane Hastie Global Delivery Lead @SoftEd, Lead Editor for Culture & Methods @InfoQ 14:55 Churchill (Ground Fl.) Session OpenTelemetry Effective and Efficient Observability with OpenTelemetry Daniel Gomez Blanco Principal Engineer @Skyscanner 16:10 Fleming (3rd Fl.) Session api Connecting the Dots: API Design in a Distributed World Ben Gamble Adviser, Architect & Speaker About Interactive Technology, Startups & Event Driven Systems 17:25 Churchill (Ground Fl.) Session Data What is Derived Data? (And do You Already Have Any?) Felix GV Principal Staff Engineer @LinkedIn