Lambda Architectures: a Snapshot, a Stream, & a Bunch of Deltas | Software Development Conference QCon London 2018

Abstract

We all got so excited about breaking apart our monoliths and liberating our microservices. At HBC Tech, our work on luxury e-commerce experiences across gilt.com, saks.com, lordandtaylor.com, thebay.com and saksoff5th.com shows us that unconstrained, organic growth of microservice architectures can lead to a system with high numbers of inter-service dependencies (some of them hidden!), and deep call stacks. We asked ourselves the question: “how can we architect, build and run our services resiliently and at scale so that they can respond to requests without any immediate downstream dependencies?” Our approach is to leverage event-sourced architectures that allow core services to reliably share their information with other services asynchronously. At the heart of each solution is the snapshot, the stream, and a bunch of deltas.

In this talk I’ll give examples of these architectures in practice in our product, pricing, inventory and search services, the technology choices that have stuck: AWS Kinesis, AWS S3, AWS Lambda, Elasticache, and Calatrava. I’ll discuss how we’re making our way towards adopting Kafka as our backbone streaming implementation. And, I’ll show how we’ve begun to apply the techniques not just internally but also with external third-party vendors to keep our runtime dependencies low.

Speaker: Adrian Trenaman

SVP Engineering @HBC, Gilt & Committer Apache Karaf

As SVP Engineering, HBC Digital, Ade leads the engineering and infrastructure teams for Gilt in New York and Dublin. He is an experienced, outspoken software engineer, communicator and leader with over 20 years of experience working with teams throughout Europe, US and Asia in diverse industries such as financial services, telecoms, retail, and manufacturing. In the past, he has held the positions of CTO of Gilt Japan, Tech Lead at Gilt Groupe Ireland, Distinguished Consultant at FuseSource, Progress Software and IONA Technologies, and Lecturer at the National University of Ireland in Maynooth. He became a committer for the Apache Software Foundation in 2010, has acted as an expert reviewer to the European Commission. Adrian holds a Ph.D, Computer Science from the National University of Ireland, Maynooth, a Diploma in Business Development from the Irish Management Institute, and a BA (Mod. Hons) Computer Science from Trinity College, Dublin.

Find Adrian Trenaman at

Speaker page

Last Year's Tracks

Monday, 5 March

Leading Edge Backend Languages

Code the future! How cutting-edge programming languages and their more-established forerunners can help solve today and tomorrow’s server-side technical problems.
Security: Red XOR Blue Team

Security from the defender's AND the attacker's point of view
Microservices/ Serverless: Patterns and Practices

Stories of success and failure building modern service and function-based applications, including event sourcing, reactive, decomposition, & more.
Stream Processing in the Modern Age

Compelling applications of stream processing & recent advances in the field
DevEx: The Next Evolution of DevOps

Removing friction from the developer experience.
Modern CS in the Real World

Applied trends in Computer Science that are likely to affect Software Engineers today.
Speaker AMAs (Ask Me Anything)

Tuesday, 6 March

Next Gen Banking: It’s not all Blockchains and ICOs

Great technologies like Blockchain, smartphones and biometrics must not be limited to just faster banking, but better banking.
Observability: Logging, Alerting and Tracing

Observability in modern large distributed computer systems
Building Great Engineering Cultures & Organizations

Stories of cultural change in organizations
Architectures You've Always Wondered About

Topics like next-gen architecture mixed with applied use cases found in today's large-scale systems, self-driving cars, network routing, scale, robotics, cloud deployments, and more.
The Practice & Frontiers of AI

Learn about machine learning in practice and on the horizon
JavaScript and Beyond: The Future of the Frontend

Exploring the great frontend frameworks that make JavaScript so popular and theg JavaScript-based languages revolutionising frontend development.
Speaker AMAs (Ask Me Anything)

Wednesday, 7 March

Distributed Stateful Systems

Architecting and leveraging NoSQL revisitied
Operating Systems: LinuxKit, Unikernels, & Beyond

Applied, practical, & real-world deep-dive into industry adoption of OS, containers and virtualisation, including Linux on Windows, LinuxKit, and Unikernels
Architecting for Failure

If you're not architecting for failure you're heading for failure
Evolving Java and the JVM: Mobile, Micro and Modular

Although the Java language is holding strong as a developer favourite, new languages and paradigms are being embraced on JVM.
Tech Ethics in Action

Learning from the experiences of real-world companies driving technology decisions from ethics as much as technology.
Bare Knuckle Performance

Killing latency and getting the most out of your hardware
Speaker AMAs (Ask Me Anything)

This Year's Schedule

Track: Architectures You've Always Wondered About

Location: Fleming, 3rd flr.

Duration: 1:40pm - 2:30pm

Day of week: Tuesday

Level: Intermediate - Advanced

Abstract

Find Adrian Trenaman at

Last Year's Tracks

Monday, 5 March

Tuesday, 6 March

Wednesday, 7 March

Learn trends from innovator and early adopter companies that you can bring home to your team

Presentation: Lambda Architectures: a Snapshot, a Stream, & a Bunch of Deltas

Track: Architectures You've Always Wondered About

Location: Fleming, 3rd flr.

Duration: 1:40pm - 2:30pm

Day of week: Tuesday

Level: Intermediate - Advanced

More talks on:

Share this on:

Abstract

Find Adrian Trenaman at

Last Year's Tracks

Monday, 5 March

Tuesday, 6 March

Wednesday, 7 March

Learn trends from innovator and early adopter companies that you can bring home to your team