Iteration

Pragmatic Resiliency: Super 6 & Sky Bet Evolution

What is your talk about?

You go to a lot of conferences and you hear people from Google or Netflix talking about reactive architectures or the Simian army or whatever, and it all feels quite unattainable for a lot of people. It's like this big complicated thing, there is not much like those systems. And Sky Bet has changed quite a lot over the last few years. It's scaled constantly over the years. But it's still an architecture that a lot of people will recognize. It's got a big beefy database in the middle of some older code and stored procedures and a bunch of shiny stuff around the edges that we've done over the years. But it wouldn't know a Simian army if it tripped over one, to be honest.

Can you give me example of one of the things that you might discuss?

We have an account system that's part of our monolith at the bottom, and we experienced a set of failures several times with resource starvation at the API layer because some of the processes were being consumed by some slow running processes when they were interacting with payment providers. What was happening was slow payment requests were stacking up and consuming all of the processes in that API layer, and eventually that cascades up into the higher levels consuming the resources of the upper levels until your website stops working. What we're going to talk about is how we despite the fact that we have this monolith in database in the middle, there are still ways to segment the resource usage between these different use cases. Yes, your payments might fall over, but the rest runs OK, people that still use the rest of the services. I'll talk about how we did that and how we fixed it. The other one I was thinking of talking about was around when we have very large events, biggest horse race in the country every year, and it's absolutely huge event for us and we see a huge stampede of traffic. At the end of the race specifically when people come in to see the result of that.