Presentation: Pragmatic Resiliency: Super 6 & Sky Bet Evolution

We have an account system that's part of our monolith at the bottom, and we experienced a set of failures several times with resource starvation at the API layer because some of the processes were being consumed by some slow running processes when they were interacting with payment providers. What was happening was slow payment requests were stacking up and consuming all of the processes in that API layer, and eventually that cascades up into the higher levels consuming the resources of the upper levels until your website stops working. What we're going to talk about is how we despite the fact that we have this monolith in database in the middle, there are still ways to segment the resource usage between these different use cases. Yes, your payments might fall over, but the rest runs OK, people that still use the rest of the services. I'll talk about how we did that and how we fixed it. The other one I was thinking of talking about was around when we have very large events, biggest horse race in the country every year, and it's absolutely huge event for us and we see a huge stampede of traffic. At the end of the race specifically when people come in to see the result of that.

I'm going to talk about that we don't try to cope with that. We actually incrementally banner parts of the service because during the first couple of minutes they can go to their account history, but the bet hasn't even been settled yet. So there is no information there that's useful to them. So we just put a banner up on that element at the service that says, we're still evaluating the race, come back in a few minutes. Then we incrementally start letting people through that banner. We take the banner down gradually. That protects that slow database service. There's a few reasons that we do it this way, one of which is that when it does fail,recovering all of the services, all of the processes of these different layers, you have to restart a whole bunch of stuff, and it takes a while to bring everything back up under the pressure of all the traffic. If you don't let it fall over, it becomes accessible much more quickly to people. But there's a flip side to it, if you're not careful with the service that's holding the banner people can go away until they get results. What we've seen is on some occasions we've held that banner for too long, and actually makes the situation worse. So it's a delicate balance between when you let people back in vs, not.