When you deploy some code, do you know its impact on your production stats, like memory usage, request duration, error rates?

In 2020, at a startup of 6 people, I volunteered to build the product feature that answers this, from scratch. How hard could it be? Right! It ran on real customer data for 6 years.

First, you need data from Datadog, Sentry, AWS and about 30 other tools, every 2 minutes, for every metric, for every customer. By 2023, that was 1.3 million times a day.

Then comes the algorithm: detecting pattern changes in metrics, with no thresholds set by hand and no LLMs. I'm not a data scientist, so my first attempts were... not great. Off-the-shelf ML failed, so I had to sit down, understand what production time series really look like, and reach for simpler statistical tools. They did better than anything off the shelf.

A year later, Postgres started running out of breath: too many writes too often. The first fix: simply collect less often. The data would be wrong, but we'd stay up. Then, while we kept collecting data every 2 mins for every customer metric, we also moved the data out of Postgres and into Redis.

Another time, a migration accidentally flooded the queue with 8,500 tasks, so we scaled the workers from 2 to 10, but 5 mins later the autoscaler scaled us down to 2, because it was watching the CPU, not the queue length. We could have fixed this easily, had the only person who knew the autoscaler's rules not been on vacation.

The logic worked, and yet it never became popular. Some of our customers found it really neat, cute, sleek, just not really worth paying for. Still, it was the most fun problem I ever got to work on: no library to install, no right answer to copy, just data and trial and error.