In-memory caching in a distributed setting is widely used these days, and users usually expect it to be "fast and cheap". One important but often overlooked aspect of cache is tail latency– for many real-time applications, tail latency plays an outsized role in determining the overall performance and reliability.

Controlling tail latency, however, is tricky business, and much harder to achieve than good average latency. It requires understanding the details and subtleties in low-level system behavior, including memory management, threading model, locking, network I/O, etc.

Pelikan is a framework to implement distributed caches such as Memcached and Redis. This talk discusses the system aspects that are important to the performance, especially tail latency, of such services. It also covers the decisions we made in Pelikan, and how they help us reason about performance even before we run benchmarks.