Modern data streaming systems process millions of messages per second. To extract value from their data, organizations employ horizontally scalable distributed event processors such as Apache Storm. Such architectures are frequently designed under the assumption that data loss and calculation errors are acceptable.

In other cases, Kappa architecture is used to fulfill performance requirements without sacrificing consistency and reliability. And given the typical data consistency and performance requirements, external state and reliance on world clock become a taxing and hardly maintainable choice.

In this talk, we will discuss how we handled the challenges when building 1.5M msg/sec global processing system with Apache Storm and Apache Kafka at Integral Ad Science. We will inspect technology agnostic patterns reemerging in multiple applications including stream rewind technique, volatile in-memory state, derived logical time and synchronization, benefits of components collocation, and precision/performance trade offs.