In a world where AI and ML are rapidly evolving, the need for efficient Realtime Feature Stores has never been greater. But the journey to create one is far from straightforward.
In the talk, Ivan Burmistrov will share how ShareChat - the largest social network in India - built their own Realtime Feature Store serving more than 1 billion features per second, and how they managed to make it cost-efficient.
Ivan will cover the challenges the team faced along the way, how they managed to overcome them and which ones are still not fully resolved. The talk will also cover the experience in using relatively new technologies such as ScyllaDB and RedPanda and why such technologies are crucial for building a cost efficient system. Additionally, Ivan will share how the system leverages Apache Flink in the very core of the data pipeline.
This talk will provide insights for anyone interested in real-time data pipelines and Realtime Feature Stores, in particular.
Principal Software Engineer @ShareChatRead more