Presentation: Stream processing with Apache Flink

Location:

Duration

Duration: 
11:50am - 12:40pm

Day of week:

Abstract

Data streaming is gaining popularity, as more and more organizations are realizing that the nature of their data production is continuous and unbounded, and can be better served with a streaming architecture. Streaming architectures promise decreased latency from signal to decision, a radically simplified data infrastructure architecture, and the ability to cope with new data that is generated continuously. Apache Flink is a full-featured true stream processing framework with:

  • Easy to use Java- and Scala-embedded APIs that make stream analytics easy, yet provide powerful tools to deal with time and uncertainty
  • Throughput close to a million of events per second per core
  • Latencies as low as the millisecond range
  • Full support for event time and out of order arrivals with flexible windows, watermarks, and triggers
  • Exactly-once consistency guarantees, and the ability to realize distributed transactional data movement between systems (e.g., between Kafka and HDFS)
  • Ease of configuration and separation between application logic and fault tolerance via a novel asynchronous checkpointing algorithm
  • No single point of failure
  • Integration with popular open source infrastructure (e.g., Hadoop, HBase, Kafka, Cascading, Elasticsearch, …)
  • Batch processing as a special case of stream processing, including dedicated libraries for machine learning and graph processing, managed memory on-, and off-heap, and query optimization

Flink is used in several companies, including at ResearchGate, Bouygues Telecom, the Otto Group, and Capital One, and has a large and active developer community of well over 140 contributors. In this talk, we provide an overview of the system internals and its streaming-first philosophy, as well as the programming APIs.

Tracks

Covering innovative topics

Monday, 7 March

Tuesday, 8 March

Wednesday, 9 March

Conference for Professional Software Developers