You are viewing content from a past/completed QCon

Presentation: Streaming Log Analytics with Kafka

Track: Architecting for the Cloud / Streaming Architectures

Location: Fleming, 3rd flr.

Duration: 2:55pm - 3:45pm

Day of week: Wednesday

Share this on:

Abstract

Humio is a log analytics system that is optimised to run in on-prem instances. From a software provider’s point of view, fixing issues in an on-prem solution is inherently problematic, and we have strived to make the solution simple.

 

To achieve this goal, a Humio installation consists only of a single process per node running Humio itself, being dependent on Kafka running nearby. We then use Kafka for two things: buffering ingest and as a sequencer of events among the nodes of a Humio cluster.

We love Kafka. It has made implementing a distributed system significantly easier than if we had to do the distributed queue from scratch. Using Kafka as a building block has allowed us to focusing our attention on building a fast and pleasant search experience.

 

In this technical talk, Kresten will discuss how and why we use Kafka internally and demo how we utilise it as a straightforward event-sourcing model for distributed deployments. Kresten will then present customer cases on utilising Kafka to manage and buffer massive volumes of data ingest.

Speaker: Kresten Krab Thorup

CTO @MeetHumio (Humio)

Kresten provides technical leadership and vision at Humio. In his previous role as CTO of Trifork, Kresten was responsible for technical strategy and has provided consulting advice to teams on a variety of technologies to include distributed systems and databases, erlang, Java, and mobile application development. Kresten has been a contributor to several open source projects, including GCC, GNU Objective-C, GNU Compiled Java, Emacs, and Apache Geronimo/Yoko. Prior to Trifork, Kresten worked at NeXT Software (now acquired by Apple), where he was responsible for the development of the Objective-C tool chain, the debugger, and the runtime system. Kresten has a Ph.D. in computer science from University of Aarhus.

Find Kresten Krab Thorup at

Last Year's Tracks

Monday, 4 March

Tuesday, 5 March

Wednesday, 6 March