Presentation: Effortless Eventual Consistency with Weave Mesh

Duration

Duration: 
10:35am - 11:25am

Key Takeaways

  • Gain a functioning understanding of what Gossip is and a way to contextualize it in a broader space of communication protocols and distributed systems
  • Learn various implementations of Gossip and understand where each can be applied successfully.
  • Learn how Gossip can be used to build interesting abstractions, such as a Raft consensus algorithm on top of Weave Net.

Abstract

We built Weave Net to make it easy to network your Docker containers however you like. At its core, Weave Net is powered by an eventually-consistent, partition-tolerant, completely decentralised mesh. We're packaging this technology as Weave Mesh, so that you can build your own distributed applications with the same simplicity and reliability as Weave Net. In this talk we will explain the theory behind Weave Mesh, some of the key features we learned were important, and demonstrate some exciting use-cases, like distributed caching and state replication.

Interview

Question: 
QCon: What is Weave Mesh?
Answer: 

Peter: Weave Net is the primary product from Weaveworks. It’s a SDN for the container and Docker ecosystem. It’s totally free, totally open source. At the core of Weave Net is a gossip-based communication protocol, which we developed specifically for the SDN.

Recently, we decided it would be interesting to try to extract that, and turn it into its own thing. We’ve been experimenting with this idea since then: Weave Mesh is the gossip core of the Weave Net SDN, plus a set of layers around it that allow you to do interesting things.

At the lowest level, Weave Mesh gives you:
- membership
- multicast-style broadcast
- basic UDP-style, peer to peer packet connection

With those things, you can build interesting abstraction layers. One I’m working on now is bootstrapping the Raft consensus algorithm on top of Weave Mesh.

The idea with it is if you deploy or embed Weave Mesh, it should make creating a clustered application very easy. And because it’s extracted from the battle-hardened Weave Net SDN, you can trust that it’s reliable and robust. From there, you can build a lot of abstractions that are actually really difficult right now, in this Cloud Native computing world. The sort of things where you would normally need to talk to ZooKeeper to find your peers, or use Consul to do service discovery, or something like this—Weave Mesh will hopefully be able to solve a lot of these problems with a lot fewer caveats.

Question: 
QCon: What is the focus of the talk?
Answer: 

Peter: The talk is going to first of all be about gossip. We will discuss what gossip is, and then walk through the principles and applications of gossip protocols.

We will go through several gossip implementations. We’ll talk about what makes them interesting, and then (given this foundation) we’ll then walk through what can you build on top them.

These things apply generally to any gossip foundation, but for this example, we will talk about the specific things that we have built on top of Mesh.

Question: 
QCon: What do you want the attendees to walk away with from this talk?
Answer: 

Peter: The biggest takeaway is that the people in the audience will gain a functioning understanding of what gossip is, and a way to contextualize it in a broader space of communication protocols and distributed systems.

What is gossip and what are its semantics? What it is good for? How do gossip-y protocols compare to Paxos, Raft and other strongly consistent protocol? How do these things compare to your standard RDBMS? You will be able to better know when would pick one over the other and understand their capabilities?

Then, based on that, you can decide what makes sense to use for your application. And we show some interesting examples.

Speaker: Matthias Radestock

Co-founder and CTO @Weaveworks, previously co-founded RabbitMQ

Matthias is the co-founder and CTO of Weaveworks. He previously co-founded RabbitMQ, and a number of other software technology start-ups. Matthias has a PhD in Computer Science from Imperial College, and has spent much of the past 20 years wrestling with distributed systems.

Find Matthias Radestock at

Speaker: Peter Bourgon

Engineer @Weaveworks and Maintainer of Go Kit

Peter Bourgon is a engineer for Weaveworks. He previously worked on infrastructure, search, and distributed systems projects for SoundCloud, Oracle, and Bloomberg. Peter is the author of Go kit, a toolkit for building microservices in Go.

Find Peter Bourgon at

Similar Talks

Tracks

Monday, 6 March

Tuesday, 7 March

Wednesday, 8 March

Conference for Professional Software Developers