You are viewing content from a past/completed QCon -

Track: Modern CS in the Real World

Location: Whittle, 3rd flr.

Day of week:

Many of us went to school, spent countless hours learning theory, and breathed a sigh of relief when we finally graduated and entered the work force. It meant that we could build real systems and software and worry less about math and proofs. However, the exploding popularity of the internet and of mobile devices means that our systems need to handle tremendous load, 24-7, across the world and across changes and outages. These needs resulted in a budding collaboration between industry and academia in several areas including but not limited to Databases, AI, ML, Distributed Systems, Information Theory, Control Systems, Probability and Data Mining. What are some of these advances and which ones should you know more about? Come to this track to learn more.

Track Host: Sid Anand

Hacker at Large, Co-chair @QCon & Data Council, PMC & Committer @ApacheAirflow

Sid Anand recently served as PayPal's Chief Data Engineer, focusing on ways to realize the value of data. Prior to joining PayPal, he held several positions including Agari's Data Architect, a Technical Lead in Search & Data Analytics @ LinkedIn, Netflix’s Cloud Data Architect, Etsy’s VP of Engineering, and several technical roles at eBay. Sid earned his BS and MS degrees in CS from Cornell University, where he focused on Distributed Systems. In his spare time, he is a maintainer/committer on Apache Airflow, a co-chair for QCon, and a frequent speaker at conferences. When not working, Sid enjoys spending time with family and friends.

10:35am - 11:25am

Infinite Parallel Universes: State at the Edge

Databases and other systems of state often rely on abstractions that break down at large physical scale. In this talk I'll describe the problems with some of these traditional assumptions, and suggest better approaches for effectively managing data in geographically large networks. With these ideas in mind, I'll then describe the architecture and communication model of a global-scale, edge state system I'm prototyping at Fastly. Attendees should leave with a good understanding of the challenges of state management, when the speed of light can't be ignored.

Peter Bourgon, Principal Engineer @fastly

11:50am - 12:40pm

Coccinelle: 10 Years of Automated Evolution in the Linux Kernel

Over the last 10+ years, we have been developing the Coccinelle C-program matching and transformation tool. Coccinelle allows developers to match and transform C code via a generalization of the familiar patch syntax. As Coccinelle is independent of the C preprocessor, Coccinelle rules can be applied across a code base, regardless of configuration constraints. Over the years, Coccinelle has been extensively used in Linux kernel development, resulting in over 7000 commits to the Linux kernel, and has found its place as part of the Linux kernel development process. This talk gives an introduction to the use of Coccinelle and gives an overview of its impact on the Linux kernel.

Julia Lawall, Senior Researcher @Inria

1:40pm - 2:30pm

Record, Replay, Rinse, & Repeat: Easily Rebuilding Programmatic State

Join this talk to learn about the various implementations of record and replay systems that can be used to debug software applications. The talk begins with a discussion of the current state of the art, from both academia and the real world. We then provide an overview of the pros and cons, mostly along the axis of ease of implementation versus the capabilities of the implementation. Finally, we will compare the open-source vs proprietary models for the productising of such research systems.

The problem of understanding the reasons why a computer program has behaved in an unexpected way has troubled programmers for as long as programmable computers have existed. Due to society’s increasing dependency on computers and the rise in complexity of modern software architectures, the problem grows ever more important and ever more difficult to solve. Fortunately, today’s hardware and software systems make it feasible to provide a complete recording of a program’s execution.

In the 1970s, the idea of reversible debugging as a potential solution to this problem was first discussed in academic papers, and a naive implementation of reversible debugging was added to the GNU “GDB” interactive debugger in 2009. These systems allow developers to debug a ‘recording’ (also known as a ‘trace’) of a program’s execution, stepping forwards and backwards. Full realisation of this allows developers to rewind to any instruction in the program’s history, and examine any variable’s value (ie. any register value and any memory location) for any instruction that executed.

Until recently, the high overheads and/or incompleteness of these systems meant they were impractical for recording anything but the simplest of “Hello world” type programs. A new breed, including Undo’s LiveRecorder, the open-source rr project, and Microsoft’s Time Travel Debugger, take advantage of advances in compute, network bandwidth, and storage costs, and make it practical for use on large, complex applications.

Greg Law, Co-founder & CTO @undo_io

2:55pm - 3:45pm

Cloudstate—Towards Stateful Serverless

Serverless experience is revolutionary and will grow to dominate the future of Cloud. Function-as-a-Service (FaaS) however—with its ephemeral, stateless, and short-lived functions—is only the first step. FaaS is great for processing-intensive, parallelizable workloads, moving data from A to B providing enrichment and transformation along the way. But it is quite limited and constrained in what use-cases it addresses well, which makes it very hard/inefficient to implement general-purpose application development and distributed systems protocols.

What’s needed is a next-generation Serverless platform and programming model for general-purpose application development in the new world of real-time data and event-driven systems. What is missing is ways to manage distributed state in a scalable and available fashion, support for long-lived virtual stateful services, ways to physically co-locate data and processing, and options for choosing the right data consistency model for the job.

This talk will discuss the challenges, requirements, and introduce you to our proposed solution: Cloudstate—an Open Source project building the next generation Stateful Serverless and leveraging state models such as Event Sourcing, CQRS, and CRDTs, running on Akka, gRPC, Knative, Kubernetes, and GraalVM, in a polyglot fashion with support for Go, JavaScript, Java, Swift, Scala, Python, Kotlin, and more.

Sean Walsh, Field CTO @Lightbend

4:10pm - 5:00pm

Kafka: a Modern Distributed System

You may have heard that Apache Kafka plays a central role in modern event-driven architectures. It lies at the heart of successful microservices implementations -- its loyal following is well versed in Kafka's key APIs: the Producer and Consumer, Kafka Streams, and maybe even KSQL. But have you ever wanted to take a peek under the hood and understand how it all works?

In this talk, we’ll cover Kafka's distributed system fundamentals: the role of the Controller, the mechanics of leader election, the role of Zookeeper today and in the future. We’ll look at how read and write consistency work, how they are tunable, and how how recent innovations like exactly-once semantics and observer replicas work. Come to this talk to view Kafka through the lens of distributed systems fundamentals.

Tim Berglund, Senior Director of Developer Experience @Confluent

5:25pm - 6:15pm

Applying AI/ML for Trusted Commerce Connections @Facebook Marketplace

Facebook Marketplace is aimed at making buying and selling seamless for people locally within their communities and open up economic opportunity through social commerce. Given that Trust is a key driver of consumer decisions in Commerce, it is important for Facebook to build trust amongst buyers and sellers and on the platform. This talk will explore how we apply Machine Learning techniques to detect and take down bad/policy violating content and behavior as well as promote good content/behavior to build a trustworthy online marketplace. You will get to hear examples of adversarial behavior such as Scam, Spam, Harassment etc and how we apply ML towards solving these problems to keep people safe.

Liza Fireman, Engineering Manager @Facebook Marketplace Trust & Safety


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.