You are viewing content from a past/completed QCon

Presentation: Crushing Tech Debt Through Automation at Coinbase

Track: Dark Code: The Legacy/Tech Debt Dilemma

Location: Mountbatten, 6th flr.

Day of week: Monday

Level: Intermediate

Persona: Architect

Share this on:

What You’ll Learn

  • Discuss the true cost of tech debt including security, hiring and productivity problems
  • Gain knowledge on how to maintain high engineering velocity and confidence as you scale
  • Learn how to work your way out a debt-riddled production environment to a cleaner foundation with 0-downtime
  • Assess how to improve automation across your environment by forcefully identifying and exercising weak spots

Abstract

In the past 4 years Coinbase has grown from one service in production to over 100. Our team has grown 50x, we’ve launched in 30 countries and reinvented ourselves several times as we mature and our industry evolves. We believe a failure mode of financial companies is the death of innovation once encumbered by tech debt and the fear to change that follows. While maintaining an uncompromising level of increasingly scrutinized security, we’ve systematically attacked tech debt to stay agile and step ahead of our competitors. This talk will take a deep dive into how Coinbase attacks tech debt, including deployments, testing, immutable + codified infrastructure. We’ll share a recent "Scorched Earth" event where we upgraded and rebuilt 100% of our infrastructure from scratch in 24 hours. Tech includes AWS, Docker, CoreOS, Terraform, GeoEngineer and more.

Question: 

What are your focused in your work/research today?

Answer: 

My highest engineering priority right now is implementing logging/alerting/incident response on critical systems so we can grant engineers elevated permissions while ensuring that an independent security organization can audit/monitor/remove access if credentials appear to be leaked or misused. This is another way of improving velocity through organizational resiliency. Also working on a number of new service designs and refactoring old systems.

Question: 

What is goal for your talk?

Answer: 

To reflect on what i perceive to be one of my biggest accomplishments of coinbase of designing an organization resilient to tech debt.

Question: 

Ask and answer a question you feel gets to the hear of your talk... your choice.

Answer: 

My production environment is filled with tech debt, my team is afraid to change the code and our productivity is grinding to a halt. Can I save this without starting from scratch?

Yes! If you're ready to decide that you want to rid yourself of these shackles (and i generally believe you do to maintain a high-caliber environment) then you're going to need to start by establishing an expectation of automation around changes and deployments and exercise them on a regular basis. Along with the mantra of "always leaving things better than you found them" and educating engineers that deployments are always safe, repeating these deployments will incrementally turn a fearful system into one that you can safely make large changes.

Question: 

What is the level an target persona for your talk?

Answer: 

This talk targets infrastructure engineers and team leaders interested in maintaining high velocity and removing costly snowflakes in your environment that are prone to causing failures.

Speaker: Rob Witoff

Chief Architect @Coinbase

Rob is a director at Coinbase and building systems that connect you with (what may be) the future of finance. His team is responsible for the systems that collect, stream, power and analyze the data supporting transactions globally. Previously, he led Data Science from the IT CTO’s office at NASA’s Jet Propulsion Laboratory. In this capacity he helped to expand the agency’s access, opportunities and capabilities with their data. This work included the systems powering massive martian telemetry + imagery analysis, developing data capabilities for the Astronaut Office and modernizing space mission infrastructure He has also worked to extend earth’s interstellar bandwidth as a lead engineer on the International Space Station’s laser communication system, OPALS. Rob led SpaceX Launch Systems and Space Station Systems Engineering. His work contributed to the first successful american optical communication link with the International Space Station. When his head is out of the clouds, Rob is a longtime entrepreneur endurance runner and Y Combinator founder of Apigy inc.

Find Rob Witoff at

Last Year's Tracks