Conference:March 6-8, 2017
Workshops:March 9-10, 2017
Presentation: Crushing Tech Debt Through Automation at Coinbase
Location:
- Mountbatten, 6th flr.
Duration
Day of week:
- Monday
Level:
- Intermediate
Persona:
- Architect
Key Takeaways
- Discuss the true cost of tech debt including security, hiring and productivity problems
- Gain knowledge on how to maintain high engineering velocity and confidence as you scale
- Learn how to work your way out a debt-riddled production environment to a cleaner foundation with 0-downtime
- Assess how to improve automation across your environment by forcefully identifying and exercising weak spots
Abstract
In the past 4 years Coinbase has grown from one service in production to over 100. Our team has grown 50x, we’ve launched in 30 countries and reinvented ourselves several times as we mature and our industry evolves. We believe a failure mode of financial companies is the death of innovation once encumbered by tech debt and the fear to change that follows. While maintaining an uncompromising level of increasingly scrutinized security, we’ve systematically attacked tech debt to stay agile and step ahead of our competitors. This talk will take a deep dive into how Coinbase attacks tech debt, including deployments, testing, immutable + codified infrastructure. We’ll share a recent "Scorched Earth" event where we upgraded and rebuilt 100% of our infrastructure from scratch in 24 hours. Tech includes AWS, Docker, CoreOS, Terraform, GeoEngineer and more.
Interview
My highest engineering priority right now is implementing logging/alerting/incident response on critical systems so we can grant engineers elevated permissions while ensuring that an independent security organization can audit/monitor/remove access if credentials appear to be leaked or misused. This is another way of improving velocity through organizational resiliency. Also working on a number of new service designs and refactoring old systems.
To reflect on what i perceive to be one of my biggest accomplishments of coinbase of designing an organization resilient to tech debt.
My production environment is filled with tech debt, my team is afraid to change the code and our productivity is grinding to a halt. Can I save this without starting from scratch?
Yes! If you're ready to decide that you want to rid yourself of these shackles (and i generally believe you do to maintain a high-caliber environment) then you're going to need to start by establishing an expectation of automation around changes and deployments and exercise them on a regular basis. Along with the mantra of "always leaving things better than you found them" and educating engineers that deployments are always safe, repeating these deployments will incrementally turn a fearful system into one that you can safely make large changes.
This talk targets infrastructure engineers and team leaders interested in maintaining high velocity and removing costly snowflakes in your environment that are prone to causing failures.
Similar Talks
Tracks
-
Architecting for Failure
Building fault tolerate systems that are truly resilient
-
Architectures You've Always Wondered about
QCon classic track. You know the names. Hear their lessons and challenges.
-
Modern Distributed Architectures
Migrating, deploying, and realizing modern cloud architecture.
-
Fast & Furious: Ad Serving, Finance, & Performance
Learn some of the tips and technicals of high speed, low latency systems in Ad Serving and Finance
-
Java - Performance, Patterns and Predictions
Skills embracing the evolution of Java (multi-core, cloud, modularity) and reenforcing core platform fundamentals (performance, concurrency, ubiquity).
-
Performance Mythbusting
Performance myths that need busting and the tools & techniques to get there
-
Dark Code: The Legacy/Tech Debt Dilemma
How do you evolve your code and modernize your architecture when you're stuck with part legacy code and technical debt? Lessons from the trenches.
-
Modern Learning Systems
Real world use of the latest machine learning technologies in production environments
-
Practical Cryptography & Blockchains: Beyond the Hype
Looking past the hype of blockchain technologies, alternate title: Weaselfree Cryptography & Blockchain
-
Applied JavaScript - Atomic Applications and APIs
Angular, React, Electron, Node: The hottest trends and techniques in the JavaScript space
-
Containers - State Of The Art
What is the state of the art, what's next, & other interesting questions on containers.
-
Observability Done Right: Automating Insight & Software Telemetry
Tools, practices, and methods to know what your system is doing
-
Data Engineering : Where the Rubber meets the Road in Data Science
Science does not imply engineering. Engineering tools and techniques for Data Scientists
-
Modern CS in the Real World
Applied, practical, & real-world dive into industry adoption of modern CS ideas
-
Workhorse Languages, Not Called Java
Workhorse languages not called Java.
-
Security: Lessons Learned From Being Pwned
How Attackers Think. Penetration testing techniques, exploits, toolsets, and skills of software hackers
-
Engineering Culture @{{cool_company}}
Culture, Organization Structure, Modern Agile War Stories
-
Softskills: Essential Skills for Developers
Skills for the developer in the workplace