Silicon Valley
Presentations about Silicon Valley
Cloud Native Data Pipelines With Apache Kafka
Cloud Native CI/CD with Jenkins X and Knative Pipelines
Cloud Native CI/CD with Jenkins X and Knative Pipelines
Airbnb’s Great Migration: Building Services at Scale
Develop Hundreds of Kubernetes Services at Scale With Airbnb
Automatic Clustering At Snowflake
What We Got Wrong: Lessons from the Birth of Microservices
Cultivating Production Excellence - Taming Complex Distributed Systems
Securing Services Using SSO
Building and Scaling a High-Performance Culture
Discovering Culture Through Artifacts
Interviews
Develop Hundreds of Kubernetes Services at Scale With Airbnb
Tell me a bit about your work at Airbnb.
Our biggest problem right now is trying to scale both our engineering productivity as well as the availability of our site. It involves breaking apart a giant application into services, and applying the best practices available. Previously we had all these bare metal instances on AWS configured with Chef. We're moving them to a more scalable SOA solution that's backed by Kubernetes, and there were some limits we hit with our old infrastructure around service discovery and stuff like that. I would consider us a fairly early adopter, and we did run into some issues. What I'm working on is solving these issues for our engineers, and more broadly for the community.
Tell me a bit about the goals for the talk.
The goals of the talk are to go over broad level things that we think you should watch out for as well as very specific things. If there are specific things we got stuck on, I'll just throw it out there. There are lots of examples about that like certain ways we annotate our pods, stuff like that. I'll try to get low level when I can, when I don't think it's worth speaking in vague terms, but also address big ideas like, what happens when we store infrastructure this way or this way. What are the problems when everything is configured through a console, or as code. When we configure as code we get a bunch of validation and continuous integration for free. Also, it's really powerful for us as being able to refactor all of our projects. One thing I want to dive into is our refactoring tool and how important that has been for all kinds of migrations. Many companies have a lot of overhead, updating security, patches, software and hardware reaching end-of-life. All migrations, even big ones, can be automated in some way. We have a Kubernetes cron job script that can update every single service. We have over 500 services now. It gets to a point where you can't do this manually or if you're doing it manually you're not doing it on a very good time frame.
