Speaker
Abstract
Validating your code against actual production data can be challenging. We have all been at least once on the receiving end of a "test1" email subject because somebody somewhere did a test with the production database. And that is one of the most benign things that can happen when production data is used by mistake. But what if you could use production-level data for testing without the dangers of using the actual production data?
That was the question we tried to answer at Xata. What we came up with was a branching strategy that basically allows your data to follow your code. You branch, you test, you merge back with the fix. Let's make it safe and mask your data while we're at it.
Because data is fundamentally NOT code, this opens up a new class of tough problems to solve: how to ensure merging your data branch back happens painlessly and without downtime? Schema migrations and major version upgrades with zero downtime are one of the big pain points to deal with when using Postgres, but we are confident we managed to find a good solution for it.
Putting all these elements together allows a data developer to safely deploy changes through a data CI/CD pipeline until production. And also revert those changes if it turns out testing was insufficient and a bug sneaked through
Interview
Simona: I am currently building bits and pieces of the backend services powering Xata.
Noémi: These days I focus on bringing PostgreSQL direct access to Xata users. We provided access to the users data through a REST API and various SDKs. But there was a significant demand for SQL access to let more advanced users do whatever they want.
Simona: I think we are at a day and age where software is everywhere. Digital competencies are now considered fundamental knowledge. Software developers are a considerable part of the workforce and their workplace “ergonomics” does not reduce to a good desk and chair. I think it is time we raise awareness around the developer experience and show that there are many small things that can be improved and can go a long way.
Noémi: While working with PostgreSQL to make it more accessible, I have learned a lot about developer experience. I kept asking myself how this service can be easier to use for fellow developers. Today anyone can be a software developer. Development has become accessible to everyone. Thus, we must consider how they are serving developers. There is a lot of competition around easy-to-use tools, and the developer experience is what can set you apart from others. Who wants to waste time these days to set up a database service themselves or write useless boilerplate? If your service is not comfortable to use, people will move to the simpler solution.
The target audience for this talk is probably a seasoned developer who wants to find some new ideas they can then pitch to their leadership team about improving their day-to-day work. Improving the dev ex does not mean “tear down everything you have and use these new things”. It mostly means small incremental changes that make the developers more productive while not costing a fortune.
We would like the persona to walk away with the required toolkit in order to make their own and fellow developers' interaction with code and software in general smoother. If we can have the people hearing our talk leave with a good idea of something that they can take and adjust and use, then i think we hit our goal.
Topics
QCon London 2024 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
From the same track
Tuesday 9 April
10:35 Windsor (5th Fl.) Session database Powering User Experiences with Streaming Dataflow Alana Marzoev Founder & CEO @ReadySet Streaming dataflow provides a unique solution to scaling OLTP applications by allowing for an efficient cache implementation that does not diverge from the relational model of the underlying data store. 11:45 Windsor (5th Fl.) Session ML Feature Store The Harsh Reality of Building a Realtime ML Feature Platform Ivan Burmistrov Principal Software Engineer @ShareChat In a world where AI and ML are rapidly evolving, the need for efficient Realtime Feature Platforms has never been greater. But the journey to create one is far from straightforward. 13:35 Windsor (5th Fl.) Session architecture High Performance Time-Series Database Design With QuestDB Vlad Ilyushchenko Co-Founder & CTO @QuestDB, OG Author of PSY-Probe, Geek In this talk we will explore the world of time series and unique set of problems time series present to the developers. We will discuss the engineering principles behind QuestDB's design, focusing on high performance. 14:45 Whittle (3rd Fl.) Session architecture Improving Developer Experience Using Automated Data CI/CD Pipelines Noémi Ványi, Simona Pencea Validating your code against actual production data can be challenging. We have all been at least once on the receiving end of a "test1" email subject because somebody somewhere did a test with the production database. 15:55 Windsor (5th Fl.) Session Building Databases Rockset - Building a Modern Analytics Database on Top of RocksDB Igor Canadi Founding Engineer and Architect @Rockset, Previously at RocksDB and Facebook RocksDB, a key-value store built on the foundation of Log-Structured Merge-Tree data structures and originally open-sourced by Facebook, has played a significant role in shaping data systems over the past decades. 17:05 Rutherford (4th Fl.) Unconference Unconference: Innovations in Data Engineering An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.