Improving Developer Experience Using Automated Data CI/CD Pipelines

Validating your code against actual production data can be challenging. We have all been at least once on the receiving end of a "test1" email subject because somebody somewhere did a test with the production database. And that is one of the most benign things that can happen when production data is used by mistake. But what if you could use production-level data for testing without the dangers of using the actual production data?

That was the question we tried to answer at Xata. What we came up with was a branching strategy that basically allows your data to follow your code. You branch, you test, you merge back with the fix. Let's make it safe and mask your data while we're at it.

Because data is fundamentally NOT code, this opens up a new class of tough problems to solve: how to ensure merging your data branch back happens painlessly and without downtime? Schema migrations and major version upgrades with zero downtime are one of the big pain points to deal with when using Postgres, but we are confident we managed to find a good solution for it.

Putting all these elements together allows a data developer to safely deploy changes through a data CI/CD pipeline until production. And also revert those changes if it turns out testing was insufficient and a bug sneaked through

What's the focus of your work these days?

Simona: I am currently building bits and pieces of the backend services powering Xata. 

Noémi: These days I focus on bringing PostgreSQL direct access to Xata users. We provided access to the users data through a REST API and various SDKs. But there was a significant demand for SQL access to let more advanced users do whatever they want.

What's the motivation for your talk at QCon London 2024?

Simona: I think we are at a day and age where software is everywhere. Digital competencies are now considered fundamental knowledge. Software developers are a considerable part of the workforce and their workplace “ergonomics” does not reduce to a good desk and chair. I think it is time we raise awareness around the developer experience and show that there are many small things that can be improved and can go a long way.

Noémi: While working with PostgreSQL to make it more accessible, I have learned a lot about developer experience. I kept asking myself how this service can be easier to use for fellow developers. Today anyone can be a software developer. Development has become accessible to everyone. Thus, we must consider how they are serving developers. There is a lot of competition around easy-to-use tools, and the developer experience is what can set you apart from others. Who wants to waste time these days to set up a database service themselves or write useless boilerplate? If your service is not comfortable to use, people will move to the simpler solution.

How would you describe your main persona and target audience for this session?

The target audience for this talk is probably a seasoned developer who wants to find some new ideas they can then pitch to their leadership team about improving their day-to-day work. Improving the dev ex does not mean “tear down everything you have and use these new things”. It mostly means small incremental changes that make the developers more productive while not costing a fortune.

Is there anything specific that you'd like people to walk away with after watching your session?

We would like the persona to walk away with the required toolkit in order to make their own and fellow developers' interaction with code and software in general smoother. If we can have the people hearing our talk leave with a good idea of something that they can take and adjust and use, then i think we hit our goal.


Speaker

Noémi Ványi

Senior Software Engineer @Xata

Noémi is a Senior Software engineer at Xata working with PostgreSQL. She was a maintainer of searx, the open source, privacy-respecting metasearch engine for 6 years. She also worked at Elastic as a software engineer on the open source data collectors called Beats.

Read more

Speaker

Simona Pencea

Staff Software Engineer @Xata

Simona is currently Staff Software Engineer at Xata. She has been working in IT for more than a decade, across a variety of roles ranging from embedded systems to cloud computing. Most recently she worked on building services as part of AWS IoT and she is now focused on building the backend services that power Xata.

Read more
Find Simona Pencea at:

Date

Tuesday Apr 9 / 02:45PM BST ( 50 minutes )

Location

Whittle (3rd Fl.)

Topics

architecture Data engineering PostgreSQL

Share

From the same track

Session ML Feature Store

The Harsh Reality of Building a Realtime ML Feature Platform

Tuesday Apr 9 / 11:45AM BST

In a world where AI and ML are rapidly evolving, the need for efficient Realtime Feature Platforms has never been greater. But the journey to create one is far from straightforward.

Speaker image - Ivan Burmistrov
Ivan Burmistrov

Principal Software Engineer @ShareChat

Session Building Databases

Rockset - Building a Modern Analytics Database on Top of RocksDB

Tuesday Apr 9 / 03:55PM BST

RocksDB, a key-value store built on the foundation of Log-Structured Merge-Tree data structures and originally open-sourced by Facebook, has played a significant role in shaping data systems over the past decades.

Speaker image - Igor Canadi
Igor Canadi

Founding Engineer and Architect @Rockset, Previously at RocksDB and Facebook

Session database

Powering User Experiences with Streaming Dataflow

Tuesday Apr 9 / 10:35AM BST

Streaming dataflow provides a unique solution to scaling OLTP applications by allowing for an efficient cache implementation that does not diverge from the relational model of the underlying data store.

Speaker image - Alana Marzoev
Alana Marzoev

Founder & CEO @ReadySet

Session architecture

High Performance Time-Series Database Design With QuestDB

Tuesday Apr 9 / 01:35PM BST

In this talk we will explore the world of time series and unique set of problems time series present to the developers. We will discuss the engineering principles behind QuestDB's design, focusing on high performance.

Speaker image - Vlad Ilyushchenko
Vlad Ilyushchenko

Co-Founder & CTO @QuestDB, OG Author of PSY-Probe, Geek

Session

Unconference: Innovations in Data Engineering

Tuesday Apr 9 / 05:05PM BST

An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.