Many data initiatives fail to obtain reliable and consistent data. There are many different approaches to solving this problem, some of which are borrowed from well-known software engineering best practices. I would like to present a few of these solutions and, above all, show how we have implemented them in practice. Without the costs exploding.

Of course, the talk is primarily aimed at data architects and data engineers. But I also invite all software engineers to take a look at the topic. They too are increasingly coming into contact with the supply of data and can make a big difference. Plus knowing about whats possible there could make their lives easier.

Ideas on how I can provide data better in the sense of more stable and correct. Tangible possibilities of which tools and processes can be used to implement this. Without sinking into complexity and ultimately costs.

I believe that one of the big issues will be the reduction of complexity. Nowadays, modern IT landscapes are a patchwork of barely comprehensible building blocks that are somehow held together. In addition to maintenance, this also makes it difficult to test and implement new features in order to quickly adapt to the market. In this context, automation (e.g. through AI) will play a major role on both the business and technical side.

I was already familiar with duckdb, but in 2023 I understood in Hannes Mühleisen's presentation what possibilities the technology offers and what diverse applications it enables. That was the first time I was able to imagine the changes in data architectures that are possible thanks to the new, lean processing frameworks.