Thanks Niels for sharing your thoughts in this article. Compare data and software seems wrong to me. At the end of the day, data engineers and software engineers both use software to generate an output based on an input. And because software is the means in both cases, software good practises can be applied in my opinion.

Having said that, if you look at the work of data engineers from the perspective of "data pipelines", most of the things you've said make sense. But what if you look at it from the angle of "Data Products" ?

By having a Data Product mindset, you can define an MVP in the same way you can do it for an API. You can talk to your customers to understand their needs and therefore develop enough data pipelines to provide the minimum data needed. This looks pretty similar to writing endpoints to provide the minimum functionality for an API.

Because you have the customer in mind, you can validate with them constantly wether or not the data your are generating make sense in short iterations (agile) in the same way you would validate that the endpoints work as expected.

In conclusion, I agree with everything you said if you work with "data pipelines" abstraction. But I completely disagree if you consider "data products". I've personally seen this working in a Data Mesh paradigm.