Applying agile to data engineering
Looking at the four statements of the Agile Manifesto, the key message is that people, results, and following real-world concerns are more important than plans and papers. This does not mean that the other elements are bad per se, just that they are not the most important ones, and their importance is in contributing to the other elements.
Here are a few general considerations that apply to the data engineering domain:
- Agile is not improvising. Having a plan is important, as it is the understanding that any plan is not set in stone, and it will keep changing. The point of the plan is to keep track of where we are, the current desired goal, and the knowledge that we have accumulated about the surrounding areas of the domain. Quick failures are important to learn blocked roads as soon as possible .
- Pick a direction in collaboration with the people who know the domain (aka business) and the ones who know the technology. Deliver and verify...
