Modularity and auditing
In the real world, most data (especially interesting data) is tracked and monitored to ensure it’s shared and used correctly. This is crucial for traceability, debugging, and ensuring compliance with data governance policies, industry standards, or general cohesion between data output across departments. We’ve referenced the importance of creating a beautiful, DRY (which stands for don’t repeat yourself) coding environment for your pipeline as well as a balanced, Goldilocks logging strategy, but modularity and auditing take these concepts to a new level. When used in tandem, modularity and auditing create efficient, maintainable, and transparent data pipelines.
Modularity
Modularity is the concept of breaking down your code into modular components in order to reduce the complexity of your code. There’s a soft rule in code where anything with a complexity exceeding “7” (seven mentions in seven instances) should...
