Introducing TDD
TDD is a topic that is broad and deserves its own book. However, we will cover the basics so that you can apply TDD to your Scala data engineering projects.
One essential aspect of TDD in data engineering is testing the data transformations and manipulations within the pipelines you create. This involves creating unit tests that verify the correctness and accuracy of data transformations, aggregations, filters, and other data manipulation operations. Unit tests also ensure the code you create or change doesn’t break any existing processes that were previously created by you or anyone else on your team.
To accomplish this, it is important to develop code that is easily testable. You can do this by creating functions that perform one action and then composing multiple functions together to build your applications. Doing so will help to maintain code health and maintainability because you have small functions that make refactoring those functions easy.
...
