Chapter 8: Handling Atypical Data Scenarios with Delta

"Every problem has a solution. Sometimes it just takes a long time to find the solution – even if it's right in front of your nose."       

– Daniel Handler, American author and musician

In the previous chapters, we established the need for a Lakehouse architecture paradigm to handle a wide range of use cases, from BI to AI. Data wrangling by itself may not be sufficient to get the data ready for consumption. Several conditions need to be addressed to ensure not only that the data is cleansed and transformed as per the business requirements but also that it is fit for the use case at hand. So, even when the logic of the pipelines has been ironed out, other statistical attributes of the data need to be addressed. This helps ensure that the data patterns for which it was initially designed still hold and are making the most...