Understanding the problem of data quality at the source, in transformations, and in data governance
It is hard to understate the problem of data quality and its impact on you, your organization, and even society. Let us consider a few examples to understand the breadth and impact of data with problematic quality:
- Has your dentist or doctor’s appointment ever been moved because of a double booking or scheduling issue? This is due to a simple data quality issue with inconsistency between the two systems.
- Over the last two decades, the academic fields of medicine and psychology have been shaken. Many previously accepted findings suddenly became questionable when other researchers could not reproduce the same results. This so-called replication crisis had, at its foundation, a misunderstanding and too much trust in the statistical concept of significance. It serves as a lesson for all of us: truly valid methodologies for any statistical measure are hard to come by. ...
