Data Discovery

Regularly in my data quality career, customers and stakeholders have told me that they know their data "inside out". However, from my experience, the application of data profiling will surprise even these stakeholders. For example, at one organization, the procure to pay process owner assured me that no suppliers were on “pay immediately” terms (meaning that invoices would be paid as soon as they were issued). Data profiling revealed that in fact, 40 suppliers were set to these terms, with a total spend of several million dollars being paid immediately instead of accruing interest for the organization.

Data profiling helps to identify the data quality rules that organizations would like their data to comply with by pointing out the “extremities” of the data. Often, these extremities are examples of something that has gone wrong with the data and needs to be corrected.

To detect these extremities, a tool typically evaluates...