Building changing information data profiling scripts
To understand information, it is important to identify how information changes within the system. In order to do this, you will need to track the changes over a period of time. This can sometimes be days or months depending on your source systems, and how frequently information is modified and updated.
Getting ready
Identify all the entities and the relevant source tables. Review the definition of the tables, and group them into two buckets:
Tables with audit columns — identify any table with columns which can identify when a record was created or updated. These are normally the event or fact entities.
Tables without audit columns — identify any table which does not have any way of easily identifying a change. These are normally the dimension entities.
How to do it...
From this recipe, identifying the rate of change is the goal. This allows you to identify how you can track changes for each table:
1. Connect to the source system using...
