Working with dirty data
The process of cleaning data involves tidying the data, which usually results in making the dataset smaller because we have cleaned out some of the dirty data. What makes data dirty?
Dirty data can be due to invalid data, which is data that is false, incomplete, or doesn't conform to the accepted standard. An example of invalid data could be formatting errors, or data that is out of an acceptable range. Invalid data could also have the wrong type. For example, the Asterix is invalid because the acceptable formatted data is for letters only, so it can be removed.
Dirty data can be due to missing data, which is data where no value is stored. An example of missing data is data that has not been stored due to a faulty sensor. We can see that some data is missing, so it is removed from consideration.
Dirty data could also have null values. If data has null values, then programs may respond differently to the data on that basis. The nulls will need to be considered...
