Compensating for missing and out-of-range data
There will be cases where some columns may have missing data. The business use case will determine how serious it is and what to do about it. If a field is being used as an input to a model, it needs a data point. Here are some strategies regarding what you can do:
- Drop the affected records. This is OK when you do not need to use the information for downstream workloads.
- Flag the row/column by adding a marker value (for example, -1). This allows you to see missing data later on without violating a schema:

- Perform basic imputing so that you have a "best guess" regarding what the data could have been, often by using the mean of non-missing data:
- The following is an example of filling default values for specific columns:
- The following is an example of filling default values for specific columns:
- The following is an example of using the "average strategy" to impute the values of the specified columns:
