Conducting basic outlier detection and removal
Outlier detection is a field of study in its own right, and deals with the detection of data that does not fit in a particular dataset. Advanced outlier detection techniques can be considered a part of data wrangling, but often draw from other fields, such as statistics and machine learning. For the purposes of this book, I will conduct a very basic form of outlier detection to find values that are too high. Values that are too high might be aggregates of the data or might reflect erroneous entries.
In these next few steps, you will use the built-in plotting functionality in R to observe the data and look for particularly high values.
The first step is to put the data in a form that can be easily visualized. A simple technique to capture the trend in the data by row is to find the means of all of the non-NA values in each data entry. This can be done using the rowMeans()
function in R.
Before using the roawMeans()
function, you will need to remove...